Simplify Endpoint Abstraction

[Mierdin]

Currently, endpoints within a lesson definition are not built to be flexible when it comes to how they're presented to the front-end, or how they're configured by syringe when moving between stages. This presents a few problems with respect to accomplishing the goals of the project, that we want to solve.

• Network device configuration is very limited. While we selected NAPALM as the mechanism for providing configurations to all network devices to preserve the multi-vendor aspect, not all network devices are supported by NAPALM.
• Not all endpoints are network devices, but require configurations just the same. Currently there is no mechanism for providing configuration changes other than to network devices directly supported by NAPALM
• A CLI terminal isn't always the right thing to use to show content, yet it's really the only first-class citizen when it comes to presenting lesson resources to the browser. It would be very useful to be able to show web-based resources, for instance, as an iframe. Down the road it might be useful to show non-web GUI applications, such as wireshark. These are presentation formats that are either not supported at all by Antidote today, or barely supported (meaning the UX around it is horrible and might as well not exist).

This issue is meant to document an effort to simplify endpoints within a lesson definition, and provide the flexibility we need with respect to how they're configured, and how they're presented to the front-end.

Configuration

For configuration, we place an endpoint into devices in the YAML definition to indicate the endpoint is a network device and should use NAPALM. However, this doesn't allow us to specify the type of configuration necessary (i.e. what if the device isn't supported by NAPALM? Most open source routers work better by using shell scripts or Ansible playbooks.

In addition, the actual struct used in Syringe to represent a device, or a blackbox, or a utility is exactly the same. The only difference is the map in which it is loaded on startup. This is quite messy.

What we'll do instead, is have one high-level key in the lesson definition called "endpoints". All endpoints in a lesson - applications, network devices, whatever - will be defined here. A property will be added to Endpoint that allows the author to define the type of configuration that's needed. Probably script, napalm, and ansible will suffice for now. Just like it is now with NAPALM, the author will be expected to place the relevant configuration files or scripts based on the selected configuration type, for each stage in the lesson.

We'll also need to modify the configurator image to ensure it can support all of the configuration options we want it to - we don't want to have multiple configurator images for each type.

Finally, it might also be useful to revisit how the configurator image is deployed. A kubernetes job might be overkill. It might just be best to manually execute a pod from Syringe, and keep track of it statefully until completion or failure.

Presentation

In addition, the top-level key is used to modify how the endpoint is presented to the web UI. For instance, to add an endpoint that doesn't show itself in the web UI, you add the endpoint under "blackboxes". For CLI, you add under utility or devices. You can begin to see how we're conflating the presentation use case with the configuration use case.

Also, to get iframes working, we had to do something quite hacky, outside the endpoint definition. Not ideal. In addition, there's a request to also support VNC in the future. (#110 and nre-learning/antidote-web#65)

To allow for these use cases, we should provide a field directly into the endpoint that allows you to state the presentations you wish to create. This will be an optional field, because you might not want any presentations at all (i.e. blackboxes). It will also be a list because you might want more than one (i.e. provide CLI as well as a web UI to the same endpoint). On day one, we can support both CLI and Web presentations, and this restructuring will make it possible to support VNC much more easily in the near future.

Lastly, health checks should be modified to check for all of the presentations if possible. If none, we can continue to do TCP based health checks, but would be nice if Syringe could try to connect via HTTP or VNC, just like it's doing for SSH right now.

Conclusion

I feel this is a much cleaner abstraction, which allows us to be flexible with both endpoint configuration and presentation, without conflating the two, or making it difficult to reason about.

I imagine Presentation will be addressed in one PR, and configuration in another.

/cc @nre-learning/technical-reviewers

Required Changes

The following PRs have been opened in support of this issue. This issue will remain open until all of these pull requests are merged.

• syringe
• antidote-web
• curriculum
• docs

[olberger]

+1 for the possibility to access by multiple UI channels the same enpoint. I can imagine a networking lesson where I'd use tcpdump via SSH in a first place and, on the same host, use wireshark's GUI in a second time, for instance. That endpoint could then be accessed through 2 "tabs" sumultaniously.

[Mierdin]

Quick update. Progress is going well, but I'm not going to be able to introduce the two sides of this functionality in separate patches. It would actually end up taking quite a bit more work to try to get to some middle ground, since the two concepts are sort of mixed with each other (which is what this is all about fixing anyways). Rather, I'm going to rip the band-aid off and do both in the same PR.

[Mierdin]

Progress is continuing. The intended functionality is mostly in place, just gotta go through and make sure everything is cleaned up and tests built where possible, as well as documentation updates.

Three really cool new features are emerging from this effort:

• Multiple (or none, if you want) tabs to the same resource
• web-based presentations are now a first-class resource
• All endpoints are configurable

Multiple Tabs to Same Resource:

Each endpoint has a presentations attribute, which is a slice (list) of Presentation structures. This attribute is also optional, meaning it doesn't have to be defined at all (useful for endpoints like API servers that must exist but don't need to be presented in any way to the user on the front-end. This list can also have more than one presentation. In this case, the tabs on the front-end will be named similarly, but with the name of the presentations used as a distinguisher.

Web-Based Presentations

Web-based presentations are now a first-class citizen. No more do you have to wire up some "blackbox" endpoint type, then declare some non-endpoint "iframe" thingie, etc. None of that. Just use the same presentation syntax as you do with ssh resources, but with http instead.

If the above was foreign to you, it's because I didn't really like talking about this existing feature, due to it's shoddy implementation. Now that this is in place, I'm going to feel much better about talking about this feature, and see it used in lessons.

All endpoints configurable

This is made possible by the fact the new abstraction supports Ansible (as does the configurator image). The vqfx continues to use NAPALM but through a Python script. The linux container is configured via an Ansible playbook. These two options provide a lot of flexibility beyond configuring network devices.

[Mierdin]

UPDATE: This feature is going to be merged this week. This means that if you're working on any lessons, and your selfmedicate environment pulls a new version of the syringe container, your existing lesson definition will fail to load until you convert them to the new format.

Please read the revised endpoint documentation for instructions on how to revise your endpoint definitions to the new format. You should only need to update the lesson definition file (lesson.meta.yaml) and that's it. Everything else should be backwards-compatible.

If you're currently using network devices, you'll likely want to use configurationType: napalm-junos since you're probably using the vqfx at this point. You will almost certainly want to use some kind of presentation for CLI access. Please look at the existing lessons that I converted last week for examples, you'll likely want to do something similar.

If you encounter bugs not explicitly handled in other announcements or documentation, please open a separate issue.