Proof point: See if notebook-http mode can be made a "plugin"

[parente]

Right now, the code for the Jupyter Notebook APIs (jupyter-websocket mode) is mixed in with the code for the notebook-defined REST API (notebook-http mode). This was done for convenience during incubation to prove that KG could support other means (transports, APIs, and protocols) for working with kernels. Interestingly enough, I had a conversation the other day that resulted in my thinking about how kernels might operate if hooked to a message queue (e.g., Rabbit MQ). Immediately, I wondered if such a setup could be made possible via the kernel gateway.

I certainly don't want to keep hacking support for additional kernel comm mechanisms into the KG code base. I wonder instead of an API could be defined to allow such things to be plugged-in easily. My hesitation is that not everything will so easily reuse the APIs and classes from the notebook server like the microservice approach did.

I'd like to take a pass at turning the notebook-http mode, the jupyter-websocket mode, and maybe even the message-queue idea into "plugins" (for lack of a better word) to the kernel gateway. If there's common ground, it'll improve the extensibility and maintainability of the project. If there's very little commonality beyond the boilerplate CLI, it may suggest each of these deserves to be in its own project.

Just writing this down. Not planning to jump on it right away.

---

From discussion with @nitind:

Clarifying the changes being implemented:

• changing "API" parameter from a string to a fully qualified module name (PR [Issue: 103] Proof point: See if notebook-http mode can be made a "plugin" #173)
• moving the current jupyter-websocket and notebook-http functionality into their own modules, not buried in services/ (PR [Issue: 103] Proof point: See if notebook-http mode can be made a "plugin" #173)
• relocate functionality specific to either mode into their modules (PR [Issue: 103] Proof point: See if notebook-http mode can be made a "plugin" #173)
• update documentation to call "modes" personalities and describe the steps to implement one (e.g., the factory function, the package, how to pass it to KG)
• relocate traitlets specific to one mode into its module (e.g. KG_LIST_KERNELS)

/cc @fperez @minrk

[minrk]

👍 thanks, @parente!

[fperez]

+1, great writeup, thanks!

[parente]

@nitind and I spoke about this face-to-face. I think it's worth taking a crack at this now since "all's quiet on the western front" at the moment. Basics we discussed:

• try the plug-in approach first, don't worry about separate repos and so on
• identify where we're doing if/else checks for mode and think about refactoring that into separate packages named after the mode (e.g., jupyter_websocket, notebook_http)
• move mode specific command line parameters to classes in the subpackages (traitlets already supports namespaces like this, so shouldn't be hard)
• implement a basic lifecycle API that the kernelgatewayapp.py invokes upon bring up (a refactoring of all the initialization methods in the main app class)
• review and doc the API, along with a short tutorial on writing your own (notebook_http mode might serve as an example)

We can discuss design here as we go. Hopefully this can be done as a series of PRs instead of one massive fork merge.

IMHO, if we can come up with a well-defined API, we should call that release 1.0.

[nitind]

The bulk of the branching is done in gatewayapp.py itself, one for choosing the kernel pool implementation, the other for populating the list of request handlers. Breaking them out into different modules to understand how much of the main app they need would be a start.

[nitind]

Clarifying the changes being implemented:

• changing "API" parameter from a string to a fully qualified module name
• moving the current jupyter-websocket and notebook-http functionality into their own modules, not buried in services/
• relocate functionality specific to either mode into their modules
• relocate traitlets specific to one mode into its module (e.g. KG_LIST_KERNELS)

[parente]

@nitind, I think you've got a clear path toward making "personalities" as you call them pluggable. I'd like to make a release of KG 0.6.0 now with the other fixes that have gone in, and start merging the PRs you stated in the prior comment into 1.0.0.dev in master, leading up to a 1.0 release where we declare the API stable and start semantic versioning.

@minrk, @fperez does that make sense to you?

[minrk]

Makes sense to me, thanks @parente and @nitind!

[parente]

@nitind, I added the list from your previous comment to the description and turned it into one of those checkbox todo lists so we don't lose track across the handful of PRs this is going to take to close.

[parente]

With #177 in, I think we can close this. @nitind, you can do the honors if you agree. Thanks for all the work on this. It's great.

[nitind]

Closing.
/me raises the roof