Transition to Kernel Provider model for kernel management

[kevin-bates]

Some time ago, @takluyver introduced a proposal for a new mechanism of kernel management. This framework essentially supplants jupyter_client, splitting it into juptyer_kernel_mgmt and jupyter_protocol leading to a better separation of functionality. I have spent some time over the past several weeks working on juptyer_kernel_mgmt and migrating Enterprise Gateway's remote kernel support into this framework as a proof of concept.

jupyter_kernel_mgmt, among other things, introduces the notion of a Kernel Provider. Kernel Providers essentially take on the responsibility of discovery, launch and lifecycle management of a kernel. With this approach, multiple third-party application Kernel Providers can co-exist within the same server. This was not possible before since there could only be one override of KernelSpecManager (for discovery) and KernelManager (for launch and lifecycle management). This is a huge win relative to jupyter-server and allowing third-party innovations!

The base implementation provides two kernel providers out of the box - the KernelSpecProvider (for kernelspec-based kernels) and the IPykernelProvider for kernels running in the typical python/conda-installed kernels.

@takluyver has also provided examples of an ssh-based and docker-based providers, while I have migrated EG's kubernetes, yarn, and docker implementations as proof of concepts.

We would likely ONLY support async kernels (work in progress) and enable the ability to use parameterized kernels (work in progress), among other things.

I would like to move forward with incorporating these changes - similar to what has been proposed in Notebook (although the plan would be that this framework only get used in jupyter_server) - but wanted to run this past the project first.

[Zsailer]

I absolutely love this. I think we should move forward. I'm happy to help with reviewing/contributing, just let me know.

I think the most important thing we need to document is a list of traits and configurables that change as we go. This will obviously break a bunch of KernelSpec configuration in a backward-incompatible way. This is fine as long as we give a clear path to migrating.

[kevin-bates]

Sounds good @Zsailer - thanks for the input.

The KernelSpec configuration (i.e., kernel.json) does not change. However, the "sensitive" areas are for those applications that bring their own KernelSpecManager and/or KernelManager implementations (registered via the trait class overrides). In those cases, the third-party will likely need to create a Kernel Provider that performs the customization. However, since those implementations will undoubtedly be based on kernelspec configurations (since there's no other option today), we have enabled the ability to bring your own KernelSpec-based Kernel Provider. This is what all the remote kernels from EG leverage.

I'll start implementing this via a WIP PR shortly.

[Zsailer]

we have enabled the ability to bring your own KernelSpec-based Kernel Provider

Great.

I was thinking more along the lines that...
KernelSpecManager and KernelManager are both configurable from jupyter_*_config.py and CLI. This means users might have custom configuration for these objects. This proposal (I believe) replaces KernelSpecManager and changes KernelManager right? If any object and traits change names, those configurations will get lost without warning. We will want to track changing traits so that we can provide documentation for these cases.

[echarles]

I hope to get some free time soon to contribute with at least reviews and user testing this initiative.

@kevin-bates

1. You say We would likely ONLY support async kernels - Any reason for this? It sounds to me that supporting current non-async kernels would be easier/faster to implement, but maybe I am missing something?
2. I interpret this as a mandatory step to support features like remote kernel, whatever the implementation of the remote mechanism is. Correct?

[kevin-bates]

@echarles - hello again!

I apologize, I should have been more explicit in the quoted text above. I meant to imply that we should only support async kernel management. My statement has no bearing on the kernels themselves - they would not change - nor should anything we do in the server introduce any new requirements on kernel implementations themselves.

Yes, with the different kernel providers, we won't just be launching local processes. Instead, some kernel startups may require 5 to 10 seconds and we need to accommodate much better throughput in the server. We also shouldn't assume the server is servicing a single user - so the currently synchronous kernel startups (and their lifecycle management) is something that must be made "asynchronous". Besides the fact that its an obvious evolutionary step. 😄

[Zsailer]

Hey @kevin-bates, how's this work coming? Excited to see this PR 😉

[kevin-bates]

Thanks for asking @Zsailer! I've been side-tracked by other work lately, but...

It's really a matter of bringing over jupyter/notebook#4837. However, due to the amount of work just to bring this in, and despite my earlier statement of providing a WIP PR "shortly", I was hoping to get further discussion (and buy-in) from the community before committing to this work. I figure folks can look at jupyter/notebook#4837 to see what is happening since this would be that same exercise.

I also want to make sure there's a level of tolerance for this because its essentially refactoring the entire kernel management framework. So if 100% backward compatibility is expected, that's probably not going to happen. For example, traitlets is something @takluyver didn't really want to carry forward in the provider framework. That said, I believe this is good for the community long term as it paves a way for multiple custom kernel (and spec) managers to be in play simultaneously. Something that can't happen today.

I hope you understand my hesitancy here. If that's how things work (via a PR) then I'll be happy to focus on this.

[echarles]

I have been travelling through the repos and PRs... regarding remote/async feature but was not able to have a good picture on the next step for me to compose a features rich set of branches and kick that UI to try the remote/async with the new kernels management.

It's really a matter of bringing over jupyter/notebook#4837.

@kevin-bates @Zsailer From this, it sounds like the step is still to inject features in notebook and backport it to server.

To make life easier, would it make sense to skip notebook and work only on server?

• This would mean new notebook features would still be more difficult to backport, but at least, people will have good reasons to migrate to this server repo.
• The advantage would be that reviews and impacts for such breaking changes as the new kernel management will be limited to server users which could also evolve faster.
• This would avoid double work/discussions and would bring more clarity for dudes like me.

[kevin-bates]

To make life easier, would it make sense to skip notebook and work only on server?

@echarles - yes, that's the complete intention. The idea behind jupyter/notebook#4837 was to bring jupyter/notebook#4170 up to a current code base - which was nearly a year old - relative to Notebook, with the sole intention of making the transition easier to Server. Once the jupyter_server momentum picked up, between those two PRs, there was never an intention of adding Kernel Providers to Notebook. The update was merely to use a transitive property approach to getting Kernel Providers into Server.

My main concern, as I tried to note previously, is that Kernel Providers have not had a lot of discussion - yet there have been TWO PRs already. And, given the commitment, I don't want to waste time moving them again when there has been ample opportunity for discussion previously.

Regarding async support, my PR to introduce async support into the provider framework has not been merged, but I believe it can be. We wanted to get an idea of what other applications would need to do should they NOT also want to move to async.

After thinking about this last night, I think having Jupyter Server use a different kernel management framework is the right approach as this framework is more agile in the sense that parties are free to introduce new providers. Also, things like metrics will vary between providers as @minrk implies in this comment. Where careful thought needs to occur is in defining the interface (in the form of an abstract base class) that providers need to implement and that acts as the "bridge" between system-related functionality and custom, provider-specific behavior.

If the discussion would be better "seeing" the code relative to server (although, in this area of code, its identical to notebook sans a couple import paths), then I'm happy to cherry-pick the various commits into server.

Thank you for being interested in this and helping stoke the discussion!

[echarles]

@kevin-bates I am a bit frustrated to only bring questions and not real code, but this is all I can do for now upon maybe end-user testing. Anyway, good to know that focus is only server, not notebook.

I see here 2 points:

1. The usage of jupyter_kernel_mgmt by server
2. The upgrade of jupyter_kernel_mgmt to support async features.

Do you see those 2 points coupled? Otherwise, why not merging jupyter_kernel_mgmt to already move forward?

Side question: Do you need Async support to run your gateway experiments? (at the end, I am interested in remote kernels on k8s).

[kevin-bates]

Do you see those 2 points coupled?

Kernel Providers and async support are not coupled. However, introducing kernel-mgmt that does not use async, then switching it for async could introduce issues where kernel providers are not async, while the server is expecting them to be async. As a result, I'd prefer to do both at the same time - although we could probably manage transition to async via version requirements management.

Otherwise, why not merging jupyter_kernel_mgmt to already move forward?

Read concerns above. I want to make sure the community is on board here. It's a commitment and I'd like some agreement, at a minimum, to the degree of backwards compatibility that is acceptable. For example, applications that have overridden either KernelSpecManager or KernelManager will not have their implementations called as these two classes can no longer be custom-configured. Instead, those applications will need to implement kernel providers to address their customizations - which is what I've done via the gateway experiments since Enterprise Gateway created a subclass of KernelManager. If that's a show-stopper, then there's no sense in doing this work. Personally, I believe the benefits of Kernel Providers outweigh the, probably few, applications that would require changes in order to use Jupyter Server.

Do you need Async support to run your gateway experiments? (at the end, I am interested in remote kernels on k8s).

No - although, again, the implementations need to be in-sync relative to async (see what I did there 😄). Down the road, we definitely need async kernel management, period, but the code in the master branches for each of the remote kernel providers is relative to synchronous management (since the async PR for kernel-mgmt has not been merged).

[Zsailer]

My main concern, as I tried to note previously, is that Kernel Providers have not had a lot of discussion

@kevin-bates You probably know this, but... I wouldn't perceive the lack-of-response as lack-of-interest. Rather, I think very few people feel qualified to review this proposal. Those that feel qualified to review it lack the time (as this proposal takes a lot of thought).

That said, I think Jupyter Server is the perfect place to experiment with this kind of thing.

In earlier conversations about Jupyter Server, it seemed that backwards compatibility was important.

In more recent conversations, I've been told that we should not let backwards compatibility hold us back. Let's make Jupyter Server a better server component, and it will make it more enticing for projects to switch to Jupyter Server (we just have to be ready to help people make the switch later 😄).

[Zsailer]

@echarles, would you be interested in joining a Jupyter Server weekly meeting? See #111

[kevin-bates]

Thanks for the response @Zsailer! I completely agree that Server should be an idea of where we want to go and its tough to get places if we have to be 100% B/C. I would absolutely help folks get going in the new framework - no worries there.

Ok, I'll stoke up the coals and get this thing moving once I reach stopping points on other tasks (should be a day or two).

[Zsailer]

Great! No rush, @kevin-bates. The urgency earlier in the thread was from my own excitement for this work 😉

[echarles]

@echarles, would you be interested in joining a Jupyter Server weekly meeting? See #111

@Zsailer I can come as listener on the first meeting October 10th 8AM Pacific. Hopefully, I can try out as end-user by that time to give more feedback. For this, it would be great to have a recipe to follow and prepare a working jupyter distribution with the branches we are targeting. Any hint on this? (I can take back my repo/branch odyssey, but I am sure it is easier than it looks with hints from the source guys).

[willingc]

@rgbkrk @captainsafia @betatim @Carreau FYI

[Zsailer]

Thanks @willingc ❤️

[rgbkrk]

Thank you for adding me in, I need to be following this and helping wherever I can.

[blink1073]

Closing, since this is now implemented as kernel provisioners in jupyter client.