Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Use new kernel management APIs in notebook server 6.x #4837

Closed
wants to merge 1 commit into from
Closed

WIP: Use new kernel management APIs in notebook server 6.x #4837

wants to merge 1 commit into from

Conversation

kevin-bates
Copy link
Member

This PR is a redo of @takluyver's PR #4170 applied against the Notebook 6 code base since the original PR was produced in November of last year. Although there's no intention of merging this into Notebook, the idea is that it would be used in jupyter_server and, now that it's relative to the current code base, can be easier to port to jupyter_server.

Please refer to #4170 for the rationale and discussion behind these changes. What follows is a log of the squashed commits, then "forward-ported" to 6.x.

These are the commits included. Their hashes can be found on this branch:
https://github.com/takluyver/notebook/tree/jupyter-kernel-mgmt

0dc8b70 Initial work towards using jupyter_kernel_mgmt for notebook server
ae15888 Provide kernel specs from kernel finder machinery
ddcc3dd Display kernel type ID alongside display name
27fc52a Add dependency on jupyter_kernel_mgmt & jupyter_protocol
102ae7b Switch single kernelspec API handler to kernel_finder
5765854 Drop kernel_spec_manager
1dfc5d4 Fix interrupting a kernel
76ab738 Fix up kernelspec API tests to pass
f1a79bf Default kernel type name pyimport/kernel
b0e0fec Use pyimport/kernel for session API tests
fd0b60c Use pyimport/kernel for kernels API tests
c76e6bf Fix dummy objects for SessionManager tests
2d9e021 Use reworked restarter callback system
a15a1f9 Kernel restart always means new manager & new connection info
b427507 Use updated API for message handlers
90f9e68 Restarter callbacks parameters swapped again
f6e7cfd Clean up some unused imports
d87f862 Reworked kernel restart handling
24d3d0b Reorganise code for kernel websocket handler
cdd45a6 Require jupyter_kernel_mgmt >=0.3
9ed09ae Change selectors used by tests to create new notebook
00ab20c Fix race condition making AJAX requests for session
20e9db4 Fix selector for new kernel menu entry
69a79ac Try to get more information on test failure
d63cd96 Don't try to forward message from kernel when websocket closed
237a212 Avoid race condition creating duplicate sessions
80c278a Fix starting a kernel when modifying a session
0a18fb9 Fix sessionmanager tests
c3d63d0 Propagate future out for sending status messages
65d49a4 Don't crash closing websocket if handler not connected
6efb932 Rework how kernel startup works again
8233daa msg_type is part of the message header, not top-level
e80cafe There is no longer a good way to prevent a kernel from restarting
cab3416 Wait to send starting status until new client is reconnected
a8dbd48 Expect kernel_starting event in autorestart test
f6b0674 Ensure message in frontend always has buffers array
76b9044 Wait for kernel to be connected to send shutdown_request
c42fb21 There is no longer a good way to prevent a kernel auto-restarting
336a321 404 error for websocket opened to nonexistant kernel
3ab2d12 Wait for kernel for cell mode-switching tests
826ee65 Message for callback is converted to update_display_data
bb54f34 Fix deserialisation test
3bc13fb Allow for kernel state changing during session test
e4eff7a Allow execution_state to change in test_modify_kernel_id
e5bb55c Use appropriate attribute in cull debug
d721064 Culling should go through MappingKernelManager
ab84f2a Use kernel_id from provider's manager

plus a couple fixups from cherry-pick conflict resolution issues.

These are the commits included.  Their hashes can be found on this branch:
https://github.com/takluyver/notebook/tree/jupyter-kernel-mgmt

0dc8b70 Initial work towards using jupyter_kernel_mgmt for notebook server
ae15888 Provide kernel specs from kernel finder machinery
ddcc3dd Display kernel type ID alongside display name
27fc52a Add dependency on jupyter_kernel_mgmt & jupyter_protocol
102ae7b Switch single kernelspec API handler to kernel_finder
5765854 Drop kernel_spec_manager
1dfc5d4 Fix interrupting a kernel
76ab738 Fix up kernelspec API tests to pass
f1a79bf Default kernel type name pyimport/kernel
b0e0fec Use pyimport/kernel for session API tests
fd0b60c Use pyimport/kernel for kernels API tests
c76e6bf Fix dummy objects for SessionManager tests
2d9e021 Use reworked restarter callback system
a15a1f9 Kernel restart always means new manager & new connection info
b427507 Use updated API for message handlers
90f9e68 Restarter callbacks parameters swapped again
f6e7cfd Clean up some unused imports
d87f862 Reworked kernel restart handling
24d3d0b Reorganise code for kernel websocket handler
cdd45a6 Require jupyter_kernel_mgmt >=0.3
9ed09ae Change selectors used by tests to create new notebook
00ab20c Fix race condition making AJAX requests for session
20e9db4 Fix selector for new kernel menu entry
69a79ac Try to get more information on test failure
d63cd96 Don't try to forward message from kernel when websocket closed
237a212 Avoid race condition creating duplicate sessions
80c278a Fix starting a kernel when modifying a session
0a18fb9 Fix sessionmanager tests
c3d63d0 Propagate future out for sending status messages
65d49a4 Don't crash closing websocket if handler not connected
6efb932 Rework how kernel startup works again
8233daa msg_type is part of the message header, not top-level
e80cafe There is no longer a good way to prevent a kernel from restarting
cab3416 Wait to send starting status until new client is reconnected
a8dbd48 Expect kernel_starting event in autorestart test
f6b0674 Ensure message in frontend always has buffers array
76b9044 Wait for kernel to be connected to send shutdown_request
c42fb21 There is no longer a good way to prevent a kernel auto-restarting
336a321 404 error for websocket opened to nonexistant kernel
3ab2d12 Wait for kernel for cell mode-switching tests
826ee65 Message for callback is converted to update_display_data
bb54f34 Fix deserialisation test
3bc13fb Allow for kernel state changing during session test
e4eff7a Allow execution_state to change in test_modify_kernel_id
e5bb55c Use appropriate attribute in cull debug
d721064 Culling should go through MappingKernelManager
ab84f2a Use kernel_id from provider's manager

plus a couple fixups from cherry-pick conflict resolution issues.
kevin-bates pushed a commit to kevin-bates/jupyter_server that referenced this pull request Oct 4, 2019
kevin-bates pushed a commit to kevin-bates/jupyter_server that referenced this pull request Dec 7, 2019
kevin-bates pushed a commit to kevin-bates/jupyter_server that referenced this pull request Dec 8, 2019
kevin-bates pushed a commit to kevin-bates/jupyter_server that referenced this pull request Jan 6, 2020
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Jan 6, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Jan 6, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Jan 9, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods
- Fix display of connection count when culling kernel
- Stop ping_callback if stream or socket closed exceptions occur
- Protect against KeyError, raise 404 if kernel_id is not found

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Jan 10, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods
- Fix display of connection count when culling kernel
- Stop ping_callback if stream or socket closed exceptions occur
- Protect against KeyError, raise 404 if kernel_id is not found

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Jan 16, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods
- Fix display of connection count when culling kernel
- Stop ping_callback if stream or socket closed exceptions occur
- Protect against KeyError, raise 404 if kernel_id is not found

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
davidbrochart pushed a commit to davidbrochart/jupyter_server that referenced this pull request Mar 18, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods
- Fix display of connection count when culling kernel
- Stop ping_callback if stream or socket closed exceptions occur
- Protect against KeyError, raise 404 if kernel_id is not found

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
kevin-bates added a commit to kevin-bates/jupyter_server that referenced this pull request Apr 6, 2020
- Initial work towards using jupyter_kernel_mgmt in jupyter server
- Cherry-pick Notebook PR jupyter/notebook#4837
- Remove jupyter_client, use KernelFinder in Gateway
- Minor refactor to preserve MappingKernelManager behavior
- Session is no longer a Configurable, removed from classes list.
  Also removed some of the Gateway classes that shouldn't be there either.
- Get gateway functionality working
- Fix SessionHandler call to start kernel
- Initial support for async kernel management
- Plumb launch parameters
- Adjust kernel management with recent async updates
- Don't get child_watcher if on Windows
- Fix gateway kernelspec tests to updated JKM call
  Also fixed windows testing by increasing delay during cleanup of session
  and kernel tests - otherwise the temp directory could not be cleaned up,
  resulting in downstream side-affects.
- Require JKM >= 0.5, bump core min release
- Remove install of special patch branches for jkm
- Merge pytest PR, encode/decode kernel type
- Merge/convert missing sessions tests for JKM
- Add session and kernel equality methods
- Fix display of connection count when culling kernel
- Stop ping_callback if stream or socket closed exceptions occur
- Protect against KeyError, raise 404 if kernel_id is not found

Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>
@kevin-bates
Copy link
Member Author

This effort will take place in Jupyter Server at some point via jupyter-server/jupyter_server#112.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants