New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cull idle kernels #2215

Merged
merged 4 commits into from Apr 5, 2017

Conversation

Projects
None yet
7 participants
@kevin-bates
Contributor

kevin-bates commented Feb 22, 2017

(Updated to include review recommendations below...)

This change introduces the ability to cull kernels that have been idle for a specified amount of time. Off by default, it is enabled by setting --MappingKernelManager.cull_idle_timeout to a positive value representing the number of seconds a kernel must remain inactive to be culled. This feature leverages activity tracking's last_activity value to determine when the last activity for a given kernel occurred. Typical values for cull_idle_timeout will be on the order of 43200 (12 hours) or 86400 (24 hours) seconds as described in this JKG issue. These changes originally targeted Jupyter Kernel Gateway per issue #226 but really belongs in the Notebook server.

An optional property has been introduced to adjust the interval in which idle kernels are culled provided they have exceeded the aforementioned timeout property. This property is adjusted by setting --MappingKernelManager.cull_interval to a positive value. If not set or non-positive, the default value of 300 (5 minutes) seconds will be used.

JKG Issue 227 has been opened to remove the activity tracking that was also in place in the JKG. Prior to that, there could be false reports of active kernels that have been culled by the underlying MappingKernelManager. If not removed, we should minimally adjust the JKG's activity tracking handler to consult the active kernel manager class on each request.

@minrk

Thanks for the PR! I left a few comments inline. Let me know if you'd like any help finishing it up, or clarifying things.

You can push new commits to this same branch and it will update the PR.

@Carreau

+1 I think many people will be happy to see this ! It deserved a good note in what's new.

A couple of suggestion, but it's look ok as is for 5.1

if self.cull_interval <= 0: #handle case where user set invalid value
self.log.warning("Invalid value for 'cull_interval' detected (%s) - using default value (%s).",
self.cull_interval, self.cull_interval_default)
self.cull_interval = self.cull_interval_default

This comment has been minimized.

@Carreau

Carreau Feb 23, 2017

Member

Should we also warn for cull_interval < 60s ? it's technically not wrong, but it's likely that user might have thoughs it was minutes. Even anything below 300s seem it can be a mistake.

@Carreau

Carreau Feb 23, 2017

Member

Should we also warn for cull_interval < 60s ? it's technically not wrong, but it's likely that user might have thoughs it was minutes. Even anything below 300s seem it can be a mistake.

This comment has been minimized.

@kevin-bates

kevin-bates Feb 23, 2017

Contributor

Yeah, I agree. I thought implementing a minimum timeout would be good (now that I've tested it :-)). 300 strikes me as good minimum. If folks are making changes in these areas and need shorter timeouts for testing, they can temporarily adjust the constant.

Positive values less than the minimum would result in a warning and auto-adjustment to the minimum (similar to what happens if the interval is <= 0 and culling is enabled).

So, to your comment above, the description would be...

"Timeout (in seconds) after which a kernel is considered idle and ready to be culled. Values of 0 or lower disable culling. The minimum timeout is 300 seconds. Positive values less than the minimum will be adjusted to the minimum."

(or is that too verbose?)

@kevin-bates

kevin-bates Feb 23, 2017

Contributor

Yeah, I agree. I thought implementing a minimum timeout would be good (now that I've tested it :-)). 300 strikes me as good minimum. If folks are making changes in these areas and need shorter timeouts for testing, they can temporarily adjust the constant.

Positive values less than the minimum would result in a warning and auto-adjustment to the minimum (similar to what happens if the interval is <= 0 and culling is enabled).

So, to your comment above, the description would be...

"Timeout (in seconds) after which a kernel is considered idle and ready to be culled. Values of 0 or lower disable culling. The minimum timeout is 300 seconds. Positive values less than the minimum will be adjusted to the minimum."

(or is that too verbose?)

This comment has been minimized.

@minrk

minrk Feb 24, 2017

Member

@Carreau a warning is a good idea.

@kevin-bates that sounds perfect. You might add (5 minutes) next to 300 seconds.

@minrk

minrk Feb 24, 2017

Member

@Carreau a warning is a good idea.

@kevin-bates that sounds perfect. You might add (5 minutes) next to 300 seconds.

This comment has been minimized.

@kevin-bates

kevin-bates Feb 24, 2017

Contributor

Will do.

@kevin-bates

kevin-bates Feb 24, 2017

Contributor

Will do.

Show outdated Hide outdated notebook/services/kernels/kernelmanager.py
Show outdated Hide outdated notebook/services/kernels/kernelmanager.py
@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Feb 26, 2017

Member

@kevin-bates excellent, thanks! We're in the process of finishing up the 5.0 release from the master branch, but this can be merged once that release has been pushed out.

Member

minrk commented Feb 26, 2017

@kevin-bates excellent, thanks! We're in the process of finishing up the 5.0 release from the master branch, but this can be merged once that release has been pushed out.

@minrk

minrk approved these changes Feb 26, 2017

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 5, 2017

Member

5.0 is out, long live 5.1!

Member

minrk commented Apr 5, 2017

5.0 is out, long live 5.1!

@minrk minrk merged commit 8d6460a into jupyter:master Apr 5, 2017

4 checks passed

codecov/patch 38.09% of diff hit (target 0%)
Details
codecov/project 76.72% (-0.22%) compared to 9dd3818
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@kevin-bates kevin-bates deleted the kevin-bates:cull-idle-kernels branch Apr 5, 2017

@parente

This comment has been minimized.

Show comment
Hide comment
@parente

parente May 17, 2017

Member

After finally getting a test bed setup for this change, I realize that it does not take into account whether the kernel is busy or not, or whether there are connections to the kernel or not. I'm going to submit a follow-on PR that adds settings for whether these attributes should affect the culling decision or not.

Member

parente commented May 17, 2017

After finally getting a test bed setup for this change, I realize that it does not take into account whether the kernel is busy or not, or whether there are connections to the kernel or not. I'm going to submit a follow-on PR that adds settings for whether these attributes should affect the culling decision or not.

@bigquant

This comment has been minimized.

Show comment
Hide comment
@bigquant

bigquant Aug 3, 2017

when will 5.1 go release?

bigquant commented Aug 3, 2017

when will 5.1 go release?

@kevin-bates

This comment has been minimized.

Show comment
Hide comment
@kevin-bates

kevin-bates Aug 3, 2017

Contributor

@bigquant - Sounds like its very soon per the latest Jupyter Weekly Summary (2017, week 29): https://groups.google.com/forum/#!topic/jupyter/NgR4XpUf66E

Notebook (Grant)

    Grant is closing out outstanding issues marked with 5.1 milestone
    Planning on releasing a 5.1 release candidate by EOW
Contributor

kevin-bates commented Aug 3, 2017

@bigquant - Sounds like its very soon per the latest Jupyter Weekly Summary (2017, week 29): https://groups.google.com/forum/#!topic/jupyter/NgR4XpUf66E

Notebook (Grant)

    Grant is closing out outstanding issues marked with 5.1 milestone
    Planning on releasing a 5.1 release candidate by EOW
@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver Aug 3, 2017

Member

Yup, we're hoping to put the release candidate out any day now.

Member

takluyver commented Aug 3, 2017

Yup, we're hoping to put the release candidate out any day now.

@birdstar

This comment has been minimized.

Show comment
Hide comment
@birdstar

birdstar Sep 24, 2017

This is exactly the feature I wanted!

birdstar commented Sep 24, 2017

This is exactly the feature I wanted!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment