Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jupyterlab polls the API a bit too much #3929

Closed
1 of 2 tasks
minrk opened this issue Feb 21, 2018 · 1 comment
Closed
1 of 2 tasks

jupyterlab polls the API a bit too much #3929

minrk opened this issue Feb 21, 2018 · 1 comment

Comments

@minrk
Copy link
Contributor

@minrk minrk commented Feb 21, 2018

Hi folks!

First of all, congratulations on the JupyterLab announcement. Very cool stuff!

We (Binder team) are coming out the other side of the big hug of traffic sent our way by the announcement. After a while, JupyterHub fell down pretty hard due to very high load. This was unusual, because the total number of users was not especially high. However, we have noticed that JupyterLab seems to make rather a lot more API requests than the previous generation's notebook client, especially when idle. In particular, in the classic tree view, we are careful to stop our periodic updates to the file listing when the tab is out of focus. This avoids wasting network and server resources when nobody's looking. I think JupyterLab should strive to do the same. The second issue came when jupyterlab instances started being culled, but users might have left a tab open in the background. When this occurs, requests are going to start failing because the server isn't there anymore. However, we still are getting lots of failed requests as the requests are being handled by jupyterhub once the server is gone. This appears to have been the biggest source of problem-causing load: requests from jupyterlab clients that could never succeed, but kept retrying. If these requests had an exponential backoff on errors and eventually gave up, that would have behaved much better, and jupyterlab wouldn't waste resources making requests that can't succeed.

Action items, assuming our observations are accurate:

  • make sure that polling updates stop when jupyterlab tab is not focused
  • exponential backoff when poll requests are failing
@minrk
Copy link
Contributor Author

@minrk minrk commented Feb 21, 2018

For reference, we've identified JupyterLab as the cause of close to all of the 4XX responses in this diagnostic, where jupyterlab clients connected to servers that have been shutdown are collectively making ~1k requests/minute to servers that haven't been running for some time.

@saulshanabrook saulshanabrook added this to the Future milestone Sep 10, 2018
@jasongrout jasongrout removed this from the Future milestone Sep 12, 2018
@jasongrout jasongrout added this to the 1.0 milestone Sep 12, 2018
@afshin afshin self-assigned this Sep 21, 2018
@lock lock bot locked as resolved and limited conversation to collaborators Aug 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants