Requests to /hub/user/<username>/api/* does not trigger spawn when server is down

[yuvipanda]

Me and @ryanlovett are investigating reports of launch errors from our hub users.

My understanding is that if a user's server is down, and they hit /hub/user//*, it should trigger respawn. This lets users close their laptops for the day, come back the next day, and sortof have their servers automagically start back up.

It's possible that my understanding is now out of date. But I see several places in logs where the following sequence happens:

I 2018-11-07 17:55:46.391 JupyterHub log:158] 302 GET /user/<user>/api/sessions?_=1541456675914 -> /hub/user/<user>/api/sessions?_=1541456675914 (@10.240.0.251) 0.73ms
[W 2018-11-07 17:55:46.608 JupyterHub log:158] 404 GET /hub/user/<user>/api/sessions?_=1541456675914 (<user>@10.240.0.251) 158.19ms

The second line makes me think the user is authenticated (since the log contains the username as additional info at the end), but respawn is not triggered. This leads to users seeing errors when they'd normally just see a respawn.

[yuvipanda]

With some more investigation, almost all the 404s are for API requests.

All 404s: 3176
API 404s: 3108
Contents API: 982
Terminals API: 1024
Sessions API: 1027

This seems high. In another high activity hub I run, there are 4 404s - to sessions and terminals API calls.

[minrk]

This is the result of #2001 because the implicit spawns were causing a problem ( a culled server with a left-open browser tab would respawn indefinitely, effectively making it impossible to cull idle servers). Note that any non-api request (e.g. opening a notebook, tree view, terminal, etc.) will trigger the spawn. It is only api requests that are not allowed to trigger respawn.

So it seems like we have two concrete cases where the automatic respawn on API requests is both a problem if it happens and a problem if it doesn't. I'm not a fan of a config option for everything, but maybe there isn't another solution to two mutually exclusive use cases.

Another option could be to allow POST/PUT/DELETE requests to trigger a respawn, that way a tab left-open that keeps refreshing the API wouldn't do anything, but any request to take a clear action (e.g. save the notebook or restart the kernel) would trigger a respawn (I think the initial request would still fail). That might be a bit too much, though.

[yuvipanda]

@minrk another option is to add code to the jupyterhub-singleuser related code that does some JS that tells the user their server is dead and they should refresh. That seems much more explicit and smoother than autospawn from API requests.

[minrk]

Yup! In that case, it's probably mostly a jupyterlab;jupyter/notebook javascript PR to be made?

[yuvipanda]

@minrk yeah, I think so. We don't do any JS in jupyter notebook right now, I think. How do you think we should detect liveness? Add an API endpoint that returns true if it's running with no side effects? Maybe such an endpoint already exists.

[minrk]

There's an /api/status endpoint. That's usually what I recommend for checking if a server is running.

[manics]

Can this be closed?

[minrk]

Yes, I think this should have been closed by #2435 - implicit spawning has proved to be something we shouldn't be doing.