JupyterHub spawner: upstream request timeout

[jlewi]

When I go to JupyterHub and spawn my notebook I get

upstream request timeout

The jupyter pod is actually in the container creating state. Pulling the image take a long time because it is so large.

If I refresh it shows up.

My guess is there is some sort of timeout in JupyterHub and maybe we just need to increase it? Perhaps its a setting in KubeSpawner?

/cc @yuvipanda

[jlewi]

We already set a large timeout of 10 minutes here
https://github.com/kubeflow/kubeflow/blob/master/kubeflow/core/jupyterhub_spawner.py#L81

[lluunn]

I tried to reproduce:

In my own cluster, create new ksonnet app (ks init) and deploy the kubeflow-core.
After clicking the "Spawn", it took a long time (keep spinning for ~5min).
Then it worked, didn't see "upstream request timeout".

Is this fixed? Is the 5min wait expected?

[jlewi]

The 5 minute timeout is probably do to the image pull timeout.

How were you connecting to the cluster?

I've observed this upstream request timeout when using IAP on GKE. So one possibility is that its a timeout in the GCE loadbalancer, or envoy. If we observe it when using IAP but not when using kubectl port-forward to connect directly to JupyterHub that would indicate the problem is upstream of JupyterHub.

[lluunn]

Ah, I didn't enable IAP

[ankushagarwal]

I can take a look at this. I think this is coming from envoy as well.

[ankushagarwal]

The timeout specified in envoy config is 10 seconds for jupyterhub https://github.com/kubeflow/kubeflow/blob/master/kubeflow/core/iap.libsonnet#L397

I'll try increasing this to see if it helps