Failed to connect to Hub API at 'http://jupyter-0:8081/hub/api'. Is the Hub accessible at this URL (from host: jupyter-q)?

[ma2331550908]

• gcr.io/kubeflow-images-public/tensorflow-1.12.0-notebook-cpu: v0.4.0 images is ok；
• gcr.io/kubeflow-images-public/tensorflow-1.10.1-notebook-gpu: v0.4.0 has problems； but pod jupyer-0 and new pod(jupyter-q) is running;
  

kubectl logs jupyter-q -n kubeflow

checking if /home/jovyan volume needs init... .../home/jovyan already has content... ...done Execute the command [W 2019-01-30 12:36:28.120 SingleUserNotebookApp configurable:168] Config option open_browsernot recognized bySingleUserNotebookApp. Did you mean browser? [I 2019-01-30 12:36:28.316 SingleUserNotebookApp extension:53] JupyterLab beta preview extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab [I 2019-01-30 12:36:28.316 SingleUserNotebookApp extension:54] JupyterLab application directory is /opt/conda/share/jupyter/lab [I 2019-01-30 12:36:28.321 SingleUserNotebookApp singleuser:405] Starting jupyterhub-singleuser server version 0.9.1 [W 2019-01-30 12:36:39.990 SingleUserNotebookApp log:158] 404 GET / (@127.0.0.1) 37.82ms [W 2019-01-30 12:36:40.110 SingleUserNotebookApp log:158] 404 GET /hub/logo (@127.0.0.1) 1.83ms [I 2019-01-30 12:36:45.144 SingleUserNotebookApp log:158] 302 GET /user/q/tree -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-q&redirect_uri=%2Fuser%2Fq%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 2.99ms [W 2019-01-30 12:36:45.158 SingleUserNotebookApp log:158] 404 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-q&redirect_uri=%2Fuser%2Fq%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 2.11ms [W 2019-01-30 12:36:45.214 SingleUserNotebookApp log:158] 404 GET /hub/logo (@127.0.0.1) 1.79ms [W 2019-01-30 12:36:47.446 SingleUserNotebookApp auth:586] Detected unused OAuth state cookies [I 2019-01-30 12:36:47.447 SingleUserNotebookApp log:158] 302 GET /user/q/tree -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-q&redirect_uri=%2Fuser%2Fq%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 1.99ms [W 2019-01-30 12:36:47.461 SingleUserNotebookApp log:158] 404 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-q&redirect_uri=%2Fuser%2Fq%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 2.04ms [W 2019-01-30 12:36:47.514 SingleUserNotebookApp log:158] 404 GET /hub/logo (@127.0.0.1) 1.96ms [E 2019-01-30 12:36:48.323 SingleUserNotebookApp singleuser:388] Failed to connect to my Hub at http://jupyter-0:8081/hub/api (attempt 1/5). Is it running? Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 385, in check_hub_version resp = await client.fetch(self.hub_api_url) File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 315, in wrapped ret = fn(*args, **kwargs) File "/opt/conda/lib/python3.6/site-packages/tornado/simple_httpclient.py", line 271, in _on_timeout raise HTTPError(599, error_message) tornado.httpclient.HTTPError: HTTP 599: Timeout while connecting [E 2019-01-30 12:36:48.337 SingleUserNotebookApp ioloop:792] Exception in callback functools.partial(<function wrap.<locals>.wrapped at 0x7efc11039e18>, <Future finished exception=gaierror(-3, 'Temporary failure in name resolution')>) Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py", line 759, in _run_callback ret = callback() File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 344, in wrapped raise_exc_info(exc) File "<string>", line 4, in raise_exc_info File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 315, in wrapped ret = fn(*args, **kwargs) File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 293, in <lambda> future, lambda future: callback(future.result())) File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run yielded = self.gen.throw(*exc_info) File "/opt/conda/lib/python3.6/site-packages/tornado/tcpclient.py", line 226, in connect addrinfo = yield self.resolver.resolve(host, port, af) File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run value = future.result() File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run yielded = self.gen.throw(*exc_info) File "/opt/conda/lib/python3.6/site-packages/tornado/netutil.py", line 364, in resolve None, _resolve_addr, host, port, family) File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run value = future.result() File "/opt/conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/opt/conda/lib/python3.6/site-packages/tornado/netutil.py", line 349, in _resolve_addr addrinfo = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM) File "/opt/conda/lib/python3.6/socket.py", line 745, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution [W 2019-01-30 12:36:49.152 SingleUserNotebookApp log:158] 404 GET /hub/home (@127.0.0.1) 2.23ms [W 2019-01-30 12:36:49.210 SingleUserNotebookApp log:158] 404 GET /hub/logo (@127.0.0.1) 1.85ms [W 2019-01-30 12:36:50.454 SingleUserNotebookApp log:158] 404 GET /hub/home (@127.0.0.1) 2.17ms [W 2019-01-30 12:36:50.511 SingleUserNotebookApp log:158] 404 GET /hub/logo (@127.0.0.1) 1.80ms [E 2019-01-30 12:37:10.342 SingleUserNotebookApp singleuser:388] Failed to connect to my Hub at http://jupyter-0:8081/hub/api (attempt 2/5). Is it running?

kubectl logs jupyter-0 -n kubeflow

[I 2019-01-30 12:30:18.054 JupyterHub base:498] User logged in: q [I 2019-01-30 12:30:18.055 JupyterHub log:158] 302 POST /hub/login?next= -> /user/q/ (q@10.244.0.22) 9.29ms [I 2019-01-30 12:30:18.073 JupyterHub log:158] 302 GET /user/q/ -> /hub/user/q/ (@10.244.0.22) 0.81ms [I 2019-01-30 12:30:18.092 JupyterHub log:158] 302 GET /hub/user/q/ -> /hub/spawn?next=%2Fhub%2Fuser%2Fq%2F (q@10.244.0.22) 6.94ms [I 2019-01-30 12:30:18.182 JupyterHub log:158] 200 GET /hub/spawn?next=%2Fhub%2Fuser%2Fq%2F (q@10.244.0.22) 76.80ms 12:33:26.169 - info: [ConfigProxy] 200 GET /api/routes [I 2019-01-30 12:33:26.170 JupyterHub proxy:299] Checking routes [I 2019-01-30 12:33:54.201 JupyterHub spawner:460] PVC q-workspace was successfully retrieved [I 2019-01-30 12:33:59.723 JupyterHub base:627] User q took 5.567 seconds to start [I 2019-01-30 12:33:59.724 JupyterHub proxy:240] Adding user q to proxy /user/q/ => http://10.244.1.93:8888 12:33:59.725 - info: [ConfigProxy] Adding route /user/q -> http://10.244.1.93:8888 12:33:59.726 - info: [ConfigProxy] 201 POST /api/routes/user/q [I 2019-01-30 12:33:59.735 JupyterHub log:158] 302 POST /hub/spawn?next=%2Fhub%2Fuser%2Fq%2F -> /hub/user/q/ (q@10.244.0.22) 5587.01ms [I 2019-01-30 12:33:59.837 JupyterHub log:158] 302 GET /hub/user/q/ -> /user/q/?redirects=1 (q@10.244.0.22) 8.22ms [I 2019-01-30 12:33:59.901 JupyterHub log:158] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-q&redirect_uri=%2Fuser%2Fq%2Foauth_callback&response_type=code&state=[secret] -> /user/q/oauth_callback?code=[secret]&state=[secret] (q@10.244.0.22) 17.16ms 12:38:26.170 - info: [ConfigProxy] 200 GET /api/routes [I 2019-01-30 12:38:26.178 JupyterHub proxy:299] Checking routes

[ma2331550908]

ENV

• two Physical Server
• KUBEFLOW_TAG=v0.4.1
• k8s 1.13

[jlewi]

@ma2331550908 What are you trying to do? Were you able to deploy notebooks successfully?

How are you doing auth?

[ma2331550908]

@ma2331550908 What are you trying to do? Were you able to deploy notebooks successfully?

How are you doing auth?

Browser changed to Firefox is ok, the reason has been not found yet.