[KernelGatewayApp] Searching [u'/', '/root/.jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files [KernelGatewayApp] Looking for jupyter_config in /etc/jupyter [KernelGatewayApp] Looking for jupyter_config in /usr/local/etc/jupyter [KernelGatewayApp] Looking for jupyter_config in /usr/etc/jupyter [KernelGatewayApp] Looking for jupyter_config in /root/.jupyter [KernelGatewayApp] Looking for jupyter_config in / [KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /etc/jupyter [KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /usr/local/etc/jupyter [KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /usr/etc/jupyter [KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /root/.jupyter [KernelGatewayApp] Looking for jupyter_kernel_gateway_config in / [KernelGatewayApp] Jupyter Kernel Gateway at http://0.0.0.0:8889 [KernelGatewayApp] Found kernel kubeflow in /usr/share/jupyter/kernels [KernelGatewayApp] Found kernel python2 in /usr/share/jupyter/kernels [I 190718 03:45:58 web:2162] 200 GET /api/kernelspecs (172.17.0.1) 166.34ms [KernelGatewayApp] Found kernel kubeflow in /usr/share/jupyter/kernels [KernelGatewayApp] Found kernel python2 in /usr/share/jupyter/kernels [I 190718 03:46:07 web:2162] 200 GET /api/kernelspecs (172.17.0.1) 1.67ms [KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-7563ec03-5d46-4444-b8b8-b810d18f7847.json'] [KernelGatewayApp] Connecting to: tcp://127.0.0.1:50547 [KernelGatewayApp] Connecting to: tcp://127.0.0.1:49545 [KernelGatewayApp] Kernel started: 7563ec03-5d46-4444-b8b8-b810d18f7847 [KernelGatewayApp] Kernel args: {'kernel_name': u'kubeflow', 'env': {'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'KERNEL_LAUNCH_TIMEOUT': u'40', u'KERNEL_WORKING_DIR': u'/home/eric/Downloads/ciao/docs/examples/pytorch'}} [I 190718 03:46:07 web:2162] 201 POST /api/kernels (172.17.0.1) 32.01ms [I 190718 03:46:07 web:2162] 200 GET /api/kernels/7563ec03-5d46-4444-b8b8-b810d18f7847 (172.17.0.1) 1.03ms 2019/07/18 03:46:07 Using config file: /etc/ciao/config.yaml 2019/07/18 03:46:07 Use the KubeConfig file ERROR: logging before flag.Parse: W0718 03:46:07.928684 11 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ERROR: logging before flag.Parse: W0718 03:46:07.928697 11 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined 2019/07/18 03:46:07 Error building kubeConfig: invalid configuration: no configuration has been provided [KernelGatewayApp] Initializing websocket connection /api/kernels/7563ec03-5d46-4444-b8b8-b810d18f7847/channels [KernelGatewayApp] WARNING | No session ID specified [KernelGatewayApp] Requesting kernel info from 7563ec03-5d46-4444-b8b8-b810d18f7847 [KernelGatewayApp] Connecting to: tcp://127.0.0.1:48605 [KernelGatewayApp] KernelRestarter: restarting kernel (1/5), new random ports [KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-7563ec03-5d46-4444-b8b8-b810d18f7847.json'] [KernelGatewayApp] Connecting to: tcp://127.0.0.1:55135 2019/07/18 03:46:10 Using config file: /etc/ciao/config.yaml 2019/07/18 03:46:10 Use the KubeConfig file ERROR: logging before flag.Parse: W0718 03:46:10.913237 22 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ERROR: logging before flag.Parse: W0718 03:46:10.913261 22 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined 2019/07/18 03:46:10 Error building kubeConfig: invalid configuration: no configuration has been provided [KernelGatewayApp] KernelRestarter: restarting kernel (2/5), new random ports [KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-7563ec03-5d46-4444-b8b8-b810d18f7847.json'] [KernelGatewayApp] Connecting to: tcp://127.0.0.1:49571 2019/07/18 03:46:13 Using config file: /etc/ciao/config.yaml 2019/07/18 03:46:13 Use the KubeConfig file ERROR: logging before flag.Parse: W0718 03:46:13.918582 30 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ERROR: logging before flag.Parse: W0718 03:46:13.918603 30 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined 2019/07/18 03:46:13 Error building kubeConfig: invalid configuration: no configuration has been provided [KernelGatewayApp] KernelRestarter: restarting kernel (3/5), new random ports [KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-7563ec03-5d46-4444-b8b8-b810d18f7847.json'] [KernelGatewayApp] Connecting to: tcp://127.0.0.1:53927 2019/07/18 03:46:16 Using config file: /etc/ciao/config.yaml 2019/07/18 03:46:16 Use the KubeConfig file ERROR: logging before flag.Parse: W0718 03:46:16.926361 38 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ERROR: logging before flag.Parse: W0718 03:46:16.926384 38 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined 2019/07/18 03:46:16 Error building kubeConfig: invalid configuration: no configuration has been provided [KernelGatewayApp] KernelRestarter: restarting kernel (4/5), new random ports [KernelGatewayApp] Starting kernel: [u'kubeflow-kernel', u'run', u'--connection-file', u'/root/.local/share/jupyter/runtime/kernel-7563ec03-5d46-4444-b8b8-b810d18f7847.json'] [KernelGatewayApp] Connecting to: tcp://127.0.0.1:34197 2019/07/18 03:46:19 Using config file: /etc/ciao/config.yaml 2019/07/18 03:46:19 Use the KubeConfig file ERROR: logging before flag.Parse: W0718 03:46:19.932601 46 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ERROR: logging before flag.Parse: W0718 03:46:19.932625 46 client_config.go:557] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined 2019/07/18 03:46:19 Error building kubeConfig: invalid configuration: no configuration has been provided [KernelGatewayApp] WARNING | KernelRestarter: restart failed [KernelGatewayApp] WARNING | Kernel 7563ec03-5d46-4444-b8b8-b810d18f7847 died, removing from map. [KernelGatewayApp] Websocket closed 7563ec03-5d46-4444-b8b8-b810d18f7847:dbdea7d9-20297e58365b0c8f9a5da4cc [KernelGatewayApp] WARNING | Timeout waiting for kernel_info reply from 7563ec03-5d46-4444-b8b8-b810d18f7847 [I 190718 03:47:07 web:2162] 101 GET /api/kernels/7563ec03-5d46-4444-b8b8-b810d18f7847/channels (172.17.0.1) 60017.48ms [KernelGatewayApp] Opening websocket /api/kernels/7563ec03-5d46-4444-b8b8-b810d18f7847/channels [KernelGatewayApp] Getting buffer for 7563ec03-5d46-4444-b8b8-b810d18f7847 [KernelGatewayApp] ERROR | Error opening stream: HTTP 404: Not Found (Kernel does not exist: 7563ec03-5d46-4444-b8b8-b810d18f7847) [E 190718 03:47:07 http1connection:744] Uncaught exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tornado/http1connection.py", line 735, in _server_request_loop ret = yield conn.read_response(request_delegate) File "/usr/lib/python2.7/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/usr/lib/python2.7/site-packages/tornado/concurrent.py", line 261, in result raise_exc_info(self._exc_info) File "/usr/lib/python2.7/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/usr/lib/python2.7/site-packages/tornado/http1connection.py", line 160, in _read_message max_bytes=self.params.max_header_size) File "/usr/lib/python2.7/site-packages/tornado/iostream.py", line 353, in read_until_regex future = self._set_read_callback(callback) File "/usr/lib/python2.7/site-packages/tornado/iostream.py", line 860, in _set_read_callback assert self._read_future is None, "Already reading" AssertionError: Already reading