jupyter server: default image not working

[yanniszark]

How to reproduce:

docker run -it --entrypoint=/bin/bash gcr.io/kubeflow-images-public/tensorflow-1.10.1-notebook-cpu:v0.4.0
# Run the jupyter server
/opt/conda/bin/jupyter-notebook

Images tested:

• tensorflow-1.10.0-notebook-cpu
• tensorflow-1.11.0-notebook-cpu
• tensorflow-1.12.0-notebook-cpu

The server fails with:

Traceback (most recent call last):                                                                                                                                                                          
 File "/opt/conda/bin/jupyter-notebook", line 11, in <module>                                                                                                                                              
   sys.exit(main())                                                                                                                                                                                        
 File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance                                                                                                  
   return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)                                                                                                                                      
 File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance                                                                                              
   app.initialize(argv)                                                                                                                                                                                    
 File "<decorator-gen-7>", line 2, in initialize                                                                                                                                                          
 File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error                                                                                            
   return method(app, *args, **kwargs)                                                                                                                                                                    
 File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1602, in initialize                                                                                                          
   self.init_webapp()                                                                                                                                                                                      
 File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1381, in init_webapp
   self.http_server.listen(port, self.ip)
 File "/opt/conda/lib/python3.6/site-packages/tornado/tcpserver.py", line 143, in listen
   sockets = bind_sockets(port, address=address)
 File "/opt/conda/lib/python3.6/site-packages/tornado/netutil.py", line 168, in bind_sockets
   sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

The reason is because the /etc/hosts file is misconfigured and binding to localhost fails.
More details here: ipython/ipython#6193 (comment)
A quick solution would be to include the argument --ip=127.0.0.1

/opt/conda/bin/jupyter-notebook --ip=127.0.0.1

Related Issue: #3451

Ideally, we should change the default image to work without having to take extra steps.

/cc @jlewi @kimwnasptd
/priority p1

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.94. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

@yanniszark It looks like you are using the 0.4 docker image.

What is the misconfiguration in /etc/hosts? Are we modifying /etc/hosts in our Docker images?

[yanniszark]

@jlewi I tried with the 3 latest images and all had the same error.

• tensorflow-1.10.0-notebook-cpu
• tensorflow-1.11.0-notebook-cpu
• tensorflow-1.12.0-notebook-cpu

What is the misconfiguration in /etc/hosts?

We use localhost for both ipv4 and ipv6.
This line is wrong:

::1     localhost ip6-localhost ip6-loopback

It should be:

::1     ip6-localhost ip6-loopback

More details here: ipython/ipython#6193 (comment)

Are we modifying /etc/hosts in our Docker images?

Where are we building these images?

[yanniszark]

cc @jlewi

Update:
The reason it's not working is because the jupyter-notebook images are wrong.
Why are they wrong though?

These images are specified in a ConfigMap, used by jupyter-web-app.
That ConfigMap is out of date.
This PR by @kimwnasptd should fix it kubeflow/manifests#163

[yanniszark]

Fixed in kubeflow/manifests#163