Scheduler web interface fails with TypeError with bokeh 0.12.10

[jeffreyliu]

Installing the latest version of bokeh (0.12.10) breaks the bokeh web interface when launching the scheduler, throwing TypeError: __init__() got an unexpected keyword argument 'host'. The scheduler itself runs fine despite this. I am not setting host in either .dask/config.yaml nor in the command line arguments. I was able to replicate on both MacOS Sierra 10.12.6 and Ubuntu 14.04.

using bokeh=0.12.10:

~ jeffliu$ dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Could not launch service: ('bokeh', 8787)
Traceback (most recent call last):
  File "/Users/jeffliu/miniconda3/lib/python3.6/site-packages/distributed/scheduler.py", line 430, in start_services
    service.listen((listen_ip, port))
  File "/Users/jeffliu/miniconda3/lib/python3.6/site-packages/distributed/bokeh/core.py", line 31, in listen
    **kwargs)
  File "/Users/jeffliu/miniconda3/lib/python3.6/site-packages/bokeh/server/server.py", line 371, in __init__
    tornado_app = BokehTornado(applications, extra_websocket_origins=extra_websocket_origins, prefix=self.prefix, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'host'
distributed.scheduler - INFO -   Scheduler at:   tls://ip.address:8786
distributed.scheduler - INFO -        http at:                     :9786
distributed.scheduler - INFO - Local Directory: /var/folders/path
distributed.scheduler - INFO - -----------------------------------------------

Fixed when downgrading to bokeh 0.12.7:

lumiere:~ jeffliu$ conda install bokeh=0.12.7
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /Users/jeffliu/miniconda3:

The following packages will be DOWNGRADED:

    bokeh:      0.12.10-py36hfd5be35_0 --> 0.12.7-py36hf2e3fb6_1

Proceed ([y]/n)? y
bokeh-0.12.7-p 100% |######################################################################| Time: 0:00:01   2.95 MB/s

~ jeffliu$ dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO -   Scheduler at:   tls://ip.address:8786
distributed.scheduler - INFO -        http at:                     :9786
distributed.scheduler - INFO -       bokeh at:                     :8787
distributed.scheduler - INFO - Local Directory: /var/folders/path
distributed.scheduler - INFO - -----------------------------------------------

[mrocklin]

Known issue, fixed in latest release:

conda install -c conda-forge distributed

Thanks for reporting

[jruizaranguren]

I am reproducing this issue, no matter version of bokeh or distributed.

cluster = LocalCluster(processes=False)

Environment:

• Ubuntu 16.04
• dask, dask-core: 0.16.1
• distributed: 1.20.2 (conda and conda-forge version tested).
• bokeh: 0.12.7, 0.12.10 and 0.12.13 tested.

Error:

        try:
>           tornado_app = BokehTornado(applications, extra_websocket_origins=extra_websocket_origins, prefix=self.prefix, **kwargs)
E           TypeError: __init__() got an unexpected keyword argument 'host'

../../anaconda2/lib/python2.7/site-packages/bokeh/server/server.py:371: TypeError

StackTrace:

../../anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py:117: in __init__
    sync(self.loop, self._start, ip)
../../anaconda2/lib/python2.7/site-packages/distributed/utils.py:241: in sync
    six.reraise(*error[0])
../../anaconda2/lib/python2.7/site-packages/distributed/utils.py:229: in f
    result[0] = yield make_coro()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:870: in run
    value = future.result()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/concurrent.py:215: in result
    raise_exc_info(self._exc_info)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:876: in run
    yielded = self.gen.throw(*exc_info)
../../anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py:149: in _start
    services=self.worker_services, **self.worker_kwargs)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:870: in run
    value = future.result()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/concurrent.py:215: in result
    raise_exc_info(self._exc_info)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:876: in run
    yielded = self.gen.throw(*exc_info)
../../anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py:155: in _start_all_workers
    yield [self._start_worker(**kwargs) for i in range(n_workers)]
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:870: in run
    value = future.result()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/concurrent.py:215: in result
    raise_exc_info(self._exc_info)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:656: in callback
    result_list.append(f.result())
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/concurrent.py:215: in result
    raise_exc_info(self._exc_info)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:876: in run
    yielded = self.gen.throw(*exc_info)
../../anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py:174: in _start_worker
    yield w._start()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:870: in run
    value = future.result()
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/concurrent.py:215: in result
    raise_exc_info(self._exc_info)
../../anaconda2/envs/datatube/lib/python2.7/site-packages/tornado/gen.py:230: in wrapper
    yielded = next(result)
../../anaconda2/lib/python2.7/site-packages/distributed/worker.py:330: in _start
    self.start_services(listen_host)
../../anaconda2/lib/python2.7/site-packages/distributed/worker.py:293: in start_services
    self.services[k].listen((listen_ip, port))
../../anaconda2/lib/python2.7/site-packages/distributed/bokeh/core.py:31: in listen
    **kwargs)

[mrocklin]

Unfortunately I'm not able to reproduce

mrocklin@carbon:~$ conda create -n test-1515 python=2.7 dask dask-core=0.16.1 distributed=1.20.2 bokeh=0.12.10
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/mrocklin/Software/anaconda/envs/test-1515:

The following NEW packages will be INSTALLED:

    backports:          1.0-py27h63c9359_1    
    backports_abc:      0.5-py27h7b3c97b_0    
    bokeh:              0.12.10-py27he46cc6b_0
    ca-certificates:    2017.08.26-h1d4fec5_0 
    certifi:            2018.1.18-py27_0      
    click:              6.7-py27h4225b90_0    
    cloudpickle:        0.5.2-py27_1          
    dask:               0.16.1-py27_0         
    dask-core:          0.16.1-py27_0         
    distributed:        1.20.2-py27_0         
    futures:            3.2.0-py27h7b459c0_0  
    heapdict:           1.0.0-py27h33770af_0  
    intel-openmp:       2018.0.0-hc7b2577_8   
    jinja2:             2.10-py27h4114e70_0   
    libedit:            3.1-heed3624_0        
    libffi:             3.2.1-hd88cf55_4      
    libgcc-ng:          7.2.0-h7cc24e2_2      
    libgfortran-ng:     7.2.0-h9f7466a_2      
    libstdcxx-ng:       7.2.0-h7a57d05_2      
    locket:             0.2.0-py27h73929a2_1  
    markupsafe:         1.0-py27h97b2822_1    
    mkl:                2018.0.1-h19d6760_4   
    msgpack-python:     0.5.1-py27h6bb024c_0  
    ncurses:            6.0-h9df7e31_2        
    numpy:              1.14.0-py27h3dfced4_1 
    openssl:            1.0.2n-hb7f436b_0     
    pandas:             0.22.0-py27hf484d3e_0 
    partd:              0.3.8-py27h4e55004_0  
    pip:                9.0.1-py27ha730c48_4  
    psutil:             5.4.3-py27h14c3975_0  
    python:             2.7.14-h1571d57_29    
    python-dateutil:    2.6.1-py27h4ca5741_1  
    pytz:               2017.3-py27h001bace_0 
    pyyaml:             3.12-py27h2d70dd7_1   
    readline:           7.0-ha6073c6_4        
    setuptools:         38.4.0-py27_0         
    singledispatch:     3.4.0.3-py27h9bcb476_0
    six:                1.11.0-py27h5f960f1_1 
    sortedcontainers:   1.5.9-py27_0          
    sqlite:             3.22.0-h1bed415_0     
    ssl_match_hostname: 3.5.0.1-py27h4ec10b9_2
    tblib:              1.3.2-py27h51fe5ba_0  
    tk:                 8.6.7-hc745277_3      
    toolz:              0.9.0-py27_0          
    tornado:            4.5.3-py27_0          
    wheel:              0.30.0-py27h2bc6bb2_1 
    yaml:               0.1.7-had09818_2      
    zict:               0.1.3-py27h12c336c_0  
    zlib:               1.2.11-ha838bed_2     

yaml-0.1.7-had 100% |################################| Time: 0:00:00   2.33 MB/s
sqlite-3.22.0- 100% |################################| Time: 0:00:00   8.51 MB/s
python-2.7.14- 100% |################################| Time: 0:00:01   8.64 MB/s
backports-1.0- 100% |################################| Time: 0:00:00   3.76 MB/s
backports_abc- 100% |################################| Time: 0:00:00  11.15 MB/s
certifi-2018.1 100% |################################| Time: 0:00:00   9.27 MB/s
click-6.7-py27 100% |################################| Time: 0:00:00   9.83 MB/s
cloudpickle-0. 100% |################################| Time: 0:00:00   8.54 MB/s
dask-core-0.16 100% |################################| Time: 0:00:00   8.89 MB/s
futures-3.2.0- 100% |################################| Time: 0:00:00   8.67 MB/s
heapdict-1.0.0 100% |################################| Time: 0:00:00   5.26 MB/s
locket-0.2.0-p 100% |################################| Time: 0:00:00   7.10 MB/s
markupsafe-1.0 100% |################################| Time: 0:00:00  14.27 MB/s
msgpack-python 100% |################################| Time: 0:00:00   9.91 MB/s
numpy-1.14.0-p 100% |################################| Time: 0:00:00   8.69 MB/s
psutil-5.4.3-p 100% |################################| Time: 0:00:00   9.15 MB/s
pytz-2017.3-py 100% |################################| Time: 0:00:00   9.61 MB/s
pyyaml-3.12-py 100% |################################| Time: 0:00:00   9.32 MB/s
six-1.11.0-py2 100% |################################| Time: 0:00:00  23.98 MB/s
sortedcontaine 100% |################################| Time: 0:00:00  14.35 MB/s
tblib-1.3.2-py 100% |################################| Time: 0:00:00  12.13 MB/s
toolz-0.9.0-py 100% |################################| Time: 0:00:00  10.67 MB/s
partd-0.3.8-py 100% |################################| Time: 0:00:00  15.60 MB/s
python-dateuti 100% |################################| Time: 0:00:00   9.08 MB/s
setuptools-38. 100% |################################| Time: 0:00:00   9.11 MB/s
singledispatch 100% |################################| Time: 0:00:00  22.22 MB/s
ssl_match_host 100% |################################| Time: 0:00:00  10.13 MB/s
zict-0.1.3-py2 100% |################################| Time: 0:00:00  16.35 MB/s
jinja2-2.10-py 100% |################################| Time: 0:00:00   9.85 MB/s
pandas-0.22.0- 100% |################################| Time: 0:00:01   8.65 MB/s
tornado-4.5.3- 100% |################################| Time: 0:00:00   8.85 MB/s
wheel-0.30.0-p 100% |################################| Time: 0:00:00  11.50 MB/s
bokeh-0.12.10- 100% |################################| Time: 0:00:00   8.71 MB/s
distributed-1. 100% |################################| Time: 0:00:00   8.92 MB/s
pip-9.0.1-py27 100% |################################| Time: 0:00:00   8.72 MB/s
dask-0.16.1-py 100% |################################| Time: 0:00:00   4.90 MB/s
#
# To activate this environment, use:
# > source activate test-1515
#
# To deactivate an active environment, use:
# > source deactivate
#

mrocklin@carbon:~$ source activate test-1515
(test-1515) mrocklin@carbon:~$ python
Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask.distributed import LocalCluster
>>> cluster = LocalCluster(processes=False)
>>> 

[jruizaranguren]

Yes, it is unfortunate. I have created a new environment exactly as you, and I get the same exception.
Creating a new LocalCluster() without parameters also fails. I add the Traceback for this case:

Traceback (most recent call last):
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 174, in _start_worker
    yield w._start()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/nanny.py", line 157, in _start
    assert self.worker_address
AssertionError
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 174, in _start_worker
    yield w._start()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/nanny.py", line 157, in _start
    assert self.worker_address
AssertionError
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 174, in _start_worker
    yield w._start()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/nanny.py", line 157, in _start
    assert self.worker_address
AssertionError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 117, in __init__
    sync(self.loop, self._start, ip)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/utils.py", line 241, in sync
    six.reraise(*error[0])
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/utils.py", line 229, in f
    result[0] = yield make_coro()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 149, in _start
    services=self.worker_services, **self.worker_kwargs)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 155, in _start_all_workers
    yield [self._start_worker(**kwargs) for i in range(n_workers)]
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/deploy/local.py", line 174, in _start_worker
    yield w._start()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/javier/anaconda2/envs/test-1515/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/home/javier/anaconda2/lib/python2.7/site-packages/distributed/nanny.py", line 157, in _start
    assert self.worker_address
AssertionError

[jruizaranguren]

May I provide any additional information?.

We have tested this problem in 4 machines. In 3 of them, using Ubuntu 16.04, it is failing. It is working in remaining one that has Ubuntu 14.04. I do not know if it is the root cause, but it is the only difference we detect now.

[mrocklin]

I recommend constructing the minimal software environment for which this fails, either with conda or pip.

If this were to behave differently on ubuntu 16.04 vs 14.04 then my guess is that you have environment issues leaking in from your system python install, which should be entirely isolated.

[kudlatygosc]

Have same issue. I'm running anaconda on windows 10.
Using conda install bokeh=0.12.7 fixed the issue.
Thanks for solution.

I have bokeh 0.12.14 and upgraded the distributed package which resolved this issue. Note: several other packages updated as well..
pip install -U distributed