Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serverextension #31

Merged
merged 52 commits into from
Dec 17, 2018
Merged

Serverextension #31

merged 52 commits into from
Dec 17, 2018

Conversation

ian-r-rose
Copy link
Collaborator

@ian-r-rose ian-r-rose commented Nov 1, 2018

There is more to do, but this is getting big enough as it is, so I'd prefer to do it in follow-up PRs. This creates a server extension for managing dask clusters. It allows you to list, start and stop them via a REST api. A UI for this is implemented in the side panel. You can also inject code to connect to any of the clusters via right-clicking on their list item

image

TODOS.

  • Add scaling functionality to the REST API.
  • Figure out tornado IOLoop isues (see discussion in Server extension to start cluster #30).
  • Add drag/drop for client connection code snippets. This will require some upstream JupyterLab enhancements.
  • Add a proxy to the server extension to get around CORS issues.
  • Figure out a reasonable config API on the server side for configuring what kind of cluster to start.

@mrocklin
Copy link
Member

mrocklin commented Nov 2, 2018

I've had some difficulty in setting this up. Help would be appreciated.

mrocklin@carbon:~/workspace/dask-labextension$ conda create -n dev jupyterlab nodejs dask bokeh ipython
Solving environment: done
	
## Package Plan ##

  environment location: /home/mrocklin/Software/anaconda/envs/dev

  added / updated specs: 
    - bokeh
    - dask
    - ipython
    - jupyterlab
    - nodejs


The following NEW packages will be INSTALLED:

    backcall:          0.1.0-py37_0           
    blas:              1.0-mkl                
    bleach:            3.0.2-py37_0           
    bokeh:             1.0.1-py37_0           
    ca-certificates:   2018.03.07-0           
    certifi:           2018.10.15-py37_0      
    click:             7.0-py37_0             
    cloudpickle:       0.6.1-py37_0           
    cytoolz:           0.9.0.1-py37h14c3975_1 
    dask:              0.20.0-py37_0          
    dask-core:         0.20.0-py37_0          
    decorator:         4.3.0-py37_0           
    distributed:       1.24.0-py37_0          
    entrypoints:       0.2.3-py37_2           
    freetype:          2.9.1-h8a8886c_1       
    gmp:               6.1.2-h6c8ec71_1       
    heapdict:          1.0.0-py37_2           
    intel-openmp:      2019.0-118             
    ipykernel:         5.1.0-py37h39e3cac_0   
    ipython:           7.1.1-py37h39e3cac_0   
    ipython_genutils:  0.2.0-py37_0           
    jedi:              0.13.1-py37_0          
    jinja2:            2.10-py37_0            
    jpeg:              9b-h024ee3a_2          
    jsonschema:        2.6.0-py37_0           
    jupyter_client:    5.2.3-py37_0           
    jupyter_core:      4.4.0-py37_0           
    jupyterlab:        0.35.3-py37_0          
    jupyterlab_server: 0.2.0-py37_0           
    libedit:           3.1.20170329-h6b74fdf_2
    libffi:            3.2.1-hd88cf55_4       
    libgcc-ng:         8.2.0-hdf63c60_1       
    libgfortran-ng:    7.3.0-hdf63c60_0       
    libpng:            1.6.35-hbc83047_0      
    libsodium:         1.0.16-h1bed415_0      
    libstdcxx-ng:      8.2.0-hdf63c60_1       
    libtiff:           4.0.9-he85c1e1_2       
    locket:            0.2.0-py37_1           
    markupsafe:        1.0-py37h14c3975_1     
    mistune:           0.8.4-py37h7b6447c_0   
    mkl:               2019.0-118             
    mkl_fft:           1.0.6-py37h7dd41cf_0   
    mkl_random:        1.0.1-py37h4414c95_1   
    msgpack-python:    0.5.6-py37h6bb024c_1   
    nbconvert:         5.3.1-py37_0           
    nbformat:          4.4.0-py37_0           
    ncurses:           6.1-hf484d3e_0         
    nodejs:            8.12.0-he6710b0_0      
    notebook:          5.7.0-py37_0           
    numpy:             1.15.3-py37h1d66e8a_0  
    numpy-base:        1.15.3-py37h81de0dd_0  
    olefile:           0.46-py37_0            
    openssl:           1.1.1-h7b6447c_0       
    packaging:         18.0-py37_0            
    pandas:            0.23.4-py37h04863e7_0  
    pandoc:            2.2.3.2-0              
    pandocfilters:     1.4.2-py37_1           
    parso:             0.3.1-py37_0           
    partd:             0.3.9-py37_0           
    pexpect:           4.6.0-py37_0           
    pickleshare:       0.7.5-py37_0           
    pillow:            5.3.0-py37h34e0f95_0   
    pip:               10.0.1-py37_0          
    prometheus_client: 0.4.2-py37_0           
    prompt_toolkit:    2.0.7-py37_0           
    psutil:            5.4.8-py37h7b6447c_0   
    ptyprocess:        0.6.0-py37_0           
    pygments:          2.2.0-py37_0           
    pyparsing:         2.2.2-py37_0           
    python:            3.7.1-h0371630_3       
    python-dateutil:   2.7.5-py37_0           
    pytz:              2018.7-py37_0          
    pyyaml:            3.13-py37h14c3975_0    
    pyzmq:             17.1.2-py37h14c3975_0  
    readline:          7.0-h7b6447c_5         
    send2trash:        1.5.0-py37_0           
    setuptools:        40.5.0-py37_0          
    six:               1.11.0-py37_1          
    sortedcontainers:  2.0.5-py37_0           
    sqlite:            3.25.2-h7b6447c_0      
    tblib:             1.3.2-py37_0           
    terminado:         0.8.1-py37_1           
    testpath:          0.4.2-py37_0           
    tk:                8.6.8-hbc83047_0       
    toolz:             0.9.0-py37_0           
    tornado:           5.1.1-py37h7b6447c_0   
    traitlets:         4.3.2-py37_0           
    wcwidth:           0.1.7-py37_0           
    webencodings:      0.5.1-py37_1           
    wheel:             0.32.2-py37_0          
    xz:                5.2.4-h14c3975_4       
    yaml:              0.1.7-had09818_2       
    zeromq:            4.2.5-hf484d3e_1       
    zict:              0.1.3-py37_0           
    zlib:              1.2.11-ha838bed_2      

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate dev
#
# To deactivate an active environment, use
#
#     $ conda deactivate

mrocklin@carbon:~/workspace/dask-labextension$ conda activate dev
mrocklin@carbon:~/workspace/dask-labextension$ git branch
  canavandl/individual_routes
  canavandl/jupyterlab_0.7.0_compat
  license
  master
  mega-update
  release-0.1.0
* serverextension
mrocklin@carbon:~/workspace/dask-labextension$ git pull ian-r-rose serverextension 
From github.com:ian-r-rose/dask-labextension
 * branch            serverextension -> FETCH_HEAD
Already up to date.
mrocklin@carbon:~/workspace/dask-labextension$ pip install -e -q . 
-q should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
mrocklin@carbon:~/workspace/dask-labextension$ pip install -e . 
Obtaining file:///home/mrocklin/workspace/dask-labextension
Requirement already satisfied: notebook in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from dask-labextension==0.1.0) (5.7.0)
Requirement already satisfied: traitlets>=4.2.1 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (4.3.2)
Requirement already satisfied: ipython-genutils in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (0.2.0)
Requirement already satisfied: jupyter-core>=4.4.0 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (4.4.0)
Requirement already satisfied: tornado>=4 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (5.1.1)
Requirement already satisfied: pyzmq>=17 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (17.1.2)
Requirement already satisfied: prometheus-client in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (0.4.2)
Requirement already satisfied: Send2Trash in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (1.5.0)
Requirement already satisfied: jinja2 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (2.10)
Requirement already satisfied: terminado>=0.8.1 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (0.8.1)
Requirement already satisfied: ipykernel in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (5.1.0)
Requirement already satisfied: nbformat in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (4.4.0)
Requirement already satisfied: nbconvert in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (5.3.1)
Requirement already satisfied: jupyter-client>=5.2.0 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from notebook->dask-labextension==0.1.0) (5.2.3)
Requirement already satisfied: decorator in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from traitlets>=4.2.1->notebook->dask-labextension==0.1.0) (4.3.0)
Requirement already satisfied: six in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from traitlets>=4.2.1->notebook->dask-labextension==0.1.0) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from jinja2->notebook->dask-labextension==0.1.0) (1.0)
Requirement already satisfied: ipython>=5.0.0 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipykernel->notebook->dask-labextension==0.1.0) (7.1.1)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbformat->notebook->dask-labextension==0.1.0) (2.6.0)
Requirement already satisfied: bleach in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (3.0.2)
Requirement already satisfied: testpath in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (0.4.2)
Requirement already satisfied: pandocfilters>=1.4.1 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (1.4.2)
Requirement already satisfied: pygments in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (2.2.0)
Requirement already satisfied: entrypoints>=0.2.2 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (0.2.3)
Requirement already satisfied: mistune>=0.7.4 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from nbconvert->notebook->dask-labextension==0.1.0) (0.8.4)
Requirement already satisfied: python-dateutil>=2.1 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from jupyter-client>=5.2.0->notebook->dask-labextension==0.1.0) (2.7.5)
Requirement already satisfied: pickleshare in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.7.5)
Requirement already satisfied: backcall in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.1.0)
Requirement already satisfied: jedi>=0.10 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.13.1)
Requirement already satisfied: pexpect; sys_platform != "win32" in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (4.6.0)
Requirement already satisfied: setuptools>=18.5 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (40.5.0)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (2.0.7)
Requirement already satisfied: webencodings in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from bleach->nbconvert->notebook->dask-labextension==0.1.0) (0.5.1)
Requirement already satisfied: parso>=0.3.0 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.3.1)
Requirement already satisfied: ptyprocess>=0.5 in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from pexpect; sys_platform != "win32"->ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.6.0)
Requirement already satisfied: wcwidth in /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->notebook->dask-labextension==0.1.0) (0.1.7)
mkl-random 1.0.1 requires cython, which is not installed.
Installing collected packages: dask-labextension
  Running setup.py develop for dask-labextension
Successfully installed dask-labextension
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
mrocklin@carbon:~/workspace/dask-labextension$ jupyter serverextension enable --sys-prefix dask_labextension
Enabling: dask_labextension
- Writing config: /home/mrocklin/Software/anaconda/envs/dev/etc/jupyter
    - Validating...
      dask_labextension 0.1.0 OK
mrocklin@carbon:~/workspace/dask-labextension$ jupyter lab
[I 06:16:31.377 LabApp] JupyterLab extension loaded from /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages/jupyterlab
[I 06:16:31.377 LabApp] JupyterLab application directory is /home/mrocklin/Software/anaconda/envs/dev/share/jupyter/lab
[W 06:16:31.378 LabApp] JupyterLab server extension not enabled, manually loading...
[I 06:16:31.380 LabApp] JupyterLab extension loaded from /home/mrocklin/Software/anaconda/envs/dev/lib/python3.7/site-packages/jupyterlab
[I 06:16:31.380 LabApp] JupyterLab application directory is /home/mrocklin/Software/anaconda/envs/dev/share/jupyter/lab
[I 06:16:31.381 LabApp] Serving notebooks from local directory: /home/mrocklin/workspace/dask-labextension
[I 06:16:31.381 LabApp] The Jupyter Notebook is running at:
[I 06:16:31.381 LabApp] http://localhost:8888/
[I 06:16:31.381 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 06:16:33.527 LabApp] Node v8.12.0
    
[I 06:16:33.530 LabApp] Build is up to date

image

@ian-r-rose
Copy link
Collaborator Author

As far as I can tell, you have not installed the client-side part of the extension. From the root of the repository

jlpm install
jlpm build
jupyter labextension install .

With a server extension the install story for users is going to be a bit more complicated (though we can talk through some steps to mitigate that), so we had better make it worth while :)

@mrocklin
Copy link
Member

mrocklin commented Nov 3, 2018

OK, I run the following:

jlpm install
jlpm build
jupyter labextension install .
pip install -e .
jupyter serverextension enable --sys-prefix dask_labextension

Things show up nicely in JLab, but pressing the "new cluster" button results in a 404

[W 13:30:52.231 LabApp] 404 PUT /dask?1541266252211 (127.0.0.1) 7.58ms referer=http://localhost:8888/lab
[W 13:30:52.379 LabApp] 404 PUT /dask?1541266252359 (127.0.0.1) 9.07ms referer=http://localhost:8888/lab

@ian-r-rose
Copy link
Collaborator Author

Hmm. Does jupyter serverextension list indicate that the server side component is installed and importable?

@ian-r-rose
Copy link
Collaborator Author

When I enter jupyter serverextension list:

config dir: /home/ian/anaconda3/envs/jupyter/etc/jupyter
    nbresuse  enabled 
    - Validating...
      nbresuse  OK
    jupyterlab  enabled 
    - Validating...
      jupyterlab 1.0.0a0 OK
    jupyterlab_latex  enabled 
    - Validating...
      jupyterlab_latex 0.4.1 OK
    jupyterlab_github  enabled 
    - Validating...
      jupyterlab_github 0.7.0 OK
    dask_labextension  enabled 
    - Validating...
      dask_labextension 0.1.0 OK

@mrocklin
Copy link
Member

mrocklin commented Nov 3, 2018

mrocklin@carbon:~/workspace/dask-labextension$ jupyter serverextension enable --sys-prefix dask_labextension
Enabling: dask_labextension
- Writing config: /home/mrocklin/Software/anaconda/etc/jupyter
    - Validating...
      dask_labextension  OK
mrocklin@carbon:~/workspace/dask-labextension$ jupyter serverextension list
config dir: /home/mrocklin/.jupyter
    dask_labextension disabled
    - Validating...
      dask_labextension  OK
config dir: /home/mrocklin/Software/anaconda/etc/jupyter
    jupyterlab  enabled 
    - Validating...
      jupyterlab  OK
    nbdime  enabled 
    - Validating...
      nbdime  OK
    nbserverproxy  enabled 
    - Validating...
      nbserverproxy  OK
    dask_labextension  enabled 
    - Validating...
      dask_labextension  OK

@ian-r-rose
Copy link
Collaborator Author

Hm, I see dask_labextension twice in that listing. Once enabled, and once disabled. Maybe go into the /home/mrocklin/.jupyter/jupyter_notebook_config.json file and manually remove that one?

@mrocklin
Copy link
Member

mrocklin commented Nov 3, 2018

That works. Everything seems pretty slick to me so far. I'll take a look at the web server side code next.

@ian-r-rose
Copy link
Collaborator Author

@jhamman, this is close-to-ready for another round of testing. I know you have a bunch of free time at AGU!

@jhamman
Copy link
Member

jhamman commented Dec 11, 2018

@ian-r-rose - took this for a spin today. Really liking where its heading. One thing, I think its still not obvious that the user needs to right click to insert the code cell. Can we make this a link/button on the cluster section in the extension?

@ian-r-rose
Copy link
Collaborator Author

@jhamman, good point. I've added another button to the cluster listing for injecting the client code. I've also fleshed out the context menu a bit, so the user can manipulate clusters using either avenue.

@consideRatio
Copy link

consideRatio commented Dec 13, 2018

Exciting work!! 👍

I ended up here after trying to use the extension to access a dashboard not accessible through the internet but only from within the network that the jupyter server is running.

@jhamman
Copy link
Member

jhamman commented Dec 14, 2018

@ian-r-rose - I'm happy here for now. Is there more you'd like to sneak in here?

@ian-r-rose
Copy link
Collaborator Author

I think this is ready from my perspective. Thanks for putting it through the paces @jhamman!

@ian-r-rose
Copy link
Collaborator Author

Okay @mrocklin, I think this is just about ready. I'm sure stuff is busy right now, but if you get the chance to take this for a spin...

@mrocklin
Copy link
Member

Spin taken. In general I think it looks great. The only-speedbump I ran into was not knowing immediately how to connect a notebook to it. The < > button ended up being correct, but I wonder if we might replace this text with something like "Connect".

I'd be more than happy to merge this in now though if you'd like to move on. I think that users will be able to figure that out.

@ian-r-rose
Copy link
Collaborator Author

We could certainly change it to CONNECT. I was feeling that the button panel was getting kind of cramped, and hoped that having a descriptive hover text would be sufficient (I was also considering restoring the up/down arrows instead of SCALE). But overall it's not a huge deal.

This PR is getting pretty big as it is, so I think having more refinements in follow-ups is a good idea. If you are happy, I'd say merge away!

@mrocklin
Copy link
Member

I'll give @jhamman an hour to look things over, and then plan to merge afterwards.

@mrocklin mrocklin merged commit fca248e into dask:master Dec 17, 2018
@mrocklin
Copy link
Member

This is in. Thanks @ian-r-rose .

Should we release?

@ian-r-rose
Copy link
Collaborator Author

Can we merge without squashing? This is a big PR and represents a lot of work, and most of the commits have semantic meaning.

@mrocklin
Copy link
Member

Ah, my apologies. I just squash-and-release by default these days. It's been soon enough that rewriting history seems reasonable to me. Care to just force push to master?

@ian-r-rose
Copy link
Collaborator Author

Sure, thanks!

@ian-r-rose ian-r-rose mentioned this pull request Dec 17, 2018
@ian-r-rose
Copy link
Collaborator Author

ian-r-rose commented Dec 17, 2018

@mrocklin can you add me as a maintainer of the dask-labextension package on PyPI (username irose)?

@mrocklin
Copy link
Member

mrocklin commented Dec 17, 2018 via email

@ian-r-rose
Copy link
Collaborator Author

Username irose

@mrocklin
Copy link
Member

mrocklin commented Dec 17, 2018 via email

@jhamman
Copy link
Member

jhamman commented Dec 17, 2018

Thanks @ian-r-rose!

@ian-r-rose
Copy link
Collaborator Author

Okay, published with server extension dask_labextension 3.1, and lab extension dask-labextension@0.3.0. The python package needed a quick patch release to correctly include the config yaml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants