Skip to content

monetate/s3nb

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

S3-backed notebook manager for IPython

Setup

  1. Install:

    from pypi:

    pip install s3nb

    from source with pip:

    pip install git+https://github.com/monetate/s3nb

    or from source the old fashioned way:

    git clone git@github.com:monetate/s3nb.git
    cd s3nb
    python ./setup.py install
  2. Configure

    # set this - notebooks will be stored relative to this uri
    S3_NOTEBOOK_URI=s3://path/to/notebooks/
    
    # and this
    IPYTHON_MAJOR_VERSION=4
    
    # optionally set this - checkpoints will be stored locally, relative to this path (for IPython 3)
    CHECKPOINT_ROOT_DIR=~/.checkpoints
    
    # optionally set this
    PROFILE=s3nbserver
    
    # shouldn't need to edit beyond this point
    
    ## IPython 2.x
    IPYNB_MANAGER=S3NotebookManager
    IPYNB_MANAGER_CFG=notebook_manager_class
    
    ## IPython 3.x or 4.x
    if [ $IPYTHON_MAJOR_VERSION == 3 ] || [ $IPYTHON_MAJOR_VERSION == 4 ]; then
        IPYNB_MANAGER=S3ContentsManager
        IPYNB_MANAGER_CFG=contents_manager_class
    fi
    
    IPYTHONDIR=${IPYTHONDIR:-$HOME/.ipython}
    PROFILE_DIR=${IPYTHONDIR}/profile_${PROFILE}
    
    if [ ! -d $PROFILE_DIR ]; then
        ipython profile create $PROFILE
        IPYNB_CONFIG=${PROFILE_DIR}/ipython_notebook_config.py
        mv $IPYNB_CONFIG $IPYNB_CONFIG.orig
        cat > $IPYNB_CONFIG <<EOF
    c = get_config()
    c.NotebookApp.${IPYNB_MANAGER_CFG} = 's3nb.${IPYNB_MANAGER}'
    c.${IPYNB_MANAGER}.s3_base_uri = '$S3_NOTEBOOK_URI'
    EOF
    fi
    
    
    if  [ $IPYTHON_MAJOR_VERSION == 3 ] || [$IPYTHON_MAJOR_VERSION == 4 ] ; then
        echo "c.S3ContentsManager.checkpoints_kwargs = {'root_dir': '${CHECKPOINT_ROOT_DIR}'}"  >> ${IPYNB_CONFIG}
    fi
  3. If you haven't already, configure AWS variables for boto. Follow these instructions.

  4. Run

    jupyter notebook --config=~/.ipython/s3nbserver/ipython_notebook_config.py

Development

  1. Provision a virtual machine with vagrant up
  2. Create an IPython profile with make configure -e S3_BASE_URI=YOUR_BUCKET
  3. Share you AWS credentials with the virtual machine with make creds -e AWS_USER=YOUR_USER
  4. Run the notebook server with make run

About

S3-backed notebook manager for IPython

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •