Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
The Sage Cell Server---providing a way to embed Sage computations into any web page.
JavaScript Python HTML CSS CoffeeScript Makefile Other
Tag: BETA_0

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
doc
minifiers
sage-patches
ssh
static
templates
timing
.gitignore
.hgignore
LICENSE.txt
README.rst
__init__.py
db.py
db_mongo.py
db_sqlalchemy.py
db_sqlite.py
db_zmq.py
device_ipython.py
device_process.py
filestore.py
interact_compatibility.py
interact_old_sagenb.py
interact_singlecell.py
ip_receiver.py
misc.py
raw_queue.py
singlecell_config.py.default
singlecell_exec_config.py
start_device.py
start_mongo
start_web.py
trusted_db.py
ultraTB_09.py
ultraTB_10.py
util.py
web_server.py

README.rst

This is a demo of a 3-component Python compute service, using MongoDB.

Installation

We depend on the following packages:

Optionally, you can also use nginx and uWSGI to have a multithreaded web server that can drastically increase your capabilities over the built-in Python web server. There are lots of other WSGI servers you could use as well.

These instructions assume that Sage is installed and that it can be run using the command sage.

Dependencies

In the following instructions, $SERVER refers to the directory containing all of the software (for example, it might be /var/singlecellserver).

Build dependencies

The builds below have their own dependences, which you will have to install before successfully configuring them. On Ubuntu, this command should take care of most or all of them:

sudo apt-get install uuid-dev libpcre3-dev zlib1g-dev openssh-server

ØMQ

Download ØMQ and build it in $SERVER/zeromq-2.1.7/install/:

cd $SERVER
wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
tar -xzvf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7
./configure --prefix=`pwd`/install && make install

nginx

Download nginx and build it in $SERVER/nginx-1.0.4/install/:

cd $SERVER
wget http://www.nginx.org/download/nginx-1.0.4.tar.gz
tar -xzvf nginx-1.0.4.tar.gz
cd nginx-1.0.4
./configure --prefix=`pwd`/install && make install

uWSGI

These instructions are based on these instructions. We don't want to require libxml2 (it's just for the config files, I believe), so we'll make our own build configuration that doesn't support XML build files. Also note that any version of uwsgi before 0.9.8 will not build with gcc-4.6.

  1. Get uWSGI:

    cd $SERVER
    wget http://projects.unbit.it/downloads/uwsgi-0.9.8.1.tar.gz
    tar -xzvf uwsgi-0.9.8.1.tar.gz
    
  2. Change the configuration file to set xml = false:

    cd uwsgi-0.9.8.1/buildconf
    cp default.ini myproject.ini
    # edit myproject.ini to make the xml line read: xml = false
    cd ..
    
  3. Build uWSGI:

    sage -python uwsgiconfig.py --build myproject
    

Python packages

Install the required Python packages. Note that we upgrade setuptools since the version that comes with Sage is too old for the most recent version of PyMongo.

The sudo in the first line is not required if Sage is not installed in a protected directory. In the penultimate line, replace $SERVER with the same directory name that it represented above (environmental variables will not be preserved inside the Sage shell).

sudo sage -sh # install into Sage's python
easy_install pip # install a better installer than easy_install
pip install --upgrade setuptools # need upgrade for pymongo
pip install flask
pip install pymongo
pip install pyzmq --install-option="--zmq=$SERVER/zeromq-2.1.7/install"
exit

Single Cell Server

The repository for this software is on GitHub, under the name simple-python-db-compute. Either download the tarball, extract the contents into $SERVER, and rename the directory to single-cell-server; or use git to clone the code:

cd $SERVER
git clone git://www.github.com/jasongrout/simple-python-db-compute.git single-cell-server

MongoDB

Download the appropriate version of MongoDB from here and extract the contents to the $SERVER directory.

Sage

Several patches enable Sage to take advantage of the enhanced protocol for communicating graphical displays. In order to patch Sage, apply the patches to your Sage installation found in the sage-patches directory. Apply them in numeric order. I suggest using Mercurial Queues so that it is easy to back out the patches if needed. After applying the patches, rebuild Sage with sage -b.

Jmol

In sage mode, Sage can output 3d graphs in Jmol format. The Jmol java applet must be installed in order to see these. It is sufficient to make a symbolic link from the /static directory over to the appropriate Jmol directory in the Sage notebook:

cd $SERVER/static
ln -s $SAGE_ROOT/sage/devel/sagenb/sagenb/data/jmol .

Configuration and Running

MongoDB

  1. Make new directories $SERVER/mongodb and $SERVER/mongodb/mongo:

    mkdir -p $SERVER/mongodb/mongo
    
  2. Make a $SERVER/mongodb/mongodb.conf file. Copy the text below into this file, replacing <MONGODB_PORT> with the port you want for your database and <$SERVER> with the path of the server directory.

    dbpath = <$SERVER>/mongodb/mongo/
    bind_ip = localhost
    port = <MONGODB_PORT>
    auth = true
    logpath = <$SERVER>/mongodb/mongodb.log
    logappend = true
    nohttpinterface = true
    
    # Comment the below out (don't just switch to false)
    # in order to cut down on logging
    verbose = true
    cpu = true
    
  3. Start up the MongoDB daemon (replace the location of mongodb as appropriate):

    cd $SERVER/mongodb/
    $SERVER/mongodb-linux-x86_64-1.8.2/bin/mongod -f mongodb.conf
    
  4. Now you need to set up usernames and passwords for database access, if the database is running on a shared server.

    Note

    MongoDB authentication documentation recommends that you run without authentication, but secure the environment so that the environment is trusted.

    Set up an admin user, authenticate, then set up a user for the singlecelldb database. Since we include the <SINGLECELL_USER> and <SINGLECELL_PASSWORD> in a URL later, it's helpful if neither of them contain any of %:/@ (any length of password with letters and numbers would be okay).

    $SERVER/mongodb-linux-x86_64-1.8.2/bin/mongo --port <MONGODB_PORT> # start up mongo client
    > use admin
    > db.addUser("<ADMIN_USER>", "<ADMIN_PASSWORD>")
    > db.auth("<ADMIN_USER>", "<ADMIN_PASSWORD>")
    > use singlecelldb
    > db.addUser("<SINGLECELL_USER>", "<SINGLECELL_PASSWORD>")
    > quit()
    

nginx

  1. Make the $SERVER/nginx-1.0.4/install/conf/nginx.conf file have only one server entry, as shown here (delete all the others). <SERVER_PORT> should be whatever port you plan to expose to the public (should be different from <MONGODB_PORT>).

    server {
        listen <SERVER_PORT>;
        server_name localhost;
        charset utf-8;
        client_max_body_size 4M; # Maximum file upload size of 4MB
        location / {
            uwsgi_pass unix:/tmp/uwsgi.sock;
            include uwsgi_params;
        }
    }
    
  2. Start nginx:

    $SERVER/nginx-1.0.4/install/sbin/nginx
    

Single Cell Server

The only thing left now is to configure and start the single-cell compute server. The server will automatically launch a number of workers via passwordless SSH into an untrusted account (i.e., an account with heavy restrictions; this account will be executing arbitrary user code).

Warning

The untrusted account will execute arbitrary user code, which may include malicious code. Make sure that you are securing the account properly. Working with a professional IT person is a very good idea here. Since the untrusted accounts can be on any computer, one way to isolate these accounts is to host them in a virtual machine that can be reset if the machine is compromised.

These instructions assume that the locked-down account is on the same computer as the server.

  1. Install OpenSSH if it is not already installed.

  2. Create a new restricted user account and enable passwordless SSH from your account to the restricted account:

    sudo adduser <UNTRUSTED_USER>
    ssh-keygen # not needed if you already have a public key
    sudo mkdir <UNTRUSTED_USER_HOME_DIR>/.ssh
    sudo cp ~/.ssh/id_rsa.pub <UNTRUSTED_USER_HOME_DIR>/.ssh/authorized_keys
    

    Test the passwordless SSH by logging in (ssh <UNTRUSTED_USER>@localhost) and out (exit). If you have a passphrase for your key, you may need to type it once, but there should be a way to store the key and log in fully automatically.

  3. Create a configuration file $SERVER/single-cell-server/singlecell_config.py by copying and modifying $SERVER/single-cell-server/singlecell_config.py.default. The mongo_uri should be set to mongodb://<SINGLECELL_USER>:<SINGLECELL_PASSWORD>@localhost:<MONGODB_PORT>. If you will be running the server using Sage, replace the line python='python' with python='sage -python'.

Warning

Make the singlecell_config.py file only readable by the trusted account, not by the untrusted account, since it contains the password to the database:

chmod 600 singlecell_config.py
  1. Start uWSGI. The -p 50 means that uWSGI will launch 50 workers to handle incoming requests. Adjust this to suit your needs.

    sage -sh
    cd $SERVER/single-cell-server
    ../uwsgi-0.9.8.1/uwsgi -s /tmp/uwsgi.sock -w web_server:app -p 50 -M
    
  2. Start up the trusted server. Replace <UNTRUSTED_USER>@localhost below with the SSH address for the untrusted account. Adjust the number of workers (-w) to meet your needs. Add the argument -q to minimize the number of log messages.

    cd $SERVER/single-cell-server/
    sage -python trusted_db.py -w 50 --untrusted-account untrusted@localhost
    

    When you want to shut down the server, just press Ctrl-C. This should automatically clean up the worker processes.

  3. Go to http://localhost:<SERVER_PORT> to use the single-cell server.

License

See the file "LICENSE.txt" for terms & conditions for usage and a DISCLAIMER OF ALL WARRANTIES.

Something went wrong with that request. Please try again.