-
Notifications
You must be signed in to change notification settings - Fork 0
Installation instructions
-
Login to Windows Azure Management Portal
-
Go to VM Screen, select "+ NEW" from the lower left corner of the screen
-
On the pane that appears select, Compute -> Virtual Machine -> Quick Create
DNS Name: pick something unique and memorable Image: Ubuntu Server 13.04 New Password/Confirm: provide a good password for the azureuser Region/Affinity Group: West US
=> Click Create a virtual machine
-
Once your virtual machine is created, select it from the VM screen
-
Go to the Endpoints menu
(+ Add) an Endpoint at the bottom of the screen
Provide the following information Name: HTTP Public Port: 80 Private Port: 8000
-
Restart the VM from the management console
Troubleshooting note: You may get a message that Endpoint was successfully created but restart failed. In this case, go back to Dashboard (Management Console) and click Restart. In some cases, multiple attempts may be required.
At this point, it appears that a restart is not mandatory to continue with configuration.
-
Now you can login to it and start configuring things
-
Login to your VM via ssh as: azureuser <password provided at vm build - Step 3)
-
Things to do once you're in Ubuntu
- sudo apt-get update
- enter password (azureuser password from Step 3)
- sudo apt-get upgrade -y
- sudo apt-get install git python-pip -y
- sudo apt-get update
-
Install Python 2.7. For current Debian-based Linux distributions (such as Ubuntu), BSD and Mac Python 2.7 is usually installed. However Redhat-based Linux distributions, such as RHEL and CentOS, are sometimes behind the curve and do not have Python 2.7. As of this writing, CentOS 6.4 is at Python 2.6, which is well on its way to EOL. Python 2.6 may work, but code will be written with 2.7 and 3.3+ in mind. Below are instructions for Ubuntu Linux 13.04+.
sudo apt-get install python2.7 python2.7-dev python-virtualenv
-
Install PIP.
sudo apt-get install python-pip
-
Install virtualenv.
sudo apt-get install python-virtualenv
-
Install Git.
sudo apt-get install git
-
Install the prerequisites for MySQL-Python.
sudo apt-get install build-essential python-dev libmysqlclient-dev
-
One of the new dependencies of CodaLab is libjpeg library:
sudo apt-get install libjpeg-dev
In order to test uploading and running bundles in CodaLab, you will need to have a Windows Azure storage account. Once you have set up your Azure account, log on to the Azure Portal and follow the steps in this section.
- Log on to the Azure Portal.
- In the left pane, click Storage.
- Select your storage account.
- At the bottom of the dashboard, click Manage Access Keys. Copy your access keys, you'll need them later.
- At the top of the dashboard page, click Containers.
- At the bottom of the Containers page click Add.
- Create a new container named "bundles". Set the Access to "Private".
- Add another container named "public". Set the Access to "Public Blob".
- Install azure-cli
In Ubuntu it can be installed using the following command:
`sudo apt-get install nodejs-legacy`
`sudo apt-get install npm`
`sudo npm install -g azure-cli`
To login run the following command:
azure login
Copy the code offered to you, above, and open a browser to http://aka.ms/devicelogin. Enter the code, and then you are prompted to enter the username and password for the identity you want to use. When that process completes, the command shell completes the log in process.
- Switch to Azure Service Mode by
azure config mode asm
- From command line
azure sb namespace create <name> <location>
where<location>
can be "East US" - Log on to the Azure Portal.
- In the left pane, click Service Bus.
- Select the service bus you just created.
- At the top of the screen click Queues.
- Click Create a new queue.
- Click Quick Create and create a new queue named "compute".
- Click Create A New Queue.
- At the bottom of the screen, click New, and create another queue named "response".
- In the left pane, click Service Bus.
- At the bottom of the page, click Connection Information.
- Copy the following connection information:
- Namespace name
- Default issuer
- Default key
-
Make sure you have the dependencies (Python 2.7 and virtualenv). If you're running Ubuntu:
sudo apt-get install python2.7 python2.7-dev python-virtualenv
-
Clone the CodaLab repository:
git clone https://github.com/codalab/codalab-cli cd codalab-cli
-
Run the setup script (will install things into a Python virtual environment):
./setup.sh
-
Set your path to include CodaLab (add this line to your
.bashrc
):export PATH=$PATH:<path to codalab-cli>/codalab/bin
-
Optional: include some handy macros (add this line to your
.bashrc
):. <path to codalab-cli>/rc
-
Fork the CodaLab repo from GitHub.
-
Clone your fork.
git clone https://github.com/<username>/codalab.git
For more details and recommended practices, see Developer Guidelines.
In this segment, you will run the dev_setup script. This will install all dependencies and create a new virtual environment (venv) for CodaLab.
-
Run the dev_setup script.
Windows
cd codalab .\dev_setup.bat
If you are going to use SQL Server as a database, you will need to install the Python PyODBC library. Before running
dev_setup
, you can download the installer, then run the setup script as follows (assuming the installer was downloaded at the root of a D drive):cd codalab dev_setup.bat D:\pyodbc-3.0.7.win-amd64-py2.7.exe
Linux
cd codalab source ./dev_setup.sh
-
Activate the virtual environment.
Windows
venv\Scripts\activate
Linux
source venv/bin/activate
Now you are ready to install the application schema and default data into the database.
You can configure CodaLab to use either SQL Server or MySQL. Both of these require you to explicitly create a database.
Follow these steps to install MySQL Python.
Windows
-
Open a Windows command prompt.
-
Navigate to the virtual environment (venv) for CodaLab and use the following command to install MySql-Python.
easy_install mysql-python
-
Launch the MySQL Command Line Client.
-
Use the following command to create a new database:
create database if not exists MySQL_DevDB;
Linux
-
Open a terminal window.
-
Run the following command to install MySQL:
sudo apt-get install mysql-server
-
Login to MySQL as root by typing the following command:
mysql -u root -p
Enter your root password when prompted.
-
Use the following command to create a new database:
create database if not exists MySQL_DevDB;
-
Type
exit
to return to the terminal prompt.
Mac
-
Open a terminal window.
-
Login to MySQL as root by typing the following command:
mysql -u root -p
Enter your root password when prompted. -
Use the following command to create a new database:
create database if not exists MySQL_DevDB;
-
Type
exit
to return to the terminal prompt. -
Finish up installation for Django
export DYLD_LIBRARY_PATH=/usr/local/mysql/lib:$DYLD_LIBRARY_PATH
-
Open your local configuration file (
local.py
). If there is nolocal.py
, save a copy oflocal_sample.py
namedlocal.py
in the same directory. -
In the
Azure storage
section, enter your Azure account details:DEFAULT_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' AZURE_ACCOUNT_NAME = "<enter name>" AZURE_ACCOUNT_KEY = '<enter key>' AZURE_CONTAINER = '<enter container name>' PRIVATE_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' PRIVATE_AZURE_ACCOUNT_NAME = "<enter name>" PRIVATE_AZURE_ACCOUNT_KEY = "<enter key>" PRIVATE_AZURE_CONTAINER = "<enter container name>" BUNDLE_AZURE_CONTAINER = "<enter the name of your bundle container>" BUNDLE_AZURE_ACCOUNT_NAME = PRIVATE_AZURE_ACCOUNT_NAME BUNDLE_AZURE_ACCOUNT_KEY = PRIVATE_AZURE_ACCOUNT_KEY
-
In the
Service Bus
section, enter your service bus connection information:SBS_NAMESPACE = '<enter the name of your service bus>' SBS_ISSUER = 'owner' SBS_ACCOUNT_KEY = '<enter value for 'default key'>'
Important: Do not change the values for
DEFAULT_FILE_STORAGE
andPRIVATE_FILE_STORAGE
, as these parameters contain the name of the Python class which implements the Azure storage back-end for Django. -
In the
DATABASES
section, enter the configuration settings for the database you want to use.SQL Server*
DATABASES = { 'default': { 'ENGINE': 'sql_server.pyodbc', 'NAME': 'somename', # Leaver user and password blank to use integrated security 'USER': '', 'PASSWORD': '', 'HOST': '(localdb)\\v11.0', 'PORT': '', 'OPTIONS': { 'driver': 'SQL Server Native Client 11.0', } }
MySQL
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'MySQL_DevDB', 'USER': 'someuser', 'PASSWORD': 'somepassword', 'HOST': 'someserver', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. 'PORT': '', # Set to empty string for default. }
Note: If you want to use MySQL you'll need to manually install it and create a database before proceeding. CodaLab setup does not install MySQL.
-
In the base.py configuration file add name and password for SendGrid account:
EMAIL_HOST = 'smtp.sendgrid.net' EMAIL_HOST_USER = '<SendGrid account name>' EMAIL_HOST_PASSWORD = '<SendGrid account password>'
-
Open
codalab/codalab/codalabtools/compute/sample.config
. -
Save a copy of
sample.config
named.codalabconfig
in the same directory. -
Open
.codalabconfig
. -
In the
compute-worker
section, enter the configuration settings for the storage account and the compute queue.compute-worker: azure-storage: account-name: "your account name" account-key: "your account key" azure-service-bus: namespace: "your namespace" key: "your secret key" issuer: "owner" listen-to: "name of queue" local-root: "D:\\Temp"
-
For Ubuntu change local-root to /tmp
In order to work with competitions and bundles in your local development environment, you will need to manually enable Cross-origin resource sharing (CORS). CORS is a mechanism that allows many resources on a web page to be requested from another domain outside the domain the resource originated from. Web Browsers commonly apply same origin restriction policy to network requests. CORS relaxes these restrictions allowing domains to give each other permissions for accessing each other's resources. CORS is supported for Blob, Table and Queue services and can be enabled for each service using Azure SDK for python.
-
Make sure you have a valid management certificate to connect to the Service Management endpoint. This tutorial explains how to create a certificate and upload it to the Azure management portal: http://azure.microsoft.com/en-us/documentation/articles/cloud-services-python-how-to-use-service-management/.
-
Open
codalab/codalabtools/compute/enableCORS.py
in a text editor, and updateaccount_name
andaccount_key
with the account name and key for your blob storage account:import sys import yaml from os.path import dirname, abspath # Add codalabtools to the module search path sys.path.append(dirname(dirname(abspath(__file__)))) from codalabtools.azure_extensions import (Cors,CorsRule,set_storage_service_cors_properties) account_name = "<your blob storage account name>" account_key = "<your blob storage account key>" cors_rule = CorsRule() cors_rule.allowed_origins = '*' # this is fine for dev setup cors_rule.allowed_methods = 'PUT' cors_rule.exposed_headers = '*' cors_rule.allowed_headers = '*' cors_rule.max_age_in_seconds = 1800 cors_rules = Cors() cors_rules.cors_rule.append(cors_rule) set_storage_service_cors_properties(account_name, account_key, cors_rules)
-
Save your changes, activate your virtual environment and run the script:
Windows
python scripts\cors-enable.py
Linux
python scripts/cors-enable.py
To initialize the database, you will need to run a few standard Django commands, and the CodaLab database initialization script.
- validate: Validates all installed models (according to the INSTALLED_APPS setting) and prints validation errors to standard output.
-
syncdb: Synchronizes the database state with the current set of models and migrations (note that the
migrate
option is used). - initialize.py: Inserts initial data into the database.
-
Run the following commands to initialize the database.
Windows
cd codalab python manage.py validate python manage.py syncdb --migrate python scripts\initialize.py
Linux
cd codalab python manage.py validate python manage.py syncdb --migrate python scripts/initialize.py
-
Run tests to verify that everything is working.
python manage.py test
-
Optional: Populate the site with some sample data.
Windows
python scripts\users.py python scripts\competitions.py
Linux
python scripts/users.py python scripts/competitions.py
This note will explain how to run the CodaLab Django web site and the CodaLab bundle service (cl server) side-by-side on a single machine. This setup is useful for people doing development, especially when working at the interface between the two systems.
As pre-requisite, we assume that:
-
You can already run the Django web site on your local machine as described on this page.
-
You have also forked the codalab-cli project and have gone through the steps listed in the Readme.
With those assumptions in place:
-
Begin by enabling the Worksheet feature in your web site. In your Django settings (local.py), add:
PREVIEW_WORKSHEETS = True BUNDLE_SERVICE_URL = "http://localhost:2800" # CODE_PATH points to local source code for bundles repo. Path is relative to this file. BUNDLE_SERVICE_CODE_PATH = "..\\..\\..\\..\\codalab-cli" if len(BUNDLE_SERVICE_CODE_PATH) > 0: sys.path.append(join(dirname(abspath(__file__)), BUNDLE_SERVICE_CODE_PATH)) codalab.__path__ = extend_path(codalab.__path__, codalab.__name__)
These additional elements say to enable Worksheets using the bundle service running at the given URL (in the next steps we'll cover how to start the bundle service with the
cl server
command). Finally, the code inapps\web\bundles.py
will need to import modules from thecodalab-cli
project. Therefore, we extend the Python path by pointing to the CLI code on your machine. If you have reposcodalab
andcodalab-cli
checked in folders which are siblings, the relativeBUNDLE_SERVICE_CODE_PATH
given above should work for your setup. -
The Django web site is a client of the bundle service. For example, to get a list of worksheets, the Django web site makes a call to the
list_worksheets
API exposed by the bundle service. However, to handle authorization, the bundle service must be able to make calls back to the web site, which hosts the OAuth server. As a result, the bundle service is a trusted OAuth client of the web site. To setup this trusted relationship:a. Create a user on the web site. Typically, I call this user
codalab
. Treat it as an admin user even though the CodaLab site doesn't really have admin roles today.b. Activate your virtual environment and run the script:
codalab\codalab\scripts\sample_cl_server_config.py
. The script will generate an output of the form:Checking that confidential client exists for user codalab Client already exists. Add the following server block to your CLI config: "server": { "auth": { "address": "http://localhost:8000", "app_id": "5m <snip> Le", "app_key": "_b <snip> !_", "class": "OAuthHandler" }, "class": "SQLiteModel", "host": "localhost", "port": 2800 }
Take the "server" block in the output and insert it into your CLI config file (
.codalab\config.json
in your home directory). Since the values forapp_id
andapp_key
can be long, make sure to remove any line breaks that resulted from copying from the command prompt.c. Run the Django web site. Make sure it uses port 8000. Note for Visual Studio users: with the latest Python Tools for Visual Studio, you can set the port number in the Debug tab of the project properties.
d. Activate your virtual environment and then run the bundle service:
cl server
.
-
Use the following command to start the CodaLab server locally.
python manage.py runserver
-
Open a browser and navigate to http://127.0.0.1:8000 to preview the site.
-
When your next coding session comes along, remember to work in the virtual environment you created:
Windows
venv\Scripts\activate
Linux
source venv/bin/activate
Note: If you experience database errors try deleting the database file (\codalab\codalab\dev_db.*) and run syncdb again. After creating a new database be sure to run initialize.py
in the scripts
folder in order to insert initial data required by the app.
In order to test competitions and bundles locally, you'll need to run the compute worker roles. There are two worker.py
scripts that you will need to run.
-
Open a command prompt and activate your CodaLab virtual environment.
-
Start the first compute worker as shown here:
cd codalab python worker.py
-
Open a second command prompt and activate your CodaLab virtual environment.
-
Start the second compute worker as shown here:
cd codalabtools\compute python worker.py
-
If you plan to test competitions locally, open a third command prompt and activate the virtual environment for the CodaLab CLI, then start the bundle server:
cl server
Every execution on CodaLab (should ideally) happen in a docker container, which provides a standardized Linux environment that is lighterweight than a full virtual machine.
The current official docker image is codalab/ubuntu
, which consists of
Ubuntu 14.04 plus some standard packages. See the CodaLab docker
registery.
To install docker on your local machine (either if you want see what's actually in the environment or to run your own local CodaLab instance), follow these instructions:
sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo apt-get update
sudo apt-get install lxc-docker
sudo useradd $USER docker
Then, to test out your environment, open a shell (the first time you do this, it will take some time to download the image):
docker run -t -i codalab/ubuntu:1.8
Now, let us integrate docker into CodaLab. First, we need to setup a job
scheduling system (that manages the deployment of runs on machines). Note that
CodaLab itself doesn't do this, so that it can be easily integrated into
different systems. An easy way to set this up is to use q
from Percy Liang's fig
package:
git clone https://github.com/percyliang/fig
# Add fig/bin/q to your $PATH
q -mode master # Run in a different terminal
q -mode worker # Run in a different terminal
Now, let us tell CodaLab to use q
and run things in docker (these two things
are orthogonal choices). Edit the .codalab/config.json
as follows:
"workers": {
"q": {
"verbose": 1,
"docker_image": "codalab/ubuntu:1.8"
"dispatch_command": "python $CODALAB_CLI/scripts/dispatch-q.py"
}
}
To test it out:
cl work-manager -t q # Run in a different terminal
cl run 'cat /proc/self/cgroup' -t # Should eventually print out lines containing the string `docker`