- Public Repo: https://github.com/CenterForOpenScience/osf.io/
- Issues: https://github.com/CenterForOpenScience/osf.io/issues?state=open
- COS Development Docs: http://cosdev.readthedocs.org/
Table of contents
- Recommended: Running the OSF with docker-compose
- Running the OSF locally
- Running the API Server
- Common Development Tasks
There is now a docker-compose.yml that will stand up a basic version of the OSF using PostgreSQL, fakeCAS, Waterbutler, MFR, the OSF api, and the OSF Flask application. This removes the need for users to setup the dependencies and services on their local machine allowing them all to be run in containers. The instructions for setting up the docker-compose version of the OSF are here. Following these instructions means you will not need the rest of the instructions in this document.
invoke script provides several useful commands. For more information, run:
Running the OSF
NOTICE: The recommended way to run the OSF for development is now through docker-compose. Running locally is deprecated, unsupported, and these docs will be removed soon.
If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local test server with the following sequence:
invoke mongo -d # Runs mongod as a daemon invoke mailserver invoke rabbitmq invoke celery_worker invoke elasticsearch invoke assets -dw invoke server
Note that some or all of these commands will run attached to a console, and therefore the commands may need to be run in separate terminals. Be sure to activate your virtual environment each time a new terminal is opened. It is normal for the command to keep running!
Once started, you will be able to view the OSF in a web browser- by default, at http://127.0.0.1:5000/
In order to log in on your local server, you will also need to run the authentication server.
- For daily use, run fakeCAS. See CenterForOpenScience/fakeCAS for information on how to set up this service.
- For developing authentication-related features, run CAS. See CenterForOpenScience/docker-library/cas for information on how to set up this service.
Running the API Server
If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local API server with the sequence delineated under running the OSF and:
localhost:8000/v2/ in your browser to go to the root of the browsable API. If the page looks strange,
python manage.py collectstatic to ensure that CSS files are deposited in the correct location.
You can run the OSF server in livereload mode with:
$ invoke server --live
This will make your browser automatically refresh whenever a code change is made.
Some functionality depends on additional services that will not be started using the sequence above. For many development tasks, it is sufficient to run the OSF without these services, except as noted below. Some additional installation will be needed to use these features (where noted), in which case updates will also need to be installed separately.
An authentication server (either CAS or FakeCAS) must be available in order to log in to the OSF while running locally. This must be installed separately from the OSF. See running the OSF for details.
Waterbutler is used for file storage features. Upload and download features will be disabled if Waterbutler is not installed. Consult the Waterbutler repository and documentation for information on how to set up and run this service.
Modular File Renderer
The Modular File Renderer (MFR) is used to render uploaded files to HTML via an iFrame so that they can be viewed directly on the OSF. Files will not be rendered if the MFR is not running. Consult the MFR repository for information on how to install and run the MFR.
Normally you don't need to run celery_beat. If you work on tasks that are dispatched by celery_beat:
Some beat-dispatched tasks require metrics and release requirements. If needed:
invoke requirements --metrics
ShareJS is used for collaborative editing features, such as the OSF wiki. It will be installed by the OSF installer script, but must be run separately. To run a local ShareJS server:
$ invoke sharejs
Downloading citation styles
To download citation styles, run:
$ invoke update_citation_styles
These instructions assume a working knowledge of package managers and the command line. For a detailed step-by-step walkthrough suitable for new programmers, consult the COS Development Docs. See optional extras for information about services not included in the automated install process below.
Before attempting to run OSF setup commands, be sure that your system meets the following minimum requirements.
The following packages must be installed before running the automatic setup script:
- XCode command line tools (
- Homebrew package manager (run
brew upgrade --allbefore OSF install)
- Java (if not installed yet, run
brew install Caskroom/cask/java)
- Python 2.7
- virtualenv (
pip install virtualenv)
El Capitan and newer
brew install openssl env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography
Mac OS X
These instructions should work on Mac OSX >= 10.7
- Clone the OSF repository to your computer. Change to that folder before running the commands below.
- Create and activate your virtualenv.
virtualenv env source env/bin/activate
website/settings/local.py. NOTE: This is your local settings file, which overrides the settings in
website/settings/defaults.py. It will not be added to source control, so change it as you wish.
api/base/settings/local.py. NOTE: This is your local settings file, which overrides the settings in
website/settings/defaults.py. It will not be added to source control, so change it as you wish.
$ cp website/settings/local-dist.py website/settings/local.py $ cp api/base/settings/local-dist.py api/base/settings/local.py
- On MacOSX with homebrew, there is a script that should automate much of the install process:
$ pip install invoke==0.13.0 $ invoke setup
Additional configuration for Mac OS X
After running the automatic installer, you may find that some actions- such as running unit tests- fail due to an error with Mongo/ TokuMX. This can be resolved by increasing the system limits on number of open files and processes.
Add the following lines to
/etc/launchd.conf (creating the files if necessary):
limit maxfiles 16384 16384 limit maxproc 2048 2048
Then create or edit either
/etc/profile to include the following:
ulimit -n 2048
Additional things to install
The automated installer does not install CAS, Waterbutler, or MFR, which may be needed to run some OSF features locally. Consult the optional extras section for more details.
At present, there is no complete automated install process for other platforms. Although the process above should perform most setup steps on Mac OS, users of other platforms will need to perform the steps below manually in a manner appropriate to their system. Some steps of the installer script can be re-used, in which case the appropriate commands are noted below.
On Mac OS, we recommend using Homebrew to install external dependencies.
- Create local.py files for addons that need them (
invoke copy_settings --addons)
- Install TokuMX
- Install libxml2 and libxslt (required for installing python lxml)
- Install Java (if not already installed)
- Install elasticsearch
- Install python requirements (
invoke requirements --dev --addons)
- Install npm
- Install node and bower packages
Build assets (
invoke assets --dev)
If invoke setup hangs when 'Generating GnuPG key' (especially under linux), you may need to install some additional software to make this work. For apt-getters this looks like:
sudo apt-get install rng-tools
Followed by editing
/etc/default/rng-tools to add the line:
And finally starting the rng-tools daemon with:
sudo /etc/init.d/rng-tools start
Detailed installation and setup guides
Although some effort is made to provide automatic installation scripts, the following more detailed guides may be helpful if you are setting up the OSF on a machine already used for other development work, or if you wish to perform other advanced tasks. If the OSF is already working based on the instructions above, you can skip this section.
TokuMX is an open-source fork of MongoDB that provides support for transactions in single-sharded environments.
TokuMX supports all MongoDB features as of version 2.4 and adds
Installing with Mac OS
$ brew tap tokutek/tokumx $ brew install tokumx-bin
Installing on Ubuntu
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-key 505A7412 $ echo "deb [arch=amd64] http://s3.amazonaws.com/tokumx-debs $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tokumx.list $ apt-get update $ apt-get install tokumx
Installing on REHL/CentOS
$ sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
Migrating from MongoDB
TokuMX and MongoDB use different binary formats. To migrate data from MongoDB to TokuMX:
- Back up the MongoDB data
invoke mongodump --path dump
- Shut down the MongoDB server
- Uninstall MongoDB
- Install TokuMX (see instructions above)
- Restore the data to TokuMX
invoke mongorestore --path dump/osf20130903 --drop
- Verify that the migrated data are available in TokuMX
Installing Celery + RabbitMQ
- Install RabbitMQ. On MacOSX with homebrew,
$ brew update $ brew install rabbitmq
The scripts are installed to
/usr/local/sbin, so you may need to add
PATH=$PATH:/usr/local/sbin to your
For instructions for other OS's, see the official docs.
Then start the RabbitMQ server with
$ invoke rabbitmq
If you want the rabbitmq server to start every time you start your computer (MacOSX), run
$ ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents $ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
Starting A Celery Worker
Install Elasticsearch to use search features.
$ brew install email@example.com
note: Oracle JDK 7 must be installed for elasticsearch to run
$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb $ sudo dpkg -i elasticsearch-1.2.1.deb
$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.2.1.noarch.rpm $ sudo rpm -ivh elasticsearch-1.2.1.noarch.rpm
- In your
SEARCH_ENGINE = 'elastic'
- Start the Elasticsearch server and migrate the models.
$ invoke elasticsearch $ invoke migrate_search
Starting a local Elasticsearch server
$ invoke elasticsearch
The Node Package Manager (NPM) is required for installing a number of node-based packages.
# For MacOSX $ brew update && brew install node
Installing Node on Ubuntu is slightly more complicated. Node is installed as
nodejs, but Bower expects
the binary to be called
node to fix, then verify that
node is properly aliased:
# For Ubuntu $ sudo apt-get install nodejs $ sudo ln -s /usr/bin/nodejs /usr/bin/node $ node --version # v0.10.25
Common Development Tasks
Running the shell
To open the interactive Python shell, run:
$ invoke shell
To run all tests:
$ invoke test --all
To run a certain test method
$ nosetests tests/test_module.py:TestClass.test_method
Run OSF Python tests only:
$ inv test_osf
Run addons Python tests only:
$ inv test_addons
$ inv karma
inv karma will start a Karma process which will re-run your tests every time a JS file is changed.
To do a single run of the JS tests:
$ inv karma --single
By default, Karma will run tests using a PhantomJS headless browser. You can run tests in other browsers like so:
$ inv karma -b Firefox
If you want to run cross browser tests with SauceLabs, use "sauce" parameter:
$ inv karma --sauce
Addons tests are not run by default. To execute addons tests, run
$ invoke test_addons
localhost:1025 in you
... MAIL_SERVER = "localhost:1025" ...
Sent emails will show up in your server logs.
Optional: fire up a pseudo-mailserver with:
$ invoke mailserver -p 1025
Building front-end assets with webpack
Use the following command to update your requirements and build the asset bundles:
$ inv assets -dw
The -w option puts you in "watch" mode: the script will continue running so that assets will be built when a file changes.
Setting up the Admin Module
admin/README.mdfor information on how to set up the OSF Admin module
Getting application credentials
Many addons require application credentials (typically an app key and secret) to be able to authenticate through the
OSF. These credentials go in each addon's
local.py settings file (e.g.
COS is Hiring!
Want to help save science? Want to get paid to develop free, open source software? Check out our openings!