Facilitating Open Science
Python JavaScript Mako HTML CSS Jupyter Notebook Other
Permalink
Failed to load latest commit information.
addons Rename BaseAddonConfig -> BaseAddonAppConfig Jan 17, 2017
admin Merge branch 'hotfix/0.104.10' into develop Jan 11, 2017
admin_tests Merge branch 'develop' into feature/django-osf-update Dec 8, 2016
api Merge pull request #6724 from CenterForOpenScience/feature/move-addons Jan 17, 2017
api_tests Merge pull request #6724 from CenterForOpenScience/feature/move-addons Jan 17, 2017
framework Fix static assets; remove redundant addon static URL route Jan 13, 2017
logs Add logs/ directory for add_file_logger Feb 18, 2015
osf Merge pull request #6724 from CenterForOpenScience/feature/move-addons Jan 17, 2017
osf_tests Move addon code to the /addons directory Jan 13, 2017
requirements move local dev requirements over, use proper test factories for insti… Nov 23, 2016
scripts Prevent circular import when importing serialize_node Jan 17, 2017
tasks Fix migrate_search: Call init_app before migrating Jan 17, 2017
tests Merge pull request #6724 from CenterForOpenScience/feature/move-addons Jan 17, 2017
website Prevent circular import when importing serialize_node Jan 17, 2017
.bowerrc Add bower.json and jquery2 Mar 1, 2014
.coveragerc Pull latest develop Feb 29, 2016
.docker-compose.env allow for internal and external waterbutler urls, docker-compose fix Jan 14, 2017
.docker-compose.mfr.env Merge branch 'develop' into feature/django-osf-preprint-update Nov 7, 2016
.docker-compose.sharejs.env docker sharejs and docs Jan 10, 2017
.docker-compose.wb.env Merge branch 'develop' into feature/django-osf-preprint-update Nov 7, 2016
.dockerignore do not exclude static build from docker build Jul 19, 2016
.gitignore Ignore unison files Jan 9, 2017
.jshintrc Modify travis test tasks Oct 27, 2016
.travis.yml [WIP] Make migrations great again. (#6594) Dec 1, 2016
CHANGELOG Merge branch 'develop' into feature/django-osf Jan 6, 2017
CONTRIBUTING.md Update CONTRIBUTING.md Jul 12, 2016
Dockerfile more badges docker Jan 17, 2017
LICENSE Create LICENSE Aug 27, 2016
MANIFEST.in Adding manifest Jul 19, 2016
Migrations.ipynb Refactor GUID to store either guid or ObjectID Aug 25, 2016
NOTICE Fix transaction rollback in tests Nov 4, 2016
PULL_REQUEST_TEMPLATE.md Add PULL_REQUEST_TEMPLATE Apr 11, 2016
README-docker-compose.md allow for internal and external waterbutler urls, docker-compose fix Jan 14, 2017
README.md Make it clear that local development is bad. Jan 6, 2017
README.rst Add --branch argument and make it easier to update osf repo Aug 4, 2016
addons.json [owncloud] Minor fixes to addons.json and the v2 api Sep 13, 2016
bower.json Fix institution logo paths (#6579) Nov 21, 2016
com.runlevel1.lo0.192.168.168.167.plist Add plist for lo0 and update docs Jan 6, 2017
dev-requirements.txt Add dev requirements; add invoke tasks; add README Aug 2, 2016
docker-compose.override.yml docker celery worker and beat Jan 10, 2017
docker-compose.yml fix docker-compose postgres initdb configuration Jan 12, 2017
docker-sync.yml exclude etree.py from docker-sync Jan 13, 2017
find_field_lengths_in_mongo.js More migration fixes! (#150) Oct 11, 2016
karma.common.conf.js Fix JS tests Apr 9, 2015
karma.conf.js Add more browsers: IE9 fails a test already.. Mar 13, 2015
karma.saucelabs.conf.js Add sauce-sample.json Mar 16, 2015
main.py Adding DJANGO_SETTINGS_MODULE to inv server Mar 9, 2016
manage.py add logging config for non-init app Nov 23, 2016
package.json Bump version and update changelog Dec 19, 2016
pytest.ini Make NodeLicense.year nullable; fix Node#update; fix tests Dec 9, 2016
requirements.txt Upgrade raven Jan 14, 2017
sauce-sample.json Add sauce-sample.json Mar 16, 2015
setup.cfg Revert config changes Nov 17, 2016
setup.py rename osf_models to osf Oct 11, 2016
startover-make.sh [WIP] Make migrations great again. (#6594) Dec 1, 2016
startover.sh [WIP] Make migrations great again. (#6594) Dec 1, 2016
tasks.py rename osf_models to osf Oct 11, 2016
webpack.common.config.js Move addon code to the /addons directory Jan 13, 2017
webpack.dev.config.js Enable source maps for everything Mar 4, 2016
webpack.prod.config.js Enable source maps for everything Mar 4, 2016

README.md

OSF

Table of contents

Docker Compose

There is now a docker-compose.yml that will stand up a basic version of the OSF using PostgreSQL, fakeCAS, Waterbutler, MFR, the OSF api, and the OSF Flask application. This removes the need for users to setup the dependencies and services on their local machine allowing them all to be run in containers. The instructions for setting up the docker-compose version of the OSF are here. Following these instructions means you will not need the rest of the instructions in this document.

Help

The COS Development Docs provide detailed information about all aspects of OSF development. This includes detailed installation instructions, a list of common setup errors, and other troubleshooting.

The OSF invoke script provides several useful commands. For more information, run:

invoke --list

Running the OSF

NOTICE: The recommended way to run the OSF for development is now through docker-compose. Running locally is deprecated, unsupported, and these docs will be removed soon.

If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local test server with the following sequence:

invoke mongo -d  # Runs mongod as a daemon
invoke mailserver
invoke rabbitmq
invoke celery_worker
invoke elasticsearch
invoke assets -dw
invoke server

Note that some or all of these commands will run attached to a console, and therefore the commands may need to be run in separate terminals. Be sure to activate your virtual environment each time a new terminal is opened. It is normal for the command to keep running!

Once started, you will be able to view the OSF in a web browser- by default, at http://127.0.0.1:5000/

In order to log in on your local server, you will also need to run the authentication server.

Running the API Server

If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local API server with the sequence delineated under running the OSF and:

invoke apiserver

Browse to localhost:8000/v2/ in your browser to go to the root of the browsable API. If the page looks strange, run python manage.py collectstatic to ensure that CSS files are deposited in the correct location.

Livereload support

You can run the OSF server in livereload mode with:

$ invoke server --live

This will make your browser automatically refresh whenever a code change is made.

Optional extras

Some functionality depends on additional services that will not be started using the sequence above. For many development tasks, it is sufficient to run the OSF without these services, except as noted below. Some additional installation will be needed to use these features (where noted), in which case updates will also need to be installed separately.

Authentication

An authentication server (either CAS or FakeCAS) must be available in order to log in to the OSF while running locally. This must be installed separately from the OSF. See running the OSF for details.

Waterbutler

Waterbutler is used for file storage features. Upload and download features will be disabled if Waterbutler is not installed. Consult the Waterbutler repository and documentation for information on how to set up and run this service.

Modular File Renderer

The Modular File Renderer (MFR) is used to render uploaded files to HTML via an iFrame so that they can be viewed directly on the OSF. Files will not be rendered if the MFR is not running. Consult the MFR repository for information on how to install and run the MFR.

Celery Beat

Normally you don't need to run celery_beat. If you work on tasks that are dispatched by celery_beat:

invoke celery_beat

Some beat-dispatched tasks require metrics and release requirements. If needed:

invoke requirements --metrics

Sharejs

ShareJS is used for collaborative editing features, such as the OSF wiki. It will be installed by the OSF installer script, but must be run separately. To run a local ShareJS server:

$ invoke sharejs

Downloading citation styles

To download citation styles, run:

$ invoke update_citation_styles

Installation

These instructions assume a working knowledge of package managers and the command line. For a detailed step-by-step walkthrough suitable for new programmers, consult the COS Development Docs. See optional extras for information about services not included in the automated install process below.

Pre-requisites

Before attempting to run OSF setup commands, be sure that your system meets the following minimum requirements.

Mac OS

The following packages must be installed before running the automatic setup script:

  • XCode command line tools (xcode-select --install)
  • Homebrew package manager (run brew update and brew upgrade --all before OSF install)
  • Java (if not installed yet, run brew install Caskroom/cask/java)
  • Python 2.7
    • pip
    • virtualenv (pip install virtualenv)
El Capitan and newer

If you are using Mac OS X >= 10.11 (El Capitan), you will also need to install OpenSSL headers and set some configuration:

brew install openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

Quickstart

Mac OS X

These instructions should work on Mac OSX >= 10.7

  • Clone the OSF repository to your computer. Change to that folder before running the commands below.
  • Create and activate your virtualenv.
virtualenv env
source env/bin/activate
  • Copy cp website/settings/local-dist.py to website/settings/local.py. NOTE: This is your local settings file, which overrides the settings in website/settings/defaults.py. It will not be added to source control, so change it as you wish.
  • Copy cp api/base/settings/local-dist.py to api/base/settings/local.py. NOTE: This is your local settings file, which overrides the settings in website/settings/defaults.py. It will not be added to source control, so change it as you wish.
$ cp website/settings/local-dist.py website/settings/local.py
$ cp api/base/settings/local-dist.py api/base/settings/local.py
  • On MacOSX with homebrew, there is a script that should automate much of the install process:
$ pip install invoke==0.13.0
$ invoke setup

To verify that your installation works, follow the instructions to start the OSF and run unit tests.

Additional configuration for Mac OS X

After running the automatic installer, you may find that some actions- such as running unit tests- fail due to an error with Mongo/ TokuMX. This can be resolved by increasing the system limits on number of open files and processes.

Add the following lines to /etc/launchctl.conf and/or /etc/launchd.conf (creating the files if necessary):

limit maxfiles 16384 16384
limit maxproc 2048 2048

Then create or edit either ~/.bash_profile or /etc/profile to include the following:

ulimit -n 2048

Then reboot.

Additional things to install

The automated installer does not install CAS, Waterbutler, or MFR, which may be needed to run some OSF features locally. Consult the optional extras section for more details.

Manual installation

At present, there is no complete automated install process for other platforms. Although the process above should perform most setup steps on Mac OS, users of other platforms will need to perform the steps below manually in a manner appropriate to their system. Some steps of the installer script can be re-used, in which case the appropriate commands are noted below.

On Mac OS, we recommend using Homebrew to install external dependencies.

  • Create local.py files for addons that need them (invoke copy_settings --addons)
  • Install TokuMX
  • Install libxml2 and libxslt (required for installing python lxml)
  • Install Java (if not already installed)
  • Install elasticsearch
  • Install python requirements (invoke requirements --dev --addons)
  • Install npm
  • Install node and bower packages
  • Build assets (invoke assets --dev)

  • If invoke setup hangs when 'Generating GnuPG key' (especially under linux), you may need to install some additional software to make this work. For apt-getters this looks like:

sudo apt-get install rng-tools

Followed by editing /etc/default/rng-tools to add the line:

HRNGDEVICE=/dev/urandom

And finally starting the rng-tools daemon with:

sudo /etc/init.d/rng-tools start

source: http://www.howtoforge.com/helping-the-random-number-generator-to-gain-enough-entropy-with-rng-tools-debian-lenny

Detailed installation and setup guides

Although some effort is made to provide automatic installation scripts, the following more detailed guides may be helpful if you are setting up the OSF on a machine already used for other development work, or if you wish to perform other advanced tasks. If the OSF is already working based on the instructions above, you can skip this section.

Using TokUMX

TokuMX is an open-source fork of MongoDB that provides support for transactions in single-sharded environments. TokuMX supports all MongoDB features as of version 2.4 and adds beginTransaction, rollbackTransaction, and commitTransaction commands.

Installing with Mac OS
$ brew tap tokutek/tokumx
$ brew install tokumx-bin
Installing on Ubuntu
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-key 505A7412
$ echo "deb [arch=amd64] http://s3.amazonaws.com/tokumx-debs $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tokumx.list
$ apt-get update
$ apt-get install tokumx
Installing on REHL/CentOS
$ sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
Migrating from MongoDB

TokuMX and MongoDB use different binary formats. To migrate data from MongoDB to TokuMX:

  • Back up the MongoDB data
    • invoke mongodump --path dump
  • Shut down the MongoDB server
  • Uninstall MongoDB
  • Install TokuMX (see instructions above)
  • Restore the data to TokuMX
    • invoke mongorestore --path dump/osf20130903 --drop
  • Verify that the migrated data are available in TokuMX

Using Celery

Installing Celery + RabbitMQ
  • Install RabbitMQ. On MacOSX with homebrew,
$ brew update
$ brew install rabbitmq

The scripts are installed to /usr/local/sbin, so you may need to add PATH=$PATH:/usr/local/sbin to your .bash_profile.

For instructions for other OS's, see the official docs.

Then start the RabbitMQ server with

$ invoke rabbitmq

If you want the rabbitmq server to start every time you start your computer (MacOSX), run

$ ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
Starting A Celery Worker
invoke celery_worker

Search

Install Elasticsearch to use search features.

Mac OSX
$ brew install elasticsearch@1.7

note: Oracle JDK 7 must be installed for elasticsearch to run

Ubuntu
$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb
$ sudo dpkg -i elasticsearch-1.2.1.deb
REHL/CentOS
$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.2.1.noarch.rpm
$ sudo rpm -ivh elasticsearch-1.2.1.noarch.rpm
Using Elasticsearch
  • In your website/settings/local.py file, set SEARCH_ENGINE to 'elastic'.
SEARCH_ENGINE = 'elastic'
  • Start the Elasticsearch server and migrate the models.
$ invoke elasticsearch
$ invoke migrate_search
Starting a local Elasticsearch server
$ invoke elasticsearch

NPM

The Node Package Manager (NPM) is required for installing a number of node-based packages.

# For MacOSX
$ brew update && brew install node

Installing Node on Ubuntu is slightly more complicated. Node is installed as nodejs, but Bower expects the binary to be called node. Symlink nodejs to node to fix, then verify that node is properly aliased:

# For Ubuntu
$ sudo apt-get install nodejs
$ sudo ln -s /usr/bin/nodejs /usr/bin/node
$ node --version      # v0.10.25

Common Development Tasks

Running the shell

To open the interactive Python shell, run:

$ invoke shell

Running Tests

To run all tests:

$ invoke test --all

To run a certain test method

$ nosetests tests/test_module.py:TestClass.test_method

Run OSF Python tests only:

$ inv test_osf

Run addons Python tests only:

$ inv test_addons

Run Javascript tests:

$ inv karma

By default, inv karma will start a Karma process which will re-run your tests every time a JS file is changed. To do a single run of the JS tests:

$ inv karma --single

By default, Karma will run tests using a PhantomJS headless browser. You can run tests in other browsers like so:

$ inv karma -b Firefox

If you want to run cross browser tests with SauceLabs, use "sauce" parameter:

$ inv karma --sauce

Testing Addons

Addons tests are not run by default. To execute addons tests, run

$ invoke test_addons

Testing Email

First, set MAIL_SERVER to localhost:1025 in you local.py file.

website/settings/local.py

...
MAIL_SERVER = "localhost:1025"
...

Sent emails will show up in your server logs.

Optional: fire up a pseudo-mailserver with:

$ invoke mailserver -p 1025

Building front-end assets with webpack

Use the following command to update your requirements and build the asset bundles:

$ inv assets -dw

The -w option puts you in "watch" mode: the script will continue running so that assets will be built when a file changes.

Setting up the Admin Module

  • See admin/README.md for information on how to set up the OSF Admin module

Getting application credentials

Many addons require application credentials (typically an app key and secret) to be able to authenticate through the OSF. These credentials go in each addon's local.py settings file (e.g. website/addons/dropbox/settings/local.py).

COS is Hiring!

Want to help save science? Want to get paid to develop free, open source software? Check out our openings!