master
Build Status:develop
Build Status:- Public Repo: https://github.com/CenterForOpenScience/osf.io/
- Issues: https://github.com/CenterForOpenScience/osf.io/issues?state=open
- COS Development Docs: http://cosdev.readthedocs.org/
The COS Development Docs provide detailed information about all aspects of OSF development. This includes detailed installation instructions, a list of common setup errors, and other troubleshooting.
The OSF invoke
script provides several useful commands. For more information, run:
invoke --list
If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local test server with the following sequence:
invoke mongo -d # Runs mongod as a daemon
invoke mailserver
invoke rabbitmq
invoke celery_worker
invoke elasticsearch
invoke assets -dw
invoke server
Note that some or all of these commands will run attached to a console, and therefore the commands may need to be run in separate terminals. Be sure to activate your virtual environment each time a new terminal is opened. It is normal for the command to keep running!
Once started, you will be able to view the OSF in a web browser- by default, at http://127.0.0.1:5000/
You can run the app server in livereload mode with:
$ invoke server --live
This will make your browser automatically refresh whenever a code change is made.
Some functionality depends on additional services that will not be started using the sequence above. For most development tasks, it is sufficient to run the OSF without these services, except as noted below. No additional installation is needed to use these features.
To download citation styles, run:
$ invoke update_citation_styles
ShareJS is used for collaborative editing features, such as the OSF wiki. To run a local ShareJS server:
$ invoke sharejs
Waterbutler is used for file storage features. Upload and download features will be disabled if Waterbutler is not installed. Consult the Waterbutler repository and documentation for information on how to set up and run this service.
These instructions assume a working knowledge of package managers and the command line. For a detailed step-by-step walkthrough suitable for new programmers, consult the COS Development Docs.
Before attempting to run OSF setup commands, be sure that your system meets the following minimum requirements.
The following packages must be installed before running the automatic setup script:
- XCode command line tools (
xcode-select --install
) - Homebrew package manager (run
brew update
andbrew upgrade --all
before OSF install) - Java (if not installed yet, run
brew install Caskroom/cask/java
) - Python 2.7
- pip
- virtualenv (
pip install virtualenv
)
These instructions should work on Mac OSX >= 10.7
- Clone the OSF repository to your computer. Change to that folder before running the commands below.
- Create and activate your virtualenv.
virtualenv env
source env/bin/activate
- Copy
website/settings/local-dist.py
towebsite/settings/local.py.
NOTE: This is your local settings file, which overrides the settings inwebsite/settings/defaults.py
. It will not be added to source control, so change it as you wish.
$ cp website/settings/local-dist.py website/settings/local.py
- On MacOSX with homebrew, there is a script that should automate much of the install process:
$ pip install invoke
$ invoke setup
To verify that your installation works, follow the instructions to start the OSF and run unit tests.
After running the automatic installer, you may find that some actions- such as running unit tests- fail due to an error with Mongo/ TokuMX. This can be resolved by increasing the system limits on number of open files and processes.
Add the following lines to /etc/launchctl.conf
and/or /etc/launchd.conf
(creating the files if necessary):
limit maxfiles 16384 16384
limit maxproc 2048 2048
Then create or edit either ~./bash_profile
or /etc/profile
to include the following:
ulimit -n 2048
Then reboot.
The automated installer does not install Waterbutler, which may be needed to run some OSF features locally. Consult the Waterbutler repository for setup instructions.
At present, there is no complete automated install process for other platforms. Although the process above should perform most setup steps on Mac OS, users of other platforms will need to perform the steps below manually in a manner appropriate to their system. Some steps of the installer script can be re-used, in which case the appropriate commands are noted below.
On Mac OS, we recommend using Homebrew to install external dependencies.
-
Create local.py files for addons that need them (
invoke copy_settings --addons
) -
Install TokuMX
-
Install libxml2 and libxslt (required for installing python lxml)
-
Install Java (if not already installed)
-
Install elasticsearch
-
Install GPG
-
Install python requirements (
invoke requirements --dev --addons
) -
Create a GPG key (
invoke encryption
) -
Install npm
-
Install node and bower packages
-
Build assets (
invoke assets --dev
) -
If invoke setup hangs when 'Generating GnuPG key' (especially under linux), you may need to install some additional software to make this work. For apt-getters this looks like:
sudo apt-get install rng-tools
Followed by editing /etc/default/rng-tools
to add the line:
HRNGDEVICE=/dev/urandom
And finally starting the rng-tools daemon with:
sudo /etc/init.d/rng-tools start
Although some effort is made to provide automatic installation scripts, the following more detailed guides may be helpful if you are setting up the OSF on a machine already used for other development work, or if you wish to perform other advanced tasks. If the OSF is already working based on the instructions above, you can skip this section.
TokuMX is an open-source fork of MongoDB that provides support for transactions in single-sharded environments.
TokuMX supports all MongoDB features as of version 2.4 and adds beginTransaction
, rollbackTransaction
, and
commitTransaction
commands.
$ brew tap tokutek/tokumx
$ brew install tokumx-bin
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-key 505A7412
$ echo "deb [arch=amd64] http://s3.amazonaws.com/tokumx-debs $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tokumx.list
$ apt-get update
$ apt-get install tokumx
TokuMX and MongoDB use different binary formats. To migrate data from MongoDB to TokuMX:
- Back up the MongoDB data
invoke mongodump --path dump
- Shut down the MongoDB server
- Uninstall MongoDB
- Install TokuMX (see instructions above)
- Restore the data to TokuMX
invoke mongorestore --path dump/osf20130903 --drop
- Verify that the migrated data are available in TokuMX
- Install RabbitMQ. On MacOSX with homebrew,
$ brew update
$ brew install rabbitmq
The scripts are installed to /usr/local/sbin
, so you may need to add PATH=$PATH:/usr/local/sbin
to your .bash_profile
.
For instructions for other OS's, see the official docs.
Then start the RabbitMQ server with
$ invoke rabbitmq
If you want the rabbitmq server to start every time you start your computer (MacOSX), run
$ ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist
invoke celery_worker
Install Elasticsearch to use search features.
$ brew install elasticsearch
note: Oracle JDK 7 must be installed for elasticsearch to run
$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb
$ sudo dpkg -i elasticsearch-1.2.1.deb
- In your
website/settings/local.py
file, setSEARCH_ENGINE
to 'elastic'.
SEARCH_ENGINE = 'elastic'
- Start the Elasticsearch server and migrate the models.
$ invoke elasticsearch
$ invoke migrate_search
$ invoke elasticsearch
The Node Package Manager (NPM) is required for installing a number of node-based packages.
# For MacOSX
$ brew update && brew install node
Installing Node on Ubuntu is slightly more complicated. Node is installed as nodejs
, but Bower expects
the binary to be called node
. Symlink nodejs
to node
to fix, then verify that node
is properly aliased:
# For Ubuntu
$ sudo apt-get install nodejs
$ sudo ln -s /usr/bin/nodejs /usr/bin/node
$ node --version # v0.10.25
To open the interactive Python shell, run:
$ invoke shell
To run all tests:
$ invoke test --all
To run a certain test method
$ nosetests tests/test_module.py:TestClass.test_method
Run OSF Python tests only:
$ inv test_osf
Run addons Python tests only:
$ inv test_addons
Run Javascript tests:
$ inv karma
By default, inv karma
will start a Karma process which will re-run your tests every time a JS file is changed.
To do a single run of the JS tests:
$ inv karma --single
By default, Karma will run tests using a PhantomJS headless browser. You can run tests in other browsers like so:
$ inv karma -b Firefox
If you want to run cross browser tests with SauceLabs, use "sauce" parameter:
$ inv karma --sauce
Addons tests are not run by default. To execute addons tests, run
$ invoke test_addons
First, set MAIL_SERVER
to localhost:1025
in you local.py
file.
website/settings/local.py
...
MAIL_SERVER = "localhost:1025"
...
Sent emails will show up in your server logs.
Optional: fire up a pseudo-mailserver with:
$ invoke mailserver -p 1025
Use the following command to update your requirements and build the asset bundles:
$ inv assets -dw
The -w option puts you in "watch": assets will be built when a file changes.
Many addons require application credentials (typically an app key and secret) to be able to authenticate through the
OSF. These credentials go in each addon's local.py
settings file (e.g. website/addons/dropbox/settings/local.py
).
For local development, the COS provides test app credentials for a number of services.