OSF SciNet - Crowd sourcing academic citations
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs
scinet
test
.gitignore
LICENSE
README.md
requirements.txt
setup.py

README.md

OSF SciNet -- Crowd-Sourcing the Scientific Citation Network

What is SciNet?

SciNet is a open source, collaborative project being developed at the Center for Open Science. The primary purpose of Scholarly is to provide the public with a free, open, and comprehensive dataset containing meta-data for academic citations as well as corresponding references. This dataset will allow the public to access, analyze, and distribute academic citation meta-data without restriction.

Relevant Wiki Pages:

Getting started with SciNet

Note: This set of packages is not production ready, but it should be robust enough for some early development. Additionally, these instructions were prepared using a Digial Ocean server running Ubuntu 12.04 but they can easily be adapted for running locally or on another cloud service.

  1. Create your droplet: http://digitalocean.com

  2. Check your email and make note of: IP Address of your droplet Username created for your droplet Password for your user

  3. SSH into your droplet:

  4. Open up a terminal and type:

     ssh <username>@<ip address>
    

    enter the password located in the email

  5. Set up a local (non-root) account:

  6. Create the user account:

     adduser <username>
    
  7. Give it sudo access:

     visudo
     # Below "root ALL=(ALL:ALL) All" enter
     <username> ALL=(ALL:ALL) ALL
     # Save the file and exit
    
  8. Logout of your current account and SSH into the server with the new one:

  9. From your terminal enter:

     exit
     ssh <username>@<ip address>
    
  10. Create a directory to hold our data:

  11. From your terminal enter:

     sudo mkdir /vol
     sudo chown <username>:<username> /vol
     cd /vol
    
  12. Install necessary packages (Note: This can take a minute):

  13. From your terminal enter:

     sudo apt-get update
     sudo apt-get install git python-flask python-pip build-essential python-dev python-requests
     sudo apt-get install python-pyquery uwsgi-plugin-python uwsgi nginx
     sudo pip install pymongo reppy nameparser uwsgi
    
  14. Clone and install SciNet:

  15. Checkout the repo:

     # Unless you have a forked SciNet use: https://github.com/centerforopenscience/scinet.git
     git clone https://github.com/user_name/repo.git
    
  16. Install required libraries:

     cd /vol/scinet
     sudo pip install -r requirements.txt
    

Note: If you are installing on Ubuntu, remove the lxml requirement from requirements.txt before running pip install.

  1. Test it:

     cd /vol/scinet/scinet
     python main.py
    

You should get a pymongo "Connection Refused" error.

  1. Install and setup mongodb:

  2. Add the 10gen repo:

     sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
     echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list
     sudo apt-get update
     sudo apt-get install mongodb-10gen
    
  3. Edit /etc/mongodb.conf and change dbpath=/var/lib/mongodb to dbpath=/vol/mongodb, and update permissions:

     sudo mkdir /vol/mongodb
     sudo chown -R mongodb /vol/mongodb
    
  4. Restart mongo:

     sudo service mongodb restart
    
At this point, you should be able to (optionally) restart the instance and run:

    python /vol/scinet/scinet/main.py

without errors.
  1. Get Citelet:

  2. Checkout the repo:

     cd /vol
     # Unless you have forked scinet-citelet use: https://github.com/centerforopenscience/scinet-citelet.git
     git clone https://github.com/user_name/repo.git
    
  3. Install required libraries::

     cd /vol/citelet
     sudo pip install -r requirements.txt
    
  4. Build Citelet:

  5. Verify config file is pointing to your desired urls:

     vi /vol/citelet/cfg.py
    

For more information visit the Citelet docs.

  1. Run the fabric build process:

     fab rsudo deploy
    
  2. Test it:

     cd /vol/citelet/app
     python main.py
    
  3. Setup boto config so our keys will be loaded automatically:

  4. sudo vi /etc/boto.cfg:

     [Credentials]
     aws_access_key_id = <your access key>
     aws_secret_access_key = <your secret access key>
    
  5. Install and setup NGINX and uWSGI:

  6. Configure NGINX, for example, replace /etc/nginx/sites-available/default with:

     server {
         listen   80;
               server_name scinet;
           # scinet endpoint
             location /scinet {
                 uwsgi_pass unix:///tmp/scinet.sock;
             include uwsgi_params;
             # strip path before handing it to app
             uwsgi_param SCRIPT_NAME /scinet;
             uwsgi_modifier1 30;
         }
         # citelet endpoint
             location /citelet {
                 uwsgi_pass unix:///tmp/citelet.sock;
             include uwsgi_params;
             # strip path before handing it to app
             uwsgi_param SCRIPT_NAME /citelet;
             uwsgi_modifier1 30;
         }
     }
    
  7. Finally symlink it to its sites-enabled folder:

     sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default
    
  8. Configure UWSGI sockets, for example, create /etc/uwsgi/apps-available/scinet.ini and populate it with:

     [uwsgi]
     chdir = /vol/scinet/scinet
     uid = www-data
     gid = www-data
     chmod-socket = 666
     socket = /tmp/scinet.sock
     module = app
     callable = app
    
  9. And, create /etc/uwsgi/apps-available/citelet.ini and populate it with:

     [uwsgi]
     chdir = /vol/citelet/app
     uid = www-data
     gid = www-data
     chmod-socket = 666
     socket = /tmp/citelet.sock
     module = main
     callable = app
    
  10. Finally symlink them to their respective enabled folders:

     sudo ln -s /etc/uwsgi/apps-available/scinet.ini /etc/uwsgi/apps-enabled/scinet.ini
     sudo ln -s /etc/uwsgi/apps-available/citelet.ini /etc/uwsgi/apps-enabled/citelet.ini
    
  11. Enable the app and restart:

     sudo service nginx restart
     sudo service uwsgi restart
    

The site should now be up and running. You can, for instance, install lynx and visit Test the site by visiting, for example, http:///scinet -or- http:///citelet.

  1. Still on the to-do list:
  • Pushing the site via git, with automatic server restarts, etc., implemented as git post commit hooks.
  • Automating the server setup process via the tools that Jeff and Lindsy were talking about.