The NGEE-Tropics archive service supports the NGEE Tropics project. It is a place to upload datasets and metadata. Generated documentation (http://ngeet.github.io/ngt-archive/)
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
archive_api
docs
licenses
ngt_archive
ui
.gitignore
LICENSE.md
MANIFEST.in
README.md
Vagrantfile
circle.yml
doi.html
main.copyme
maintenance.html
manage.py
ngt_archive_init.jinja2
ngt_archive_uwsgi_conf.jinja2
settings_local_py.jinja2
setup.py
vagrant.yml
virtualhost.conf

README.md

NGEE Tropics Archive Service

NGEE Tropics Archive Service is a Django application. The NGEE-Tropics archive service sits in the middle of the NGEE Tropics data workflow

  • Place to upload datasets and metadata.
  • Datasets should consist of related data collections, not individual files (e.g. Sapflow collected at Manaus)
  • Datasets will receive a DOI, and can be updated with new data.
  • Datasets will be available for search to NGEE Tropics team, and (if policy allows) to public
  • Makes NGEE-Tropics compliant with DOE Data Management Plan

Development Practices

  • NGEE Tropics Archive Service will be using the cactus model of branching and code versioning in git.
  • Code development will be peformed in a forked copy of the repo. Commits will not be made directly to the ngt-archive repo. Developers will submit a pull request that is then merged by another team member, if another team member is available.
  • Each pull request should contain only related modifications to a feature or bug fix.
  • Sensitive information (secret keys, usernames etc) and configuration data (e.g database host port) should not be checked in to the repo.
  • A practice of rebasing with the main repo should be used rather that merge commmits.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisities

NGEE Tropics Archive is a Django application which requires:

  • Python (>= 3.4, 3.5)
  • Django (> 1.8)
  • Platform (Mac, Linux)

###Setup Development There is an option for local machine development and virtual machine development using vagrant.

Desktop

Use these instructions for setting up development on a desktop computer.

Fork the repository and then clone your fork:

# installation instructions here
git clone git@github.com:<your username here>/ngt-archive.git
cd ngt-archive

Create a virtual environment for development

virtualenv -p /Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4 .menv
source .menv/bin/activate

Install the a django project for development

python setup.py develop
./manage.py collectstatic
./manage.py migrate
./manage.py loaddata test_auth.json test_archive_api.json

Run the development server. Test users/passes are: superadmin/ngeet2016, admin/ngeetdata, auser/ngeetdata.

./manage.py runserver

Virtual Machine Development

These instructions assume that you have Vagrant installed. The default setup is Vagrant with VirtualBox. If you would like to manually prepare you own VM look at the Vagrantfile in the root of this project.

Create main.yml

Copy main.copyme as main.yml and put your sensitive information in there. If you don't know what this is, ask another developer.

Vagrant VM Configuration

If you are using at VirtualBox and Vagrant a development VM can be configured using vagrant.

# installation instructions here
git clone git@github.com:NGEET/ngt-archive.git
cd ngt-archive

The next command will take a while because it will be configuring the box for the first time.

$ vagrant up

Once that finishes you may login to your VM with the following command. The project directory is mounted into the VM at /vagrant.

$ vagrant ssh
$ cd /vagrant
$ sudo ansible-playbook vagrant.yml

The web application has been deployed to apache on your VM. Use the ngt_archive service on ubuntu. This service starts up at http://0.0.0.0:9999

sudo initctl start ngt_archive
sudo initctl stop ngt_archive
sudo initctl list | grep ngt_archive

When you are done for the day, you may shut your VM down:

$ vagrant halt

To delete your VM:

$ vagrant destroy

Local Machine Development

Install NGEE Tropics Archive Service for development

Clone the project from Github

git clone git@github.com:NGEET/ngt-archive.git
cd ngt-archive

Prepare a Python virtual environment

virtualenv .env  OR virtualenv -p python3 .env
source .env/bin/activate

Install ngt-archive for development

python setup.py develop

Create the database and load some data

./manage.py migrate
./manage.py createsuperuser

Load Test Users superadmin, admin, auser. Passwords are ngeet2016, ngeetdata, ngeetdata respectively.

./manage.py loaddata test_auth.json 

Load Archive Service Test Data

./manage.py loaddata test_archive_api.json 

Run a develop server

./manage.py runserver  0.0.0.0:8888
Performing system checks...

System check identified no issues (0 silenced).
August 05, 2016 - 23:48:34
Django version 1.9.8, using settings 'wfsfa_broker.settings'
Starting development server at http://127.0.0.1:8888/
Quit the server with CONTROL-C.

Running the tests

Automated tests are run using manage.py:

./manage.py test

Deployment

Guidelines for preparing the application for deployment. Database and operating system are up to the user.

Prepare django application distribution for deployment.

$ python setup.py sdist
Writing ngt_archive-<version>/setup.cfg
Creating tar archive
removing 'ngt_archive-<version>' (and everything under it)

Create deployment directory with a Python 3 virtual environment

$ mkdir <deploy_dir>
$ cd <deploy_dir>
$ virtualenv -p python3 .

Install NGT Archive service and its dependencies.

$ <deploy_dir>/bin/pip install ngt_archive-<version>.tar.gz
$ <deploy_dir>/bin/pip install psycopg2 (For Postgres DB)

Link to the Django applications manange.py script

$ cd <deploy_dir>
$ ln -s lib/python3.4/site-packages/manage.py manage.py

Create custom Django settings in <deploy_dir>/settings/local.py. Use settings_local_py.jinja2 as an example. Replace template variables in curly braces with your configuration.

Initialize the application

$ <deploy_dir>/manage.py migrate
$ <deploy_dir>/manage.py collectstatic

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Workflow for tagging and building release:

  1. checkout the version to tag from master
  2. git tag -a v[version]-[release] -m "Tagging release v[version]-[release]"
  3. build distribution with setup.py
  4. git push origin v[version]-[release]

Authors

  • Charuleka Varadharajan - LBL
  • Valerie Hendrix - LBL
  • Megha Sandesh - [LBL]
  • Deb Agarwal - LBL

See also the list of contributors who participated in this project.

License

See LICENSE.md file for details

Copyright Notice

NGEE Tropics Archive (NGT Archive) Copyright (c) 2017, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at IPO@lbl.gov.

NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so.