Skip to content
This repository

COI R2 services based on pyon

Merge pull request #2029 from MauriceManning/936_rebase

OOIION-936 Deployments: refinement of ops
latest commit 23b7bc4fd8
Maurice Manning MauriceManning authored April 23, 2014
Octocat-spinner-32 examples Fixed some test issues March 17, 2013
Octocat-spinner-32 extern bump ion-defs April 23, 2014
Octocat-spinner-32 ion typo2 April 23, 2014
Octocat-spinner-32 scripts Update elasticsearch head plugin download location August 23, 2013
Octocat-spinner-32 test_data merge conflict resolution April 09, 2014
Octocat-spinner-32 .coveragerc Exclude h5debug from coverage September 06, 2013
Octocat-spinner-32 .gitignore Added deployment to gitignore March 31, 2014
Octocat-spinner-32 .gitmodules Remove mi submodule September 12, 2013
Octocat-spinner-32 Doxyfile Turn off call/caller graph in doxygen January 26, 2012
Octocat-spinner-32 README OOIION-1789: Enhance tool to delete data resources and coverages March 25, 2014
Octocat-spinner-32 README.OOI_hosted_VM updated sysname February 20, 2014
Octocat-spinner-32 README.OOI_virtualbox_VM Updated readme's February 20, 2014
Octocat-spinner-32 README.postgres_setup updated sysname February 20, 2014
Octocat-spinner-32 README_DEMO OOIION-1788: DataProduct supporting PDICT from agent mapped agents April 08, 2014
Octocat-spinner-32 ape-buildout.cfg Fix buidout version pinning on gevent and coverage October 10, 2012
Octocat-spinner-32 autolaunch.cfg Fix numpy trouble October 10, 2012
Octocat-spinner-32 Buildout 2.2.0 bootstrap July 11, 2013
Octocat-spinner-32 build.xml Adds new submodule ion-functions March 08, 2013
Octocat-spinner-32 builder.cfg Try pypi simple index February 14, 2013
Octocat-spinner-32 buildout.cfg CIDEVSTC-187: Initial implementation February 28, 2014
Octocat-spinner-32 Fix typo April 18, 2013
Octocat-spinner-32 gcoverage_buildout.cfg Removes breakpoint and updates submodule references June 06, 2013
Octocat-spinner-32 obj Adding support for computed and extended attributes June 20, 2012
Octocat-spinner-32 pyon_buildout.cfg Adds new submodule ion-functions March 08, 2013
Octocat-spinner-32 pyon_mi_buildout.cfg Adds new submodule ion-functions March 08, 2013
Octocat-spinner-32 res Added symlinks for res, obj; removed reslocal November 01, 2011
Octocat-spinner-32 setup.cfg Revert "Add time stamp to nose log capture" September 14, 2012
Octocat-spinner-32 CIDEVSTC-187: Initial implementation February 28, 2014
Octocat-spinner-32 unittest Unittest script November 07, 2011
Ocean Observatories Initiative Cyberinfrastructure (OOI CI)
Integrated Observatory Network (ION) - OOINet

coi-services - OOINet system and services

(C) UC Regents, 2010-2014

This is the repository that defines the OOINet with its services and agents.
The OOINet system is started from this repository.
Submodules provide additional required functionality, such as the Pyon
Capability Container.


This assumes basic development environment setup (git, directory structure).
Please follow the "New Developers Tutorial" for basic steps.

The Pyon Capability Container is packaged as a submodule. All dependencies are
installed during the process described below. For details, refer to:

Install the following if not yet present (brew install instructions are for Mac only):

* OS Packages and package management:
For Mac, use homebrew
    > /usr/bin/ruby <(curl -fsSkL
- python 2.7.x
    > brew install python --framework --universal
    Note: see 'brew doctor' for any path issues
- git 1.7.7 or higher: Use brew or download the Mac or Linux installer and run it
    > brew install git

- Install core packages with Homebrew

    > brew install libevent libyaml zeromq rabbitmq hdf5 pkg-config netcdf udunits

    Linux: Note that many installs have much older versions installed by default.
    (If you have trouble with brew 'MD5 mismatch' errors, try running 'brew update'
    and try the install again)
    For any kind of installation trouble run 'brew doctor' and fix any issues.

- Install Postgres with Homebrew

    > brew install postgres postgis

    If you have trouble with the installation, please see:

    As user postgres:
    > createuser -s ion

- Install libgswteos-10 for marine science calculations
    > brew tap lukecampbell/homebrew-libgswteos
    > brew install libgswteos-10
    > brew test -v libgswteos-10
- Install data processing packages (Needed by udunits and scipy)
    > brew install swig (Make sure brew installed swig is used. `which swig`)
    > brew install gfortran

* Python packages and environment management:
- pip
    > easy_install pip
- virtualenv and virtualenvwrapper modules for your python 2.7 installation
    > easy_install --upgrade virtualenv
    > easy_install --upgrade virtualenvwrapper
    Note: This require Mac's XCode with installed command line tools

* Setup a virtualenv to run COI-services:
    > mkvirtualenv --no-site-packages coi
    Note: Do not use the pyon virtualenv if you are a pyon developer

* Pin setuptools in your virtualenv
    > pip install -U setuptools==0.8

* Install numpy to the virtualenv
    > pip install numpy==1.7.1


Clone coi-services (this repository, if you haven't already) in your code directory:
    > cd code   # Whatever your code repository directory is
    > git clone

Initialize submodules::
    > cd coi-services
    > git submodule update --init

(see Pyon README for hints with submodule anonymous checkout and password-less push)


Use buildout to install the python package dependencies with:
    > python -v 2.2.0
    > bin/buildout

** Add following to your .bashrc profile on Macs if buildout fails to
    compile for packages such as gevent, pyml, etc.
    export C_INCLUDE_PATH=/usr/local/include

** If matplotlib fails to build due to not finding the free type headers, try:
    if /usr/X11/lib/pkgconfig has a file named freetype2.pc, then add the
    following line to your .bash_profile
    export PKG_CONFIG_PATH=/usr/X11/lib/pkgconfig

    Mac should have libpng and libfreetype headers, which are required
    for matplotlib python build.  If for some reason your buildout fails on
    matplotlib due to missing libpng libraries, try install from here:

    For additional support to install matplotlib on MacOS X 10.7 (Lion) see:

Generate service definition interfaces and abstract base classes files:
    > bin/generate_interfaces
    Note: These will be placed in the interfaces/ subdirectory.



Run postgresql in second terminal before starting the container or running tests:
    > /usr/local/bin/postgres -D /usr/local/var/postgres

    > rabbitmq-server


Add the current directory to your PYTHONPATH for an easier time running
scripts. Add the following line to your ~/.profile (create it if it
doesn't exist, or use bash_profile if you already have that):
    export PYTHONPATH=.


Provide local configuration overrides (e.g. broker host, debug levels) in:
    > res/config/pyon.local.yml
    > res/config/logging.local.yml
    Note: DO NOT EDIT the standard configuration and logging config files.

List supported Pyon Capability Container arguments:
    > bin/pycc --help

Run unit tests:
    > bin/nosetests -v -a UNIT

Run smoke tests to verify sanity of the system:
    > bin/nosetests -v -a SMOKE

Run full integration tests - this will take a long time so may be optional:
    > bin/nosetests -v -a INT

Start development instance of the ION system with all services:
    > bin/pycc --rel res/deploy/r2deploy.yml


To reset the system use:
    > bin/pycc -X -fc -bc

Outdated *.pyc files can causes your
tests to fail unexpectedly.  Do an ant clean and try again.
    > ant clean
    > bin/generate_interface

If your tests still fail, you might have some incorrect values in
pyon.local.yml or logging.local.yml.  Check with other developers.

If your tests fail unexpectedly when you have tried everything else, your build
might be corrupted.
    > deactivate your existing virtualenv
    > rm and create the virtualenv again
    > re-run and buildout
    > If you suspect an egg is bad, remove the suspected egg from the eggs/ dir. Re-run buildout.

See also:


Use Pycharm or any other Python IDE to debug; breakpoints and step-by-step
are possible. Set the executable to the bin/pycc and set the working directory
to the git repository root. Use the -n option to avoid the interactive IPython
shell--it does not work properly with the debugger.

Start bin/pycc or bin/nosetests scripts in the debugger, with arguments.

PyCharm may need to be enabled to run with gevent. Google how to do this.

Remote debugging of services in a running container is possible too. Setup a
remote debugging configuration to use the localhost and port 8585. First start
the remote debugging server, then start the container from the command line
below using the pyccd script (the -n seems to be required):
    > bin/pyccd -n --rel res/deploy/xxx.yml


You can develop services locally in this repository.
Please follow the following steps as long as you are new:

Get the latest code before you start editing, or anytime you want:
    > git pull --rebase
    (fix any issues, git add, git rebase --continue)

    > git submodule update  # Do NOT forget. This does not happen automatically
    See below for an automated approach using git submodule hooks.

As extern submodule pointers change, service interfaces can change and old pyc files
can be left around unintentinally. Generate interfaces frequently (especially in case of error):
    > ant clean
    > bin/generate-interfaces

Note: The res/ and obj/ dirs are symlinks to a subdirectory in a git submodule.
Beware of the pitfalls of git submodule. You need to treat it as a separate GIT
module. In case of changes, both GIT modules must be pushed, submodule first:
    > cd extern/ion-definitions
    > git checkout master
    > do edits...
    > git status            # Just to see what's going on
    > git commit -am "Something smart"
    > git push origin master
    > cd ../..              # To the root of coi-services
    > git commit -am "Something smarter"
    > git push

Put your services in ion/services/<subsystem>/... (subdirectories are allowed).

The ion-definition submodules approach above applies to other submodules as well.


A git hook is a script that executes during various points of using git.
Git hooks exist automating submodules for most people. See the steps here:

They do require an initial setup. Simple instructions:

Clone this repository:
    > cd /some/tmp/directory
    > git clone

Use the provided install script:
    > sh gitutils/submodule-hooks/ /path/to/your/pyon/or/coi-services/dir

Now, when checking out a branch, pulling, merging etc, git will prompt you to
automatically update if it notices a change to the commit that your supermodule
points to.

The pre-commit script is so you don't forget to push changes to the submodule
*BEFORE* you push changes to the supermodule.

Instrument Development Kit (IDK)
There are now 4 configuration options in a new config file res/config/idk.yml
The only option you are likely to need to override is the driver_path,
currently set to ion/code. Override it in idk.local.yml to the parent directory
of coi-services, pyon, and marine-integrations if they are located elsewhere.

Use driver_path: OOI
If you have the following structure.
Something went wrong with that request. Please try again.