Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

COI R2 services based on pyon

This branch is 19 commits ahead and 8055 commits behind master

Fetching latest commit…

Cannot retrieve the latest commit at this time

README
===========================================================
Ocean Observatories Initiative Cyberinfrastructure (OOI CI)
Integrated Observatory Network (ION)

coi-servives - ION COI services

(C) UC Regents, 2010-2011
===========================================================

This is the repository that defines the COI services.
Note: Initially, all ION services are defined here and later moved to subsystem repositories.

COI services are intended to be deployed as part of the ION system launch.
Services are deployed in pyon capability containers.

INSTALL
=======

This assumes basic development environment setup (git, directory structure). Please follow the
"New Developers Tutorial" for basic steps.

Pyon: The main dependency of this repository is the pyon Capability Container. Follow the listed
steps to install the minimal needed dependencies to run pyon on a Mac. For more details and Linux
install instructions, check out the pyon README: https://github.com/ooici/pyon/blob/master/README


Install the following if not yet present:

* OS Packages and package management:
For Mac, use homebrew
    > /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"
- python 2.7.2
- git 1.7.7: Use brew or download the Mac or Linux installer and run it
- couchdb 1.1.0 (optional if memory mockdb is used)
    You will need to upgrade couchdb to at least 1.1.0.
- rabbitmq 2.6.1 or later (recommended, but can use rabbitmq on amoeba)
    Use brew.
    Alternative: download generic Linux version and unpack into a suitable directory.
- Install libevent, libyaml, zeromq, couchdb, python, and rabbitmq with Homebrew
    > brew install libevent libyaml zeromq couchdb python rabbitmq hdf5
    You can even reinstall git using brew to clean up your /usr/local directory
    Be sure to read the pyon README for platform specific guidance to installing
    dependent libraries and packages.
    Linux: Note that many installs have much older versions installed by default.
    (If you have trouble with brew 'MD5 mismatch' errors, try running 'brew update'
    and try the install again)
- Install matplotlib using pip:
    pip install matplotlib

- Install


* python packages and environment management:
- pip
    > easy_install pip
- virtualenv and virtualenvwrapper modules for your python 2.7 installation
    > easy_install --upgrade virtualenv
    > easy_install --upgrade virtualenvwrapper
    Note: This may require Mac's XCode (use XCode 3.3 free version)

* Setup a virtualenv to run COI-services (use any name you like):
    > mkvirtualenv --no-site-packages --python=python2.7 coi
    Note: Do not use the pyon virtualenv if you are a pyon developer


PACKAGE DOWNLOAD
================

Clone pyon and coi-services (if you haven't already) in your code directory:
    > cd code   # Whatever your code repository directory is

Clone pyon (needed for now):
    > git clone git@github.com:ooici/pyon.git

Clone coi-services (this repository, if you haven't already):
    > git clone git@github.com:ooici/coi-services.git

Link git submodule ion-definitions, containing object and service definitions and resource files:
    > cd coi-services
    > git submodule update --init

(see Pyon README for hints with submodule anonymous checkout and password-less push)


PACKAGE DEPENDENCIES
====================

Use Buildout to install the python package dependencies (including pyon itself) with:
    > python bootstrap.py
    > bin/buildout

Generate service definition interfaces and abstract base classes files:
    > bin/generate_interfaces
    Note: These will be placed in the interfaces/ subdirectory (and not in GIT nor in the egg).


RUN-TIME DEPENDENCIES
=====================

COUCHDB (OBJECT DATA STORE):

Run couchdb in second terminal before starting the container or running tests:
Note: optional. An in-memory mockdb is used by default and couchdb unit-tests are skipped.
    > couchdb


RABBITMQ (MESSAGE BROKER):

Run RabbitMQ in second terminal before starting the container or running tests:
Note: optional. RabbitMQ on amoeba us used by default for new developers.
    > cd rabbitmq_server-2.x.x
    > sudo sbin/rabbitmq-server


SOURCE PATH:

Add the current directory to your PYTHONPATH for an easier time running scripts. Add the following
line to your ~/.profile (create it if it doesn't exist, or use bash_profile if you already have that):
    > export PYTHONPATH=.


USE
===

Provide local configuration overrides (e.g. broker host, debug levels) in:
    > res/config/pyon.local.yml
    > res/config/logging.local.yml
    Note: DO NOT EDIT the standard configuration and logging config files.

List supported Pyon Capability Container arguments:
    > bin/pycc --help

Start an example (bank) locally:
    > bin/pycc --rel res/deploy/examples/bank_complete.yml
    Note: It does not do anything but start by itself. You need to use a client to trigger actions.

Start development instance of the ION system with all services:
    > bin/pycc --rel res/deploy/r2deploy.yml

Run unit tests:
    > bin/nosetests -a UNIT

Use integration tests:

    > bin/nosetests -a INT

TROUBLESHOOTING
===============

bin/generate_interfaces can leave outdated stuff behind that causes your
tests to fail unexpectedly.  Do an ant clean and try again.
    > ant clean

bin/generate_interfaces still has import errors after ant clean.  Then, you
might have an outdated pyon repo
    > cd ../pyon
    > git pull
    > cd -

If your tests fail unexpectedly when you have tried everything else, your build
might be corrupted.  Do an ant clean buildout and try again with a new
virtualenv and build from scratch.

    > ant clean-buildout

DEBUG
=====

USE Pycharm or any other Python IDE to debug; breakpoints and step-by-step are possible.
Set the executable to the bin/pycc and set the working directory to the repository root.
Use the -n option to avoid the interactive ipython shell. It does not work property with the debugger.

You will need to make sure the Pycharm debug egg is available in your path, something like:
    > export PYTHONPATH=.:/Applications/PyCharm\ 1.5.2.app/pycharm-debug.egg

Set your debugger to start the bin/pycc script, with the arguments you like.

Remote debugging of services in a running container is possible too. Setup a remote debugging
configuration to use the localhost and port 8585. First start the remote debuging server, then
start the container from the command line below using the pyccd script ( the -n seems to be required )
    > bin/pyccd -n --rel res/deploy/xxx.yml


DEVELOPMENT
===========

You can develop services locally in this repository. Use this repository until subsystem
specific repositories are available.

Please follow the following steps as long as you are new:

Get the latest code before you start editing, or anytime you want:
    > git pull
    > git submodule update  # Do NOT forget. This does not happen automatically

See below for an automated approach.

Once in a while, service interfaces change. Generate interfaces frequently (especially in case of error):
    > bin/generate-interfaces

Before define objects, services in ./obj, define app and deploy files in /res, checkout master:
    > cd extern/ion-definitions
    > git status              # Just to see what's going on
    > git checkout master     # To track the master branch (enables update and later push)
    > git pull origin master  # To get latest from the server

*See below for an automated approach to git-submodules*
Note: The res/ and obj/ dirs are symlinks to a subdirectory in a git submodule. Beware of the pitfalls
of git submodule. You need to treat it as a separate GIT module. In case of changes, both GIT modules
must be pushed, submodule first:
    > cd extern/ion-definitions
    > git status            # Just to see what's going on
    > git commit -am "Something smart"
    > git push origin master
    > cd ../..              # To the root of coi-services
    > git commit -am "Something smarter"
    > git push

Put your services in ion/services/<subsystem>/... (subdirectories are allowed).


GIT SUBMODULE HOOKS
===================

A git hook is a script that executes during various points of using git. Some simple hooks have been written
to help automate dealing with submodules for most people. See the steps here:
http://blog.chaitanyagupta.com/2009/08/couple-of-hooks-to-make-life-easy-with.html

They do require an initial setup. Simple instructions:

Clone this repository:
    > cd /some/tmp/directory
    > git clone https://github.com/chaitanyagupta/gitutils.git

Use the provided install script:
    > sh gitutils/submodule-hooks/install.sh /path/to/your/pyon/or/coi-services/dir


The install script does the following (you can also do it manually):
    > cd /your/pyon/.git/hooks
    > cp /your/tmp/gitutils/submodule-hooks/pre-commit pre-commit
    > cp /your/tmp/gitutils/submodule-hooks/post-merge-commit post-merge
    > ln -s post-merge post-checkout
    > chmod +x post-merge post-checkout pre-commit

Now, when checking out a branch, pulling, merging etc, git will prompt you to automatically update
if it notices a change to the commit that your supermodule points to.

The pre-commit script is so you don't forget to push changes to the submodule *BEFORE* you push changes
to the supermodule.


Something went wrong with that request. Please try again.