Skip to content
This repository

ION Capability Container and core framework (python)

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 examples Fix OOIION-938: Rolling window process saturation June 22, 2013
Octocat-spinner-32 extern Bumped ion-definitions March 07, 2014
Octocat-spinner-32 prototype Added postgres and neo4j prototypes December 19, 2013
Octocat-spinner-32 putil
Octocat-spinner-32 pyon
Octocat-spinner-32 scripts Add options to pycc script and plugin to enable gevent block monitor January 22, 2014
Octocat-spinner-32 .coveragerc Exclude coverage May 10, 2013
Octocat-spinner-32 .gitignore Ignore coverage directory February 29, 2012
Octocat-spinner-32 .gitmodules Switch back to https URL for submodules. December 08, 2011
Octocat-spinner-32 .pyon_root_marker Directory cleanup October 14, 2011
Octocat-spinner-32 CHANGES Changes, Todo updated; sorted list of messaging in shell API October 27, 2011
Octocat-spinner-32 Doxyfile Reduce doxygen gen files February 09, 2012
Octocat-spinner-32 README update installation instructions October 11, 2013
Octocat-spinner-32 TODO
Octocat-spinner-32 autolaunch.cfg Install numpy in autolaunch March 05, 2013
Octocat-spinner-32 bootstrap.py New 2.2.0 buildout bootstrap July 10, 2013
Octocat-spinner-32 build.xml Include examples directory for ant clean September 26, 2012
Octocat-spinner-32 buildout.cfg
Octocat-spinner-32 control_cc Add initial buildout with associated changes October 27, 2011
Octocat-spinner-32 fabfile.py Add release script July 03, 2012
Octocat-spinner-32 gcoverage_buildout.cfg Fix pyon buildout for coverage July 09, 2013
Octocat-spinner-32 obj Converted obj to a submodule October 26, 2011
Octocat-spinner-32 pycc Renamed cc to pycc October 27, 2011
Octocat-spinner-32 pyccd Adding easier support to remote debug a running container November 02, 2011
Octocat-spinner-32 res Added res as symlink to extern/ion-definitions/res November 01, 2011
Octocat-spinner-32 setup.cfg Add examples to coverage December 07, 2011
Octocat-spinner-32 setup.py Revert greenlet back to 0.4.0 April 18, 2014
Octocat-spinner-32 unittest default unittest stdout to unittest.out October 13, 2011
README
===========================================================
Ocean Observatories Initiative Cyberinfrastructure (OOI CI)
Integrated Observatory Network (ION)

pyon - Python COI Capability Container and Core Modules R2
https://confluence.oceanobservatories.org/x/ehdTAQ

(C) UC Regents, 2010-2011
===========================================================


INSTALL
=======

Install the following if not yet present:

* OS Packages and package management:
For Mac, use homebrew
    > /usr/bin/ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)
- python 2.7.2
    Download and use installer on a Mac.
    If you need to build from source (like on some linux distributions), enable the following in the build:
      - SSL
      - sqlite
- git 1.7.7: Use brew or download the Mac or Linux installer and run it
- couchdb 1.1.0 (optional if memory mockdb is used)
    You will need to upgrade couchdb to at least 1.1.0.
- rabbitmq 2.6.1 or later (recommended, but can use rabbitmq on amoeba)
    Use brew.
    Alternative: download generic Linux version and unpack into a suitable directory.
- Install libevent, libyaml, couchdb, python, and rabbitmq with Homebrew
    > brew install libevent libyaml couchdb python rabbitmq hdf5
    You can even reinstall git using brew to clean up your /usr/local directory
    (If you have trouble with brew 'MD5 mismatch' errors, try running 'brew update' 
    and try the install again)
    Linux: Note that many installs have much older versions installed by default.
      - libevent: Install with package manager
      - libyaml: You need the -dev or -devel version. If you don't have such a package, use extern/setup-libyaml.sh
      - couchdb: Make sure to have the appropriate version
- Install libxml2
  > Mac should already have libxml2 installed as part of the developer toolkit, if it's not or you're running an older version you can use homebrew to install it
  > Linux may have to install libxml2 using your preferred package manager. Be sure to install the development package libxml2-dev or libxml2-devel

* python packages and environment management:
- pip
    > easy_install pip
- virtualenv and virtualenvwrapper modules for your python 2.7 installation
    > easy_install --upgrade virtualenv
    > easy_install --upgrade virtualenvwrapper
    Note: This may require Mac's XCode (use XCode 3.3 free version)

* Setup a virtualenv to for Pyon (use any name you like):
    > mkvirtualenv --no-site-packages --python=python2.7 ion27
    > workon ion27
    > pip install numpy==1.7.1



DEVELOPMENT TIME DEPENDENCIES
=============================

PACKAGES:

Use Buildout to install the python package dependencies with:
    > python bootstrap.py -v 2.2.0
    > bin/buildout

** Add following to your .bashrc profile on Macs if buildout fails to
    compile for packages such as gevent, pyml, etc.
    export C_INCLUDE_PATH=/usr/local/include

ION-DEFINITIONS SUBMODULE:

Link git submodule ion-definitions, containing object and service definitions and resource files:
    git submodule update --init

Note: This git-submodule always references a specific commit of ion-definitions used by pyon.

Hint: The .gitmodules file  declares the submodule as read only GIT reference (https: instead of git@).
    If you have problems with a read only (anonymous) clone because of a missing public key,
    edit .git/config and change the last url to git://github.com/ooici/ion-definitions.git

    If you want to avoid entering your password for every push to the submodule,
    edit extern/ion-definitions/.git/config and change the remote reference to:
    git@github.com:ooici/ion-definitions.git

To update the code in pyon and the specific commit in the submodule:
    > git pull
    > git submodule update

Notes:
- The submodule update does NOT happen automatically
- The submodule update does NOT pull the branch head of the submodule.
- To change the submodule commit within pyon, cd into the submodule and do a `git pull`, then commit pyon.

The obj/ subdir is a symlink to the checkout's extern/ion-definitions/objects.
The res/ subdir is a symlink to the checkout's extern/ion-definitions/res.

See more about submodules:
- http://book.git-scm.com/5_submodules.html
- http://speirs.org/blog/2009/5/11/understanding-git-submodules.html
- http://blog.endpoint.com/2010/04/git-submodule-workflow.html


GENERATE SERVICE STUBS:

Generate service definition interfaces and abstract base classes files:
    > bin/generate_interfaces
    Note: These will be placed in the interfaces/ subdirectory (and not in GIT nor in the egg).


RUN-TIME DEPENDENCIES
=====================

COUCHDB (OBJECT DATA STORE):

Run couchdb in second terminal before starting the container or running tests:
    > couchdb


RABBITMQ (MESSAGE BROKER):
    > sudo sbin/rabbitmq-server


SOURCE PATH:

Add the current directory to your PYTHONPATH for an easier time running scripts. Add the following
line to your ~/.profile (create it if it doesnt not exist, or use bash_profile if you already have that):
    > export PYTHONPATH=.


TEST
====

Run the unittests:
    bin/unittest
    bin/nosetests -a UNIT

Run code coverage:
    bin/nosetests -w pyon --with-coverage --cover-package=pyon


USE
===

Provide local configuration overrides in:
    res/config/pyon.local.yml
    res/config/logging.local.yml

Start the capability container in interactive mode:
    > bin/pycc

List supported arguments:
    > bin/pycc --help

Start the CC with a deploy file:
    > bin/pycc --rel res/deploy/r2deploy.yml

Start the CC with a process to execute and then quit:
    > bin/pycc --proc examples.service.hello_service.HelloService

Command line configuration options for pycc:
    Use a different message broker:
    > bin/pycc server.amqp.host=amoeba.ucsd.edu

    Use a different database server:
    > bin/pycc server.couchdb.host=amoeba.ucsd.edu

    Set the container's system name:
    > bin/pycc system.name=MINE


DEBUG
=====

USE Pycharm or any other Python IDE to debug; breakpoints and step-by-step are possible.
Set the executable to the bin/pycc and set the working directory to the repository root.
Use the -n option to avoid the interactive ipython shell. It does not work property with the debugger.

You will need to make sure the Pycharm debug egg is available in your path, something like:
    > export PYTHONPATH=.:/Applications/PyCharm\ 1.5.2.app/pycharm-debug.egg

Remote debugging of services in a running container is possible too. Setup a remote debugging
configuration to use the localhost and port 8585. First start the remote debuging server, then
start the container from the command line below (the -n seems to be required)
    > bin/pyccd -n --rel res/deploy/r2services.yml


DATASTORE
=========

View and manipulate the CouchDB datastore, by accessing http://localhost:5984/_utils


EXAMPLES
========

Make sure to look in the examples directory for a README and examples.


SEE ALSO
========

https://confluence.oceanobservatories.org/display/CIDev/ION+R2+Development
https://confluence.oceanobservatories.org/display/CIDev/R2+Container+Use+Guide
https://confluence.oceanobservatories.org/display/CIDev/R2+Development+Conventions
https://confluence.oceanobservatories.org/display/CIDev/R2+Resource+Development+Guide
https://confluence.oceanobservatories.org/display/CIDev/R2+Service+Implementation+Guide
https://confluence.oceanobservatories.org/display/CIDev/R2+Service+Testing+Guide

https://confluence.oceanobservatories.org/x/ehdTAQ
https://github.com/ooici/coi-services
Something went wrong with that request. Please try again.