HTTPS clone URL
Subversion checkout URL
COI R2 services based on pyon. Parts of this project were integrated into https://github.com/scionrep/scioncc
REL2.0 ape_preload coi-conversations coi_asset_tracking dashisysname function-extensions gevent10 jira1021 master merge-M153 mi minimum_free_slots node_exclusive_bug ooiion-1389 ooiion1277_r2 ooitk-proto postgres_merge puck qc_work refactor_couchbase refactor_datastore refactor_ds refactor_eoi_tests refactor_mi_data_handler refactor_newpyon refactor_pagent_botpt refactor_persistence refactor_pika refactor_proto refactor_pyonsubscribers refactor_scheduler refactor_teststates replicas skip_test_r2 user_sites
Nothing to show
Nothing to show
Fetching latest commit...
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
=========================================================== Ocean Observatories Initiative Cyberinfrastructure (OOI CI) Integrated Observatory Network (ION) - OOI Network coi-services - ION Release 2 system and services (C) UC Regents, 2010-2013 =========================================================== This is the repository that defines the OOI Network with its services and agents. The OOI Network system is started from this repository. Submodules provide additional required functionality, such as the Pyon Capability Container. INSTALL ======= This assumes basic development environment setup (git, directory structure). Please follow the "New Developers Tutorial" for basic steps. https://confluence.oceanobservatories.org/display/CIDev/New+Developer+Tutorial The Pyon Capability Container is packaged as a submodule. All dependencies are installed during the process described below. For details, refer to: https://github.com/ooici/pyon/blob/master/README Install the following if not yet present (brew install instructions are for Mac only): * OS Packages and package management: For Mac, use homebrew > /usr/bin/ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go) - python 2.7.x > brew install python --framework --universal Note: see 'brew doctor' for any path issues - git 1.7.7 or higher: Use brew or download the Mac or Linux installer and run it > brew install git - Install core packages with Homebrew (including CouchDB and RabbitMQ) > brew install libevent libyaml zeromq couchdb rabbitmq hdf5 pkg-config netcdf udunits Linux: Note that many installs have much older versions installed by default. (If you have trouble with brew 'MD5 mismatch' errors, try running 'brew update' and try the install again) For any kind of installation trouble run 'brew doctor' and fix any issues. - Install libgswteos-10 for marine science calculations > brew tap lukecampbell/homebrew-libgswteos > brew install libgswteos-10 > brew test -v libgswteos-10 - Install data processing packages (Needed by udunits and scipy) > brew install swig (Make sure brew installed swig is used. `which swig`) > brew install gfortran * Python packages and environment management: - pip > easy_install pip - virtualenv and virtualenvwrapper modules for your python 2.7 installation > easy_install --upgrade virtualenv > easy_install --upgrade virtualenvwrapper Note: This require Mac's XCode with installed command line tools * Setup a virtualenv to run COI-services: > mkvirtualenv --no-site-packages coi Note: Do not use the pyon virtualenv if you are a pyon developer * Pin setuptools in your virtualenv > pip install -U setuptools==0.8 * Install numpy to the virtualenv > pip install numpy==1.7.1 PACKAGE DOWNLOAD ================ Clone coi-services (this repository, if you haven't already) in your code directory: > cd code # Whatever your code repository directory is > git clone firstname.lastname@example.org:ooici/coi-services.git Initialize submodules:: > cd coi-services > git submodule update --init (see Pyon README for hints with submodule anonymous checkout and password-less push) PACKAGE DEPENDENCIES ==================== Use buildout to install the python package dependencies with: > python bootstrap.py -v 2.2.0 > bin/buildout ** Add following to your .bashrc profile on Macs if buildout fails to compile for packages such as gevent, pyml, etc. export C_INCLUDE_PATH=/usr/local/include ** If matplotlib fails to build due to not finding the free type headers, try: if /usr/X11/lib/pkgconfig has a file named freetype2.pc, then add the following line to your .bash_profile export PKG_CONFIG_PATH=/usr/X11/lib/pkgconfig Mac should have libpng and libfreetype headers, which are required for matplotlib python build. If for some reason your buildout fails on matplotlib due to missing libpng libraries, try install from here: http://ethan.tira-thompson.com/Mac_OS_X_Ports.html For additional support to install matplotlib on MacOS X 10.7 (Lion) see: http://www.thisisthegreenroom.com/2011/installing-python-numpy-scipy-matplotlib-and-ipython-on-lion/ Generate service definition interfaces and abstract base classes files: > bin/generate_interfaces Note: These will be placed in the interfaces/ subdirectory. RUN-TIME DEPENDENCIES ===================== COUCHDB (DATA STORE): Run couchdb in second terminal before starting the container or running tests: Note: See the New Developer Tutorial for automatic start using supervisord > couchdb RABBITMQ (MESSAGE BROKER): Run RabbitMQ in second terminal before starting the container or running tests: Note: See the New Developer Tutorial for automatic start using supervisord > cd rabbitmq_server-2.x.x > sudo sbin/rabbitmq-server ELASTICSEARCH (SEARCH ENGINE): ------------ Installation ------------ Installation is performed with a script which downloads the file from github, extracts the contents to the INSTALL_DIR and installs the correct version of the plugins. > bash scripts/install_es.sh Note: Works on both Mac and (Untested) Linux. This installs ElasticSearch to /usr/local/elasticsearch-$(VERSION)/ by default and can be overridden with the INSTALL_DIR environment variable e.g. > INSTALL_DIR=/opt/elasticsearch bash scripts/install_es.sh --------------------- Running ElasticSearch --------------------- After ElasticSearch has been installed it can be started in foreground mode or daemon mode. Assuming that the install directory was /usr/local Foreground mode: > /usr/local/elasticsearch-0.18.7/bin/elasticsearch -f Daemon mode: > /usr/local/elasticsearch-0.18.7/bin/elasticsearch ElasticSearch is an instance of the JVM with the ElasticSearch cluster node, each node is equivalent to one JVM instance running ElasticSearch. Additional nodes means more instances of the JVM running ElasticSearch. The JVM minimum heap and maximum heap size and performance flags can be changed in the script /usr/local/elasticsearch-0.18.7/bin/elasticsearch which is strongly recommended for production. To install the ElasticSearch-Head plugin which provides a graphical interface for browsing and viewing the contents of the engine run: > bash scripts/install_es.sh -h --------------------------------- Viewing ElasticSearch Graphically --------------------------------- If the ElasticSearch-Head plugin is installed (see above) then access to the plugin is located at: http://localhost:9200/_plugin/head/ SOURCE PATH Add the current directory to your PYTHONPATH for an easier time running scripts. Add the following line to your ~/.profile (create it if it doesn't exist, or use bash_profile if you already have that): export PYTHONPATH=. USE === Provide local configuration overrides (e.g. broker host, debug levels) in: > res/config/pyon.local.yml > res/config/logging.local.yml Note: DO NOT EDIT the standard configuration and logging config files. List supported Pyon Capability Container arguments: > bin/pycc --help Run unit tests: > bin/nosetests -v -a UNIT Run smoke tests to verify sanity of the system: > bin/nosetests -v -a SMOKE Run full integration tests - this will take a long time so may be optional: > bin/nosetests -v -a INT Start an example (bank) locally: > bin/pycc --rel res/deploy/examples/bank_complete.yml Note: It does not do anything but start a service. You need a client to trigger actions. Start development instance of the ION system with all services: > bin/pycc --rel res/deploy/r2deploy.yml TROUBLESHOOTING =============== Outdated *.pyc files can causes your tests to fail unexpectedly. Do an ant clean and try again. > ant clean > bin/generate_interface If your tests still fail, you might have some incorrect values in pyon.local.yml or logging.local.yml. Check with other developers. If your tests fail unexpectedly when you have tried everything else, your build might be corrupted. > deactivate your existing virtualenv > rm and create the virtualenv again > re-run bootstrap.py and buildout > If you suspect an egg is bad, remove the suspected egg from the eggs/ dir. Re-run buildout. DEBUG ===== Use Pycharm or any other Python IDE to debug; breakpoints and step-by-step are possible. Set the executable to the bin/pycc and set the working directory to the git repository root. Use the -n option to avoid the interactive IPython shell--it does not work properly with the debugger. Start bin/pycc or bin/nosetests scripts in the debugger, with arguments. PyCharm may need to be enabled to run with gevent. Google how to do this. Remote debugging of services in a running container is possible too. Setup a remote debugging configuration to use the localhost and port 8585. First start the remote debugging server, then start the container from the command line below using the pyccd script (the -n seems to be required): > bin/pyccd -n --rel res/deploy/xxx.yml DEVELOPMENT =========== You can develop services locally in this repository. Please follow the following steps as long as you are new: Get the latest code before you start editing, or anytime you want: > git pull --rebase (fix any issues, git add, git rebase --continue) > git submodule update # Do NOT forget. This does not happen automatically See below for an automated approach using git submodule hooks. As extern submodule pointers change, service interfaces can change and old pyc files can be left around unintentinally. Generate interfaces frequently (especially in case of error): > ant clean > bin/generate-interfaces Note: The res/ and obj/ dirs are symlinks to a subdirectory in a git submodule. Beware of the pitfalls of git submodule. You need to treat it as a separate GIT module. In case of changes, both GIT modules must be pushed, submodule first: > cd extern/ion-definitions > git checkout master > do edits... > git status # Just to see what's going on > git commit -am "Something smart" > git push origin master > cd ../.. # To the root of coi-services > git commit -am "Something smarter" > git push Put your services in ion/services/<subsystem>/... (subdirectories are allowed). The ion-definition submodules approach above applies to other submodules as well. GIT SUBMODULE HOOKS =================== A git hook is a script that executes during various points of using git. Git hooks exist automating submodules for most people. See the steps here: http://blog.chaitanyagupta.com/2009/08/couple-of-hooks-to-make-life-easy-with.html They do require an initial setup. Simple instructions: Clone this repository: > cd /some/tmp/directory > git clone https://github.com/chaitanyagupta/gitutils.git Use the provided install script: > sh gitutils/submodule-hooks/install.sh /path/to/your/pyon/or/coi-services/dir Now, when checking out a branch, pulling, merging etc, git will prompt you to automatically update if it notices a change to the commit that your supermodule points to. The pre-commit script is so you don't forget to push changes to the submodule *BEFORE* you push changes to the supermodule. Instrument Development Kit (IDK) ================================ There are now 4 configuration options in a new config file res/config/idk.yml The only option you are likely to need to override is the driver_path, currently set to ion/code. Override it in idk.local.yml to the parent directory of coi-services, pyon, and marine-integrations if they are located elsewhere. Use driver_path: OOI If you have the following structure. ~/OOI/coi-services ~/OOI/pyon ~/OOI/marine-integrations