Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 310 lines (253 sloc) 12.926 kB
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
1 ==================================================
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
2 Ocean Observatories Initiative Cyberinfrastructure
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
3 Integrated Observatory Network (ION)
4 ioncore-python - Capability Container and Core Modules
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
5 ==================================================
53861dc @genkuro Touched to force build.
genkuro authored
6
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
7 April 2010 - September 2010 (C) UCSD Regents
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
8
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
9 This project provides a service framework with auxilliary functions for running
ea45e76 @mmeisinger removed rm_pyc.sh in favor of ant target; updated README with depende…
mmeisinger authored
10 architecturally complete versions of all the services of the OOI release 1
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
11 system with their full architectural dependencies in Python.
12 Functionally it provides a data distribution network based on pub-sub messaging
13 as well as instrument integration, with underlying infrastructure services
14 (security, persistence) and provisioning.
b2b045b @mmeisinger README updated with dependencies
mmeisinger authored
15
16 The "core" part of LCAarch provides base classes and a framework for spawnable,
17 message communicating processes, for services with defined names, for
ea45e76 @mmeisinger removed rm_pyc.sh in favor of ant target; updated README with depende…
mmeisinger authored
18 bootstrapping the system, for managing logging and configuration etc. This
19 is an early implementation of the OOI Python Capability Container.
b2b045b @mmeisinger README updated with dependencies
mmeisinger authored
20
ad37c05 @mmeisinger PEP8 compliance
mmeisinger authored
21 For more information, please see:
fa52725 @mmeisinger Added Python version check
mmeisinger authored
22 http://www.oceanobservatories.org/spaces/display/syseng/CIAD+COI+SV+Python+Capability+Container
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
23
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
24 Dependencies
25 ============
26
83c26fe update readme for github
Paul Hubbard authored
27 ioncore-python is compatible with Python 2.5 and higher, but not Python 3.x
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
28 This project has several dependencies on libraries and packages. Most of these
29 dependencies are resolved automatically using the provided setup script.
30
31 Step 1: Virtual env
54d0960 @mmeisinger Improved README.txt
mmeisinger authored
32 Create a Python virtual environment using virtualenv. This ensures that all
33 libraries and dependencies are installed separately from the Python
34 system libraries
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
35 ::
d117511 @jamie-cyber1 Updated README.txt with clearer instructions.
jamie-cyber1 authored
36 mkvirtualenv --no-site-packages --python=/usr/bin/python2.5 --distribute ioncore-python
83c26fe update readme for github
Paul Hubbard authored
37 workon ioncore-python
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
38
d117511 @jamie-cyber1 Updated README.txt with clearer instructions.
jamie-cyber1 authored
39 Note: When troubleshooting issues (since easy_install is easy but has its own problems)
40 , start all over again with a new virtualenv, such as:
41 mkvirtualenv --no-site-packages --python=/usr/bin/python2.5 --distribute ioncore-python2
42 You can remove old virtualenvs using `rmvirtualenv <env_name>`.
43
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
44 Step 2: Core libraries (you can skip this step)
45 Install some core libraries first. Sometimes the automatic installer
46 produces errors, if these libraries are not present beforehand.
47 ::
54d0960 @mmeisinger Improved README.txt
mmeisinger authored
48 easy_install -U twisted
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
49
50 Step 3: Run the setup script
51 ::
fa52725 @mmeisinger Added Python version check
mmeisinger authored
52 ant install # This is equivalent to python setup.py install
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
53
54 This should download and install all the dependencies and will run for a while.
55 Check the trace output that there are no substantial errors. You are now ready
56 to run.
57
58 Current dependencies include:
f255406 Improved Readme
David Stuebe authored
59 twisted, carrot, txamqp, msgpack-python, httplib2, simplejson, Telephus
60 gviz_api.py, nimboss, txrabbitmq, M2Crypto-patched, ionproto and protobuf
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
61
62 NOTE: As the project evolves and new code is added, dependencies might change.
63 Run the setup script once in a while and when you get errors
64
f0c1ad4 THis should fix the M2Crypto package for linux systems. It now has S…
Roger Unwin authored
65 *M2Crypto note: requires autoconf, automake, and g++ packages be installed prior.
66
4715758 @mmeisinger removed test from boostrap, changed datastore for new msg format
mmeisinger authored
67
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
68 Usage
69 =====
4715758 @mmeisinger removed test from boostrap, changed datastore for new msg format
mmeisinger authored
70
83c26fe update readme for github
Paul Hubbard authored
71 (all subsequent steps assume you are in the ioncore-python/ root dir)
d9eb8b7 @mmeisinger package init, ts test
mmeisinger authored
72
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
73 Start empty Python Capability Container shell with:
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
74 ::
d117511 @jamie-cyber1 Updated README.txt with clearer instructions.
jamie-cyber1 authored
75 scripts/start-cc -h amoeba.ucsd.edu
76 scripts/start-cc # to run with localhost
54d0960 @mmeisinger Improved README.txt
mmeisinger authored
77 # Alternatively the direct call to twistd developer
78 twistd -n cc -h amoeba.ucsd.edu
79 # To set a sysname, i.e. a "cluster name" for all containers in a cluster
80 twistd -n cc -h amoeba.ucsd.edu -a sysname=mycluster
3a84fde @mmeisinger Bootstrap improvements, procutils, datastore test
mmeisinger authored
81
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
82 (to end a capability container shell, press Ctrl-D Ctrl-C)
e710e44 @mmeisinger CC agent with announcement on identification request; can spawn a pro…
mmeisinger authored
83
3a84fde @mmeisinger Bootstrap improvements, procutils, datastore test
mmeisinger authored
84 Start system by executing within the CC shell:
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
85 ><>
86 from ion.core import bootstrap
87 bootstrap.start()
88
84c4757 @mmeisinger Added simple mock Java integration service for LCA and ant target to …
mmeisinger authored
89 Alternatively (better) from UNIX shell executing a script:
d9eb8b7 @mmeisinger package init, ts test
mmeisinger authored
90 ::
d117511 @jamie-cyber1 Updated README.txt with clearer instructions.
jamie-cyber1 authored
91 scripts/start-cc -h amoeba.ucsd.edu res/scripts/bootstrap.py
92 scripts/start-cc -h amoeba.ucsd.edu res/scripts/newcc.py
dcae3c7 Fixed order of consumer names in pubsub script
David Stuebe authored
93
d9eb8b7 @mmeisinger package init, ts test
mmeisinger authored
94
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
95 Testing
96 =======
97
b78736b @mmeisinger changed readme; make sure to update and install latest magnet
mmeisinger authored
98 Run trial test cases (recursively)
d9eb8b7 @mmeisinger package init, ts test
mmeisinger authored
99 ::
e710e44 @mmeisinger CC agent with announcement on identification request; can spawn a pro…
mmeisinger authored
100 trial ion
b78736b @mmeisinger changed readme; make sure to update and install latest magnet
mmeisinger authored
101 trial ion.core
102 trial ion.services.coi.test.test_resource_registry
d9eb8b7 @mmeisinger package init, ts test
mmeisinger authored
103
cad868a @mmeisinger Updated README to correct magnet dependency: now master
mmeisinger authored
104 A good learning example is the HelloService
105 ::
106 trial ion.play.test.test_hello
107
108 Or in the CC shell:
109 ><>
110 from ion.play import hello_service
111 spawn(hello_service)
112 send(1, {'op':'hello','content':'Hello you there!'})
113
114 from ion.play.hello_service import HelloServiceClient
342829f @mmeisinger Local configuration override in ionlocal.config. Unit tests get broke…
mmeisinger authored
115 hc = HelloServiceClient()
cad868a @mmeisinger Updated README to correct magnet dependency: now master
mmeisinger authored
116 hc.hello()
117
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
118
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
119 Build and Packaging using Ant
120 =============================
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
121
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
122 LCAarch provides ANT support (see http://ant.apache.org/).
123 To check that ant is installed properly, run
124 :: ant
ab3d9bb @mmeisinger base service and process, service startup
mmeisinger authored
125
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
126 To clean your working directories, run
127 :: ant clean
7d23793 @mmeisinger Added ant build file; supports clean and start of CC
mmeisinger authored
128
fa52725 @mmeisinger Added Python version check
mmeisinger authored
129 To install all Python dependencies, run
130 :: ant install
131
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
132 To compile all code to see if there are Python compile errors anywhere:
133 :: ant compile
c43dc6c @mmeisinger Readme appended with last changes, OTP functions in base_process
mmeisinger authored
134
135
136 ---------------------------------------------------------------------------
137 Change log:
138 ===========
d117511 @jamie-cyber1 Updated README.txt with clearer instructions.
jamie-cyber1 authored
139 2011-1-26:
140 - Moved scripts in bin/ dir to scripts/ dir.
461995b Removed IRODS and bumped ion version
David Stuebe authored
141
142 2011-1-24:
143 - Removed IRODS as backend storage configureation option
144
f255406 Improved Readme
David Stuebe authored
145 2011-1-21:
146 - ION is now using google protocol buffer objects in the resource object model
147 - Bumped version number to 0.4.0 consistent with the setup.py file.
148
c43dc6c @mmeisinger Readme appended with last changes, OTP functions in base_process
mmeisinger authored
149
90805ff @mmeisinger Process RPC refactoring and timeout management
mmeisinger authored
150 2010-10-28:
151 - Set RPC default timeout to 15 secs (see ion.config).
152 Use a different secs value in in rpc_send(..., ..., timeout=5)
153
a9c7c0c @mmeisinger Refactoring: Moved BaseProcess and BaseService; changed all dependents
mmeisinger authored
154 2010-10-05:
155 - REFACTORING OF BASE CLASSES CONTINUED
156 - Changed ion.core.base_process.BaseProcess to ion.core.process.process.Process
157 - Changed ion.services.base_service.BaseService to
158 ion.core.process.service_process.ServiceProcess
159 - Modified all dependent classes
160
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
161 2010-10-04:
162 - MASSIVE REFACTORING IN BASE CLASSES
ccdf027 @mmeisinger Fixed one more test
mmeisinger authored
163 - Refactored the former magnet code into more object oriented style.
6a8d5aa @mmeisinger Added a bit to README.txt
mmeisinger authored
164 - Requires Carrot 0.10.11. Carrot before does not handle all deferred
ccdf027 @mmeisinger Fixed one more test
mmeisinger authored
165 operations correctly.
166 - Refactored the Receiver use. There are now subclasses for Receivers that
167 manage and declare the specific types of AMQP resources, such as worker and
168 fanout. No more declare_messaging necessary.
169 - Refactoried the capability container classes.
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
170 - Added a FSM based StateObject. Many manager/controller level objects now make
171 use of states. States and operations INIT -> initialize() -> READY ->
6a8d5aa @mmeisinger Added a bit to README.txt
mmeisinger authored
172 activate() -> ACTIVE -> terminate() -> TERMINATED (and more, with errors)
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
173 - BaseProcess (and subclasses), Receiver, ProcessDesc, Container etc are all
174 BasicLifecyleObjects.
3d99325 @mmeisinger Added a bit to README.txt
mmeisinger authored
175 - BaseProcess now waits to activate the receiver until in ACTIVE state. Before,
176 code can do RPC, but not receive messages on the process id
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
177 - Massively enhanced the capability container API. Delegated the actual
178 implementation to manager classes: proc, exchange, app manager
3d99325 @mmeisinger Added a bit to README.txt
mmeisinger authored
179 - Refactored the way processes are spawned. Refactored ProcessDesc to use the
6a8d5aa @mmeisinger Added a bit to README.txt
mmeisinger authored
180 new container API. Processes are by default immediately initialized and
181 activated. The op_init message has been eliminated.
182 - Renamed ProtocolFactory to ProcessFactory; changed in each process module
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
183 - Message headers now contain status code for every message. 'OK is the default
184 and 'ERROR' is set on error
d70646e @mmeisinger Refactored RPC error reply into ReceivedError; fixed code and tests
mmeisinger authored
185 - BaseProcess.rpc_send now raises a ReceivedError in case the RPC comes back
186 with status='ERROR'
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
187 - Changed reply_ok and reply_err: a dict content value will not be modified
188 - Fixed imports and tests throughout the code base
ccdf027 @mmeisinger Fixed one more test
mmeisinger authored
189 - Added OTP style apps and app files as primary way to start up processes
190 in the container. See res/apps/*.app files and ion.core.pack
41492d8 @mmeisinger Receiver refactoring; removed/modified Receiver imports
mmeisinger authored
191
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
192 2010-09-20:
fa52725 @mmeisinger Added Python version check
mmeisinger authored
193 - Added start scripts in bin/
2ce7fde @mmeisinger Updated README
mmeisinger authored
194 - Use ant install to install Python dependencies (calls python setup.py install)
aabfe74 @mmeisinger Included magnet modules into ion.core packages and removed dependency…
mmeisinger authored
195 - Removed dependency on magnet. Included all relevant magnet code in ion.core
196 packages cc and messaging.
197 Start with: twistd -n cc
198 - Included all CEI services and base classes in code base
199
4a5d195 @mmeisinger Initial container class
mmeisinger authored
200 2010-08-29:
201 - Changed all logging instances for loggers to log, to avoid name clashes.
202
ba80634 @mmeisinger BaseProcess refactoring; added second receiver for backend interactio…
mmeisinger authored
203 2010-08-14:
204 - BaseProcess: added backend receiver, used for sending out any messages
205 from self.send and self.rpc_send. This keeps the message queue for the process
206 frontend separate from the process backend, e.g. for RPC during a message
207 processing.
208 - Changed BaseProcess logging to make message send and receive easier to spot.
209
911af58 @mmeisinger Fixed BaseProcess: states, message dispatching, error handling
mmeisinger authored
210 2010-08-06:
211 - BaseProcess.spawn() now calls init() automatically. No need to call init()
212 on a process anymore manually. For testing only. Note: The correct way to
213 spawn a process is through a parent process with spawn_child()
214 - Modified and fixed the BaseProcess states, when receiving messages
215 - MAJOR update to BaseProcess message dispatching and subsequent error handling.
216 On error, reply_err messages are sent back, if reply-to header set.
217
7d23793 @mmeisinger Added ant build file; supports clean and start of CC
mmeisinger authored
218 2010-08-03:
219 - Added ant build.xml file to LCAarch root dir. Start with ant.
220 Supports ant clean, which removes all *.pyc from ion path.
221
b93d956 Fixing trial errors and adding comments
David Stuebe authored
222 2010-07-23:
223 - Refactored the Registry Services to inherit from a common base class. This
224 will allow easier implementation of the many registries in the OOICI. The
225 Resource Registry and Service Registry now have basic registration of resource
226 descriptions and services.
227 - The Ion message is now encoded usig the MsgPack library to allow for binary
228 transport of message content. The JSON library was mangleing string content.
229 This is a temporary fix for a wider problem with the encoding structure which
230 needs to be addressed in construction.
231
dfe8bf2 @mmeisinger fixed problems with logging change; added log level configuration bas…
mmeisinger authored
232 2010-06-07:
233 - Redefined logging import to set as module level logger. Each module has now
234 a different logger that can be configured separately. Don't configure
235 manually though.
236 - Added possibility to modify logging levels by module/package, with support
237 for package hierarchy. See res/logging/loglevels.cfg for standard entries.
238 Do not modify. Add file res/logging/loglevelslocal.cfg for local override.
08d0190 @mmeisinger fixed some errors related to logging changes
mmeisinger authored
239 Default logging level for all ion code: WARNING
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
240
f92eeba @mmeisinger Added process shutdown support to BaseProcess; refactored BaseProcess…
mmeisinger authored
241 2010-06-02:
242 - BaseProcess self members renamed to conform to PEP8
243 - Added process shutdown to BaseProcess
982030b @mmeisinger Refactoring in BaseProcess to support an onspawn hook
mmeisinger authored
244 - Added container UNIX shell argument -a processes=<path to filename> used
245 by newcc.py script, with a list of processes to startup in standard format
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
246
3f881c9 @mmeisinger Changed IStore delete to remove; parameterizable cassandra backend
mmeisinger authored
247 2010-05-25:
248 - Made Cassandra backend parameterizable with keyspace/colfamily and added
249 SuperColumn support.
f2ff6a5 @mmeisinger Refactored IStore interface and modified backends, object store, coi …
mmeisinger authored
250 - Modified the IStore interface to support a create_store factory method. This
251 method can yield and return a deferred. Modified and fixed IStore impls.
3f881c9 @mmeisinger Changed IStore delete to remove; parameterizable cassandra backend
mmeisinger authored
252 Changed delete to remove to be more compliant with standard collections.
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
253
241d1d0 @mmeisinger BaseProcess.reply_ok and reply_err added and service operations chang…
mmeisinger authored
254 2010-05-22:
255 - Added timeout to BaseProcess.rpc_send. Use with kwarg timeout=<secs>
256 - CC-Agent detects missing known containers and removes them from the list
257 - Enhanced CC-Agent operations and CC shell helpers
258 - Added sequence numbers for messages
259 - Added glue functions BaseProcess.reply_ok and reply_err and changes some
260 RPC style service operations.
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
261
342829f @mmeisinger Local configuration override in ionlocal.config. Unit tests get broke…
mmeisinger authored
262 2010-05-20:
263 - The system now looks for a local config file ionlocal.config and if exists,
264 overrides entries in ion.config.
265 - Test cases use the config file to determine the broker host to use. If local
70573b4 @mmeisinger added BaseProcessClient and changed all BaseServiceClient subclasses …
mmeisinger authored
266 config override exists, a different broker (e.g. localhost) can be given.
267 - Added BaseProcessClient and changed BaseServiceClient and all clients and
268 all test cases (again).
3827a32 @mmeisinger shell helper functions
mmeisinger authored
269 - Added container shell helpers under 'cc', such as cc.spawn('hello')
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
270
e710e44 @mmeisinger CC agent with announcement on identification request; can spawn a pro…
mmeisinger authored
271 2010-05-16:
272 - Removed support for BaseProcess.send_message and reply_message. Always use
273 send, reply and rpc_send now.
274 - Any BaseProcess instance can now spawn_child() other processes.
275 - Removed RpcClient class, because every process can do rpc_send()
276 - Service processes now also listen to their service name's queue. The service
277 name is determined from the service declaration. Two processes will listen
278 to the same queue and take messages round robin from the queue.
279 - Startup arguments evaluated, for instance to start with system name set:
280 twistd -n magnet -a sysname=mysys
281 twistd -n magnet -a "{'sysname':'mysys'}"
282 - Added capability container agent process. Start with:
283 twistd -n magnet res/scripts/newcc.py
284 Agents announce themselves to others in the same system and can spawn procs.
285 - Name scope 'local' for messaging names means now really local to one container.
286 Use scope 'system' for names unique for each bootstrapped system. Do not use
287 global names, because they will clash.
288 - Less verbose trace output for process init messages and changes to other
289 trace output as well.
bc50a38 @mmeisinger Changed BaseServiceClient __init__ args and all client impls; takes n…
mmeisinger authored
290 - Changed BaseServiceClient and tests. Initializer arguments different.
cad868a @mmeisinger Updated README to correct magnet dependency: now master
mmeisinger authored
291 - Using master branch of magnet now
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
292
7a9f54a @mmeisinger python 2.5 compatibility: changed .format to % and json import
mmeisinger authored
293 2010-05-10:
294 - Based on entries in config files, service process modules are sought and
295 loaded in order to collect the service process declarations. This enables
296 a registration of services with versions and dependencies on startup
07a06bb @mmeisinger Revised README file to reflect automatic setup and ant build
mmeisinger authored
297
c43dc6c @mmeisinger Readme appended with last changes, OTP functions in base_process
mmeisinger authored
298 2010-05-06:
299 - Refactored the BaseProcess class.
300 - You can now do RPC directly from the process, via self.rpc_send without
5ea10a7 @mmeisinger made IonTestCase and all tests PEP8 compliant
mmeisinger authored
301 the need for an RpcClient. Works even with other messages coming in at the
302 same time (using the conv-id)
c43dc6c @mmeisinger Readme appended with last changes, OTP functions in base_process
mmeisinger authored
303 - Added aliases: self.send, self.reply (use these now)
304 - Process instance can be spawned via self.spawn()
305 - Provided an easier to use BaseServiceClient, which uses a default service
306 name lookup. Accepts BaseProcess instance as argument to use for sending/
307 receiving service calls.
0fd6d96 Testing commit. added a blank line to the end of the README.txt
Roger Unwin authored
308
34a8891 Added a . for testing purposes. it can be removed.
Roger Unwin authored
309 .
Something went wrong with that request. Please try again.