Skip to content
Newer
Older
100644 277 lines (226 sloc) 10.9 KB
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
1 ==================================================
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 16, 2010
2 Ocean Observatories Initiative Cyberinfrastructure
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 20, 2010
3 Integrated Observatory Network (ION)
4 ioncore-python - Capability Container and Core Modules
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
5 ==================================================
db03ff6 Bootstrap class and core service structure
mmeisinger authored Apr 15, 2010
6
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
7 April 2010 - September 2010 (C) UCSD Regents
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
8
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
9 This project provides a service framework with auxilliary functions for running
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
10 architecturally complete versions of all the services of the OOI release 1
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
11 system with their full architectural dependencies in Python.
12 Functionally it provides a data distribution network based on pub-sub messaging
13 as well as instrument integration, with underlying infrastructure services
14 (security, persistence) and provisioning.
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
15
16 The "core" part of LCAarch provides base classes and a framework for spawnable,
17 message communicating processes, for services with defined names, for
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
18 bootstrapping the system, for managing logging and configuration etc. This
19 is an early implementation of the OOI Python Capability Container.
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
20
ad37c05 PEP8 compliance
Michael Meisinger authored May 18, 2010
21 For more information, please see:
fa52725 Added Python version check
Michael Meisinger authored Sep 20, 2010
22 http://www.oceanobservatories.org/spaces/display/syseng/CIAD+COI+SV+Python+Capability+Container
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
23
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
24 Get LCAarch with
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
25 ::
26 git clone git@amoeba.ucsd.edu:lcaarch.git
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
27 cd lcaarch
28
29
30 Dependencies
31 ============
32
33 LCAarch is compatible with Python 2.5 and higher, but not Python 3.x
34 This project has several dependencies on libraries and packages. Most of these
35 dependencies are resolved automatically using the provided setup script.
36
37 Step 1: Virtual env
38 Create a Python virtual environment using virtualenv. This ensures that all
39 libraries and dependencies are installed separately from the Python
40 system libraries.
41 ::
42 mkvirtualenv lcaarch
43 workon lcaarch
44
45 Step 2: Core libraries (you can skip this step)
46 Install some core libraries first. Sometimes the automatic installer
47 produces errors, if these libraries are not present beforehand.
48 ::
49 easy_install twisted numpy
50
51 Step 3: Run the setup script
52 ::
fa52725 Added Python version check
Michael Meisinger authored Sep 21, 2010
53 ant install # This is equivalent to python setup.py install
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
54
55 This should download and install all the dependencies and will run for a while.
56 Check the trace output that there are no substantial errors. You are now ready
57 to run.
58
59 Current dependencies include:
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
60 twisted, carrot, numpy, txamqp, msgpack-python, httplib2, pycassa, simplejson,
d6bfb7e Added dependency for gviz_api
David Stuebe authored Aug 18, 2010
61 pydap, pydap.handlers.netcdf, pydap.responses.netcdf, pydap.handlers.nca,
3928f4b @timf Adding dependencies to lcaarch/ion: nimboss, txrabbitmq (both are pac…
timf authored Sep 10, 2010
62 gviz_api.py, nimboss, txrabbitmq
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
63
64 NOTE: As the project evolves and new code is added, dependencies might change.
65 Run the setup script once in a while and when you get errors
66
4715758 removed test from boostrap, changed datastore for new msg format
mmeisinger authored Apr 16, 2010
67
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
68 Usage
69 =====
4715758 removed test from boostrap, changed datastore for new msg format
mmeisinger authored Apr 17, 2010
70
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
71 (all subsequent steps assume you are in the lcaarch/ root dir)
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
72
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
73 Start empty Python Capability Container shell with:
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
74 ::
fa52725 Added Python version check
Michael Meisinger authored Sep 21, 2010
75 bin/start-cc -h amoeba.ucsd.edu
76 bin/start-cc # to run with localhost
3a84fde Bootstrap improvements, procutils, datastore test
mmeisinger authored Apr 15, 2010
77
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
78 (to end a capability container shell, press Ctrl-D Ctrl-C)
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 16, 2010
79
3a84fde Bootstrap improvements, procutils, datastore test
mmeisinger authored Apr 16, 2010
80 Start system by executing within the CC shell:
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
81 ><>
82 from ion.core import bootstrap
83 bootstrap.start()
84
84c4757 Added simple mock Java integration service for LCA and ant target to …
Michael Meisinger authored Aug 3, 2010
85 Alternatively (better) from UNIX shell executing a script:
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
86 ::
fa52725 Added Python version check
Michael Meisinger authored Sep 21, 2010
87 bin/start-cc -h amoeba.ucsd.edu res/scripts/bootstrap.py
88 bin/start-cc -h amoeba.ucsd.edu res/scripts/newcc.py
89 bin/start-cc -h amoeba.ucsd.edu -a nproducers=25 res/scripts/pubsub.py
dcae3c7 Fixed order of consumer names in pubsub script
David Stuebe authored Aug 12, 2010
90
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
91
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
92 Testing
93 =======
94
b78736b changed readme; make sure to update and install latest magnet
Michael Meisinger authored May 1, 2010
95 Run trial test cases (recursively)
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
96 ::
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 17, 2010
97 trial ion
b78736b changed readme; make sure to update and install latest magnet
Michael Meisinger authored May 1, 2010
98 trial ion.core
99 trial ion.services.coi.test.test_resource_registry
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
100
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
101 A good learning example is the HelloService
102 ::
103 trial ion.play.test.test_hello
104
105 Or in the CC shell:
106 ><>
107 from ion.play import hello_service
108 spawn(hello_service)
109 send(1, {'op':'hello','content':'Hello you there!'})
110
111 from ion.play.hello_service import HelloServiceClient
342829f Local configuration override in ionlocal.config. Unit tests get broke…
Michael Meisinger authored May 20, 2010
112 hc = HelloServiceClient()
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
113 hc.hello()
114
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
115
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
116 Build and Packaging using Ant
117 =============================
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
118
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
119 LCAarch provides ANT support (see http://ant.apache.org/).
120 To check that ant is installed properly, run
121 :: ant
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
122
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
123 To clean your working directories, run
124 :: ant clean
7d23793 Added ant build file; supports clean and start of CC
Michael Meisinger authored Aug 3, 2010
125
fa52725 Added Python version check
Michael Meisinger authored Sep 21, 2010
126 To install all Python dependencies, run
127 :: ant install
128
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
129 To compile all code to see if there are Python compile errors anywhere:
130 :: ant compile
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 10, 2010
131
132
133 ---------------------------------------------------------------------------
134 Change log:
135 ===========
136
41492d8 Receiver refactoring; removed/modified Receiver imports
Michael Meisinger authored Oct 4, 2010
137 2010-10-04:
138 - MASSIVE REFACTORING IN BASE CLASSES
139 - Added OTP style apps and app files as primary way to start up processes
140 in the container
141 - Added a FSM based StateObject. Many manager/controller level objects now make
142 use of states. States and operations INIT -> initialize() -> READY ->
143 activate() -> READY -> terminate() -> TERMINATED
144 - BaseProcess (and subclasses), Receiver, ProcessDesc, Container etc are all
145 BasicLifecyleObjects.
146 - Massively enhanced the capability container API. Delegated the actual
147 implementation to manager classes: proc, exchange, app manager
148 - Refactored the way processes are spawned
149 - Refactored the Receiver use. There are now subclasses for Receivers that
150 manage and declare the specific types of AMQP resources, such as worker and
151 fanout. No more declare_messaging necessary.
152 - Refactored the former magnet code into more object oriented style.
9be07fe Test fixes, including agent tests
Michael Meisinger authored Oct 4, 2010
153 - Requires Carrot 0.10.10
41492d8 Receiver refactoring; removed/modified Receiver imports
Michael Meisinger authored Oct 5, 2010
154 - Message headers now contain status code for every message. 'OK is the default
155 and 'ERROR' is set on error
d70646e Refactored RPC error reply into ReceivedError; fixed code and tests
Michael Meisinger authored Oct 5, 2010
156 - BaseProcess.rpc_send now raises a ReceivedError in case the RPC comes back
157 with status='ERROR'
41492d8 Receiver refactoring; removed/modified Receiver imports
Michael Meisinger authored Oct 5, 2010
158 - Changed reply_ok and reply_err: a dict content value will not be modified
159 - Fixed imports and tests throughout the code base
160
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
161 2010-09-20:
fa52725 Added Python version check
Michael Meisinger authored Sep 21, 2010
162 - Added start scripts in bin/
2ce7fde Updated README
Michael Meisinger authored Sep 20, 2010
163 - Use ant install to install Python dependencies (calls python setup.py install)
aabfe74 Included magnet modules into ion.core packages and removed dependency…
Michael Meisinger authored Sep 21, 2010
164 - Removed dependency on magnet. Included all relevant magnet code in ion.core
165 packages cc and messaging.
166 Start with: twistd -n cc
167 - Included all CEI services and base classes in code base
168
4a5d195 Initial container class
Michael Meisinger authored Sep 1, 2010
169 2010-08-29:
170 - Changed all logging instances for loggers to log, to avoid name clashes.
171
ba80634 BaseProcess refactoring; added second receiver for backend interactio…
Michael Meisinger authored Aug 14, 2010
172 2010-08-14:
173 - BaseProcess: added backend receiver, used for sending out any messages
174 from self.send and self.rpc_send. This keeps the message queue for the process
175 frontend separate from the process backend, e.g. for RPC during a message
176 processing.
177 - Changed BaseProcess logging to make message send and receive easier to spot.
178
911af58 Fixed BaseProcess: states, message dispatching, error handling
Michael Meisinger authored Aug 6, 2010
179 2010-08-06:
180 - BaseProcess.spawn() now calls init() automatically. No need to call init()
181 on a process anymore manually. For testing only. Note: The correct way to
182 spawn a process is through a parent process with spawn_child()
183 - Modified and fixed the BaseProcess states, when receiving messages
184 - MAJOR update to BaseProcess message dispatching and subsequent error handling.
185 On error, reply_err messages are sent back, if reply-to header set.
186
7d23793 Added ant build file; supports clean and start of CC
Michael Meisinger authored Aug 4, 2010
187 2010-08-03:
188 - Added ant build.xml file to LCAarch root dir. Start with ant.
189 Supports ant clean, which removes all *.pyc from ion path.
190
b93d956 Fixing trial errors and adding comments
David Stuebe authored Jul 23, 2010
191 2010-07-23:
192 - Refactored the Registry Services to inherit from a common base class. This
193 will allow easier implementation of the many registries in the OOICI. The
194 Resource Registry and Service Registry now have basic registration of resource
195 descriptions and services.
196 - The Ion message is now encoded usig the MsgPack library to allow for binary
197 transport of message content. The JSON library was mangleing string content.
198 This is a temporary fix for a wider problem with the encoding structure which
199 needs to be addressed in construction.
200
dfe8bf2 fixed problems with logging change; added log level configuration bas…
Michael Meisinger authored Jun 7, 2010
201 2010-06-07:
202 - Redefined logging import to set as module level logger. Each module has now
203 a different logger that can be configured separately. Don't configure
204 manually though.
205 - Added possibility to modify logging levels by module/package, with support
206 for package hierarchy. See res/logging/loglevels.cfg for standard entries.
207 Do not modify. Add file res/logging/loglevelslocal.cfg for local override.
08d0190 fixed some errors related to logging changes
Michael Meisinger authored Jun 7, 2010
208 Default logging level for all ion code: WARNING
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
209
f92eeba Added process shutdown support to BaseProcess; refactored BaseProcess…
Michael Meisinger authored Jun 2, 2010
210 2010-06-02:
211 - BaseProcess self members renamed to conform to PEP8
212 - Added process shutdown to BaseProcess
982030b Refactoring in BaseProcess to support an onspawn hook
Michael Meisinger authored Jun 4, 2010
213 - Added container UNIX shell argument -a processes=<path to filename> used
214 by newcc.py script, with a list of processes to startup in standard format
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
215
3f881c9 Changed IStore delete to remove; parameterizable cassandra backend
Michael Meisinger authored May 25, 2010
216 2010-05-25:
217 - Made Cassandra backend parameterizable with keyspace/colfamily and added
218 SuperColumn support.
f2ff6a5 Refactored IStore interface and modified backends, object store, coi …
Michael Meisinger authored May 24, 2010
219 - Modified the IStore interface to support a create_store factory method. This
220 method can yield and return a deferred. Modified and fixed IStore impls.
3f881c9 Changed IStore delete to remove; parameterizable cassandra backend
Michael Meisinger authored May 26, 2010
221 Changed delete to remove to be more compliant with standard collections.
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
222
241d1d0 BaseProcess.reply_ok and reply_err added and service operations chang…
Michael Meisinger authored May 22, 2010
223 2010-05-22:
224 - Added timeout to BaseProcess.rpc_send. Use with kwarg timeout=<secs>
225 - CC-Agent detects missing known containers and removes them from the list
226 - Enhanced CC-Agent operations and CC shell helpers
227 - Added sequence numbers for messages
228 - Added glue functions BaseProcess.reply_ok and reply_err and changes some
229 RPC style service operations.
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
230
342829f Local configuration override in ionlocal.config. Unit tests get broke…
Michael Meisinger authored May 21, 2010
231 2010-05-20:
232 - The system now looks for a local config file ionlocal.config and if exists,
233 overrides entries in ion.config.
234 - Test cases use the config file to determine the broker host to use. If local
70573b4 added BaseProcessClient and changed all BaseServiceClient subclasses …
Michael Meisinger authored May 20, 2010
235 config override exists, a different broker (e.g. localhost) can be given.
236 - Added BaseProcessClient and changed BaseServiceClient and all clients and
237 all test cases (again).
3827a32 shell helper functions
Michael Meisinger authored May 21, 2010
238 - Added container shell helpers under 'cc', such as cc.spawn('hello')
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
239
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 17, 2010
240 2010-05-16:
241 - Removed support for BaseProcess.send_message and reply_message. Always use
242 send, reply and rpc_send now.
243 - Any BaseProcess instance can now spawn_child() other processes.
244 - Removed RpcClient class, because every process can do rpc_send()
245 - Service processes now also listen to their service name's queue. The service
246 name is determined from the service declaration. Two processes will listen
247 to the same queue and take messages round robin from the queue.
248 - Startup arguments evaluated, for instance to start with system name set:
249 twistd -n magnet -a sysname=mysys
250 twistd -n magnet -a "{'sysname':'mysys'}"
251 - Added capability container agent process. Start with:
252 twistd -n magnet res/scripts/newcc.py
253 Agents announce themselves to others in the same system and can spawn procs.
254 - Name scope 'local' for messaging names means now really local to one container.
255 Use scope 'system' for names unique for each bootstrapped system. Do not use
256 global names, because they will clash.
257 - Less verbose trace output for process init messages and changes to other
258 trace output as well.
bc50a38 Changed BaseServiceClient __init__ args and all client impls; takes n…
Michael Meisinger authored May 16, 2010
259 - Changed BaseServiceClient and tests. Initializer arguments different.
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
260 - Using master branch of magnet now
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
261
7a9f54a python 2.5 compatibility: changed .format to % and json import
Michael Meisinger authored May 11, 2010
262 2010-05-10:
263 - Based on entries in config files, service process modules are sought and
264 loaded in order to collect the service process declarations. This enables
265 a registration of services with versions and dependencies on startup
07a06bb Revised README file to reflect automatic setup and ant build
Michael Meisinger authored Aug 17, 2010
266
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 11, 2010
267 2010-05-06:
268 - Refactored the BaseProcess class.
269 - You can now do RPC directly from the process, via self.rpc_send without
5ea10a7 made IonTestCase and all tests PEP8 compliant
Michael Meisinger authored May 10, 2010
270 the need for an RpcClient. Works even with other messages coming in at the
271 same time (using the conv-id)
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 11, 2010
272 - Added aliases: self.send, self.reply (use these now)
273 - Process instance can be spawned via self.spawn()
274 - Provided an easier to use BaseServiceClient, which uses a default service
275 name lookup. Accepts BaseProcess instance as argument to use for sending/
276 receiving service calls.
Something went wrong with that request. Please try again.