Skip to content
Newer
Older
100644 215 lines (183 sloc) 8.65 KB
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
1 ==================================================
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
2 LCAarch - OOI Release 1 LCA architecture prototype
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
3 ==================================================
db03ff6 Bootstrap class and core service structure
mmeisinger authored Apr 15, 2010
4
84c4757 Added simple mock Java integration service for LCA and ant target to …
Michael Meisinger authored Aug 3, 2010
5 April 2010 - August 2010
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
6
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
7 This project provides a service framwork with auxilliary functions for running
8 architecturally complete versions of all the services of the OOI release 1
9 system with their full architectural dependencies.
10 In "short" a data distribution network based on pub-sub messaging with underlying
ad37c05 PEP8 compliance
Michael Meisinger authored May 18, 2010
11 infrastructure services (security, persistence) and provisioning.
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
12
13 The "core" part of LCAarch provides base classes and a framework for spawnable,
14 message communicating processes, for services with defined names, for
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
15 bootstrapping the system, for managing logging and configuration etc. This
16 is an early implementation of the OOI Python Capability Container.
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
17
ad37c05 PEP8 compliance
Michael Meisinger authored May 19, 2010
18 For more information, please see:
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
19 http://www.oceanobservatories.org/spaces/display/CIDev/LCAARCH+Development+Project
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
20
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
21 LCAarch is compatible with Python 2.5 and higher, but not Python 3.x
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
22 This project has several dependencies on libraries and packages, see below.
db03ff6 Bootstrap class and core service structure
mmeisinger authored Apr 15, 2010
23
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
24 Get LCAarch with
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
25 ::
26 git clone git@amoeba.ucsd.edu:lcaarch.git
4715758 removed test from boostrap, changed datastore for new msg format
mmeisinger authored Apr 16, 2010
27
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
28 Usage
29 =====
4715758 removed test from boostrap, changed datastore for new msg format
mmeisinger authored Apr 17, 2010
30
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
31 (all subsequent steps assume start from lcaarch/ dir)
32
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
33 Start empty CC ("Magnet" Python Capability Container) shell with:
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
34 ::
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
35 twistd -n magnet -h amoeba.ucsd.edu
3a84fde Bootstrap improvements, procutils, datastore test
mmeisinger authored Apr 15, 2010
36
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 16, 2010
37 (to end a magnet container shell, press Ctrl-D Ctrl-C)
38
3a84fde Bootstrap improvements, procutils, datastore test
mmeisinger authored Apr 16, 2010
39 Start system by executing within the CC shell:
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
40 ><>
41 from ion.core import bootstrap
42 bootstrap.start()
43
84c4757 Added simple mock Java integration service for LCA and ant target to …
Michael Meisinger authored Aug 4, 2010
44 Alternatively (better) from UNIX shell executing a script:
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
45 ::
46 twistd -n magnet -h amoeba.ucsd.edu res/scripts/bootstrap.py
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 17, 2010
47 twistd -n magnet -h amoeba.ucsd.edu res/scripts/newcc.py
dcae3c7 Fixed order of consumer names in pubsub script
David Stuebe authored Aug 12, 2010
48 twistd -n magnet -h amoeba.ucsd.edu -a nproducers=25 res/scripts/pubsub.py
49
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
50
b78736b changed readme; make sure to update and install latest magnet
Michael Meisinger authored May 1, 2010
51 Run trial test cases (recursively)
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
52 ::
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 17, 2010
53 trial ion
b78736b changed readme; make sure to update and install latest magnet
Michael Meisinger authored May 1, 2010
54 trial ion.core
55 trial ion.services.coi.test.test_resource_registry
d9eb8b7 package init, ts test
Michael Meisinger authored Apr 25, 2010
56
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
57 A good learning example is the HelloService
58 ::
59 trial ion.play.test.test_hello
60
61 Or in the CC shell:
62 ><>
63 from ion.play import hello_service
64 spawn(hello_service)
65 send(1, {'op':'hello','content':'Hello you there!'})
66
67 from ion.play.hello_service import HelloServiceClient
342829f Local configuration override in ionlocal.config. Unit tests get broke…
Michael Meisinger authored May 20, 2010
68 hc = HelloServiceClient()
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
69 hc.hello()
70
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
71
72 Install the dependencies: Magnet (see Magnet's Readme)
73 ======================================================
74 Recommendation:
75 Create a virtualenv for installing Magnet and its dependencies.
76
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
77 Twisted Framework
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
78 :: easy_install twisted
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
79
80 txAMQP
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
81 :: easy_install txamqp
b2b045b README updated with dependencies
Michael Meisinger authored May 17, 2010
82
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
83 carrot (use txamqp branch)
ad37c05 PEP8 compliance
Michael Meisinger authored May 19, 2010
84 --------------------------
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
85 ::
86 git clone git://amoeba.ucsd.edu/carrot.git
87 (cd carrot; git checkout -b txamqp origin/txamqp)
88 (cd carrot; python setup.py install)
7d23793 Added ant build file; supports clean and start of CC
Michael Meisinger authored Aug 3, 2010
89
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
90 Magnet (NEED Magent 0.3.4 on master branch)
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
91 ------------------------------------------------------------------
ab3d9bb base service and process, service startup
mmeisinger authored Apr 19, 2010
92 Get the latest version of the repository, if you haven't already.
93 ::
94 git clone git://amoeba.ucsd.edu/magnet.git # no ooi credential
95 # OR
96 git clone git@amoeba.ucsd.edu:magnet.git # need ooi credential
97 (cd magnet; python setup.py install)
b78736b changed readme; make sure to update and install latest magnet
Michael Meisinger authored May 1, 2010
98
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
99 Other Dependencies
100 ----------------------------
101 ::
102 easy_install msgpack-python
103 easy_install pydap
104 easy_install pydap.handlers.netcdf
105 easy_install pydap.responses.netcdf
dd6bdda Fixed some bugs and added pydap.handlers.nca to build file
David Stuebe authored Aug 12, 2010
106 easy_install pydap.handlers.nca
ea45e76 removed rm_pyc.sh in favor of ant target; updated README with depende…
Michael Meisinger authored Aug 5, 2010
107 easy_install numpy
108 easy_install httplib2
109 easy_install -U --find-links http://ooici.net/packages pycassa
110 easy_install simplejson
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 10, 2010
111
112
113 ---------------------------------------------------------------------------
114 Change log:
115 ===========
116
ba80634 BaseProcess refactoring; added second receiver for backend interactio…
Michael Meisinger authored Aug 14, 2010
117 2010-08-14:
118 - BaseProcess: added backend receiver, used for sending out any messages
119 from self.send and self.rpc_send. This keeps the message queue for the process
120 frontend separate from the process backend, e.g. for RPC during a message
121 processing.
122 - Changed BaseProcess logging to make message send and receive easier to spot.
123
911af58 Fixed BaseProcess: states, message dispatching, error handling
Michael Meisinger authored Aug 6, 2010
124 2010-08-06:
125 - BaseProcess.spawn() now calls init() automatically. No need to call init()
126 on a process anymore manually. For testing only. Note: The correct way to
127 spawn a process is through a parent process with spawn_child()
128 - Modified and fixed the BaseProcess states, when receiving messages
129 - MAJOR update to BaseProcess message dispatching and subsequent error handling.
130 On error, reply_err messages are sent back, if reply-to header set.
131
7d23793 Added ant build file; supports clean and start of CC
Michael Meisinger authored Aug 4, 2010
132 2010-08-03:
133 - Added ant build.xml file to LCAarch root dir. Start with ant.
134 Supports ant clean, which removes all *.pyc from ion path.
135
b93d956 Fixing trial errors and adding comments
David Stuebe authored Jul 23, 2010
136 2010-07-23:
137 - Refactored the Registry Services to inherit from a common base class. This
138 will allow easier implementation of the many registries in the OOICI. The
139 Resource Registry and Service Registry now have basic registration of resource
140 descriptions and services.
141 - The Ion message is now encoded usig the MsgPack library to allow for binary
142 transport of message content. The JSON library was mangleing string content.
143 This is a temporary fix for a wider problem with the encoding structure which
144 needs to be addressed in construction.
145
dfe8bf2 fixed problems with logging change; added log level configuration bas…
Michael Meisinger authored Jun 7, 2010
146 2010-06-07:
147 - Redefined logging import to set as module level logger. Each module has now
148 a different logger that can be configured separately. Don't configure
149 manually though.
150 - Added possibility to modify logging levels by module/package, with support
151 for package hierarchy. See res/logging/loglevels.cfg for standard entries.
152 Do not modify. Add file res/logging/loglevelslocal.cfg for local override.
08d0190 fixed some errors related to logging changes
Michael Meisinger authored Jun 7, 2010
153 Default logging level for all ion code: WARNING
f92eeba Added process shutdown support to BaseProcess; refactored BaseProcess…
Michael Meisinger authored Jun 2, 2010
154 2010-06-02:
155 - BaseProcess self members renamed to conform to PEP8
156 - Added process shutdown to BaseProcess
982030b Refactoring in BaseProcess to support an onspawn hook
Michael Meisinger authored Jun 4, 2010
157 - Added container UNIX shell argument -a processes=<path to filename> used
158 by newcc.py script, with a list of processes to startup in standard format
3f881c9 Changed IStore delete to remove; parameterizable cassandra backend
Michael Meisinger authored May 25, 2010
159 2010-05-25:
160 - Made Cassandra backend parameterizable with keyspace/colfamily and added
161 SuperColumn support.
f2ff6a5 Refactored IStore interface and modified backends, object store, coi …
Michael Meisinger authored May 24, 2010
162 - Modified the IStore interface to support a create_store factory method. This
163 method can yield and return a deferred. Modified and fixed IStore impls.
3f881c9 Changed IStore delete to remove; parameterizable cassandra backend
Michael Meisinger authored May 26, 2010
164 Changed delete to remove to be more compliant with standard collections.
241d1d0 BaseProcess.reply_ok and reply_err added and service operations chang…
Michael Meisinger authored May 22, 2010
165 2010-05-22:
166 - Added timeout to BaseProcess.rpc_send. Use with kwarg timeout=<secs>
167 - CC-Agent detects missing known containers and removes them from the list
168 - Enhanced CC-Agent operations and CC shell helpers
169 - Added sequence numbers for messages
170 - Added glue functions BaseProcess.reply_ok and reply_err and changes some
171 RPC style service operations.
342829f Local configuration override in ionlocal.config. Unit tests get broke…
Michael Meisinger authored May 21, 2010
172 2010-05-20:
173 - The system now looks for a local config file ionlocal.config and if exists,
174 overrides entries in ion.config.
175 - Test cases use the config file to determine the broker host to use. If local
70573b4 added BaseProcessClient and changed all BaseServiceClient subclasses …
Michael Meisinger authored May 20, 2010
176 config override exists, a different broker (e.g. localhost) can be given.
177 - Added BaseProcessClient and changed BaseServiceClient and all clients and
178 all test cases (again).
3827a32 shell helper functions
Michael Meisinger authored May 21, 2010
179 - Added container shell helpers under 'cc', such as cc.spawn('hello')
e710e44 CC agent with announcement on identification request; can spawn a pro…
Michael Meisinger authored May 17, 2010
180 2010-05-16:
181 - Removed support for BaseProcess.send_message and reply_message. Always use
182 send, reply and rpc_send now.
183 - Any BaseProcess instance can now spawn_child() other processes.
184 - Removed RpcClient class, because every process can do rpc_send()
185 - Service processes now also listen to their service name's queue. The service
186 name is determined from the service declaration. Two processes will listen
187 to the same queue and take messages round robin from the queue.
188 - Startup arguments evaluated, for instance to start with system name set:
189 twistd -n magnet -a sysname=mysys
190 twistd -n magnet -a "{'sysname':'mysys'}"
191 - Added capability container agent process. Start with:
192 twistd -n magnet res/scripts/newcc.py
193 Agents announce themselves to others in the same system and can spawn procs.
194 - Name scope 'local' for messaging names means now really local to one container.
195 Use scope 'system' for names unique for each bootstrapped system. Do not use
196 global names, because they will clash.
197 - Less verbose trace output for process init messages and changes to other
198 trace output as well.
bc50a38 Changed BaseServiceClient __init__ args and all client impls; takes n…
Michael Meisinger authored May 16, 2010
199 - Changed BaseServiceClient and tests. Initializer arguments different.
cad868a Updated README to correct magnet dependency: now master
Michael Meisinger authored May 18, 2010
200 - Using master branch of magnet now
7a9f54a python 2.5 compatibility: changed .format to % and json import
Michael Meisinger authored May 11, 2010
201 2010-05-10:
202 - Based on entries in config files, service process modules are sought and
203 loaded in order to collect the service process declarations. This enables
204 a registration of services with versions and dependencies on startup
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 11, 2010
205 2010-05-06:
206 - Refactored the BaseProcess class.
207 - You can now do RPC directly from the process, via self.rpc_send without
5ea10a7 made IonTestCase and all tests PEP8 compliant
Michael Meisinger authored May 10, 2010
208 the need for an RpcClient. Works even with other messages coming in at the
209 same time (using the conv-id)
c43dc6c Readme appended with last changes, OTP functions in base_process
Michael Meisinger authored May 11, 2010
210 - Added aliases: self.send, self.reply (use these now)
211 - Process instance can be spawned via self.spawn()
212 - Provided an easier to use BaseServiceClient, which uses a default service
213 name lookup. Accepts BaseProcess instance as argument to use for sending/
214 receiving service calls.
Something went wrong with that request. Please try again.