Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 671 lines (495 sloc) 27.076 kb
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
1 Proposal - Fedora Messaging with 0mq (``fedmsg``)
2 =================================================
3
4 .. contents::
5
6 This proposal reviews existing services in the Fedora infrastructure, reviews
7 the problem of complexity in the interaction of those services, reviews previous
8 work by the `Fedora Messaging SIG (special interest group)
9 <http://fedoraproject.org/wiki/Messaging_SIG>`_ on AMQP, and introduces an
10 architecture-level description of a solution using 0mq.
11
12 ----
13
117fb04 @ralphbean Added link to the mailing list to the docs.
authored
14 For discussion, check
15 https://admin.fedoraproject.org/mailman/listinfo/messaging-sig
16
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
17 Get (or modify) the source for this document:
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
18 http://github.com/fedora-infra/fedmsg
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
19
20 Authors:
21 - Ralph Bean (threebean)
22
c2f526e @ralphbean Copy proposal -> overview.
authored
23 .. note:: This document is out of date and should just be considered a
24 "historical" document. What you're probably looking for is the
25 :doc:`overview` document: the start of the narrative documentation. It is
26 the most-up-to-date version of this page.
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
27
b39b07f @ralphbean Added a tl;dr section before the introduction.
authored
28 tl;dr
29 =====
30
31 We want to hook all the services in Fedora Infrastructure up to send messages to
32 one another over a message bus instead of communicating with each other in
33 heterogenous ways they do now.
34
35 We're writing a python library called ``fedmsg`` to help apps handle this more
36 easily. It's built on `0mq <http://zeromq.org>`_ and `moksha
37 <http://moksha.fedorahosted.org>`_.
38
39 Planned Stages of development/deployment
40 ----------------------------------------
41
42 1) Start writing ``fedmsg``
43 2) Send messages from existing services (koji, bodhi, pkgdb, fas, etc...).
44 3) Consume messages for statistics, i.e. an independent statistics webapp.
45 4) Consume messages for user experience, i.e. any or all of rss, email,
46 gnome-shell notifications, javascript notifications in FI webapps.
47 5) Consume messages for service interoperability, i.e. koji invalidates it's
48 cache when it sees pkgdb messages go by on the bus. This comes last because
49 we want to make sure that message-sending works and is reliable before we
50 start making existing services depend on it for their functioning.
51
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
52 Introduction
53 ============
54
55 Description of the problem
56 --------------------------
57
58 Fedora Infrastructure is composed of a number of services (koji, fedpkg, pkgdb,
59 etc..) some of which are maintained outside the Fedora Project and some of which
60 were built in-house by the infrastructure team. These are strung together in
61 a pipeline. Think "how an upstream release becomes a package update", "How a
62 new source distribution becomes a package."
63
64 At present, many of the steps in this process require the maintainer to wait and
65 watch for a previous step to complete. For instance once a branch of a
66 package is successfully built in koji, the maintainer must `submit their
67 update to bodhi
68 <http://fedoraproject.org/wiki/PackageMaintainers/UpdatingPackageHowTo#Submit_your_update_to_Bodhi>`_
69 (See the `new package process
70 <http://fedoraproject.org/wiki/New_package_process_for_existing_contributors>`_
71 for more details).
72
73 Other progressions in the pipeline are automated. For instance, `AutoQA
74 <http://fedoraproject.org/wiki/AutoQA_architecture>`_ defines a `set of
75 watchers
76 <http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=watchers;h=af4f6d5e68e9dfcff938d0481ac65fa52bcd1d17;hb=HEAD>`_.
77 Most watchers are run as a cron task. Each one looks for `certain events
78 <http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=events>`_ and fires off
79 tests when appropriate.
80
81 At LinuxFest Northwest (2009), jkeating gave `a talk
82 <http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf>`_ on the problem of
83 complexity in the Fedora infrastructure and how this might be addressed with a
84 message bus architecture. Each service in the infrastructure depends on
85 many of the others. Some pieces directly poke others: git (fedpkg) currently
86 pokes AutoQA from a post-update hook. Other pieces poll others' status: koji
87 scrapes pkgdb for package-owner relationships and email aliases.
88
89 This dense coupling of services makes changing, adding, or replacing services
90 more complicated: commits to one project require a spidering of code changes
91 to all the others.
92
93 How messaging might address the problem
94 ---------------------------------------
95
96 jkeating's `talk on messaging in the Fedora Instructure
97 <http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf>`_ proposed the
98 adoption of a unified message bus to reduce the complexity of multiple
99 interdependent services. Instead of a service interfacing with its
100 dependencies' implementations, it could subscribe to a `topic`, provide a
101 callback, and respond to events.
102
103 For instance, instead of having koji scrape pkgdb on an interval for changed
104 email addresses, pkgdb could emit messages to the
1815373 @ralphbean Refine hypothetical topics to match proposed topic conventions.
authored
105 ``org.fedoraproject.service.pkgdb`` topic whenever an account's email address
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
106 changes. koji could subscribe to the same topic and provide a callback that
107 updates its local email aliases when invoked.
108
109 In another case, the git (fedpkg) post-update hook could publish messages to
1815373 @ralphbean Refine hypothetical topics to match proposed topic conventions.
authored
110 the ``org.fedoraproject.service.fedpkg.post-update`` topic. AutoQA could
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
111 subscribe to the same. Now if we wanted to enable another service to act when
112 updates are pushed to fedpkg, that service need only subscribe to the topic and
113 implement its own callback instead of appending its own call to fedpkg's
114 post-update hook (instead of coupling its own implementation with fedpkg's).
115
116 A message bus architecture, once complete, would dramatically reduce the work
117 required to update and maintain services in the Fedora infrastructure.
118
270e92d @ralphbean Diagrams, more writing.
authored
119 Other benefits
120 --------------
121
122 By adopting a messaging strategy for Fedora Infrastructure we could gain:
123
124 - A stream of data which we can watch and from which we can garner statistics
125 about infrastructure activity.
126 - The de-coupling of services from one another.
127 - libnotify notifications to developers' desktops.
128 - jquery.gritter.js notifications to web interfaces.
129
130 - this could be generalized to a ``fedmsg.wsgi`` middleware layer that
131 injects a fedora messaging dashboard header into every page served by apps
132 `X`, `Y`, and `Z`.
133
134 - An irc channel, #fedora-firehose that echoes every message on the bus.
135 - An identi.ca account, @fedora-firehose, that echoes every message on the bus.
136
0471e19 @ralphbean Removed references to QMF from the docs.
authored
137 AMQP, and 0mq
138 =============
270e92d @ralphbean Diagrams, more writing.
authored
139
0471e19 @ralphbean Removed references to QMF from the docs.
authored
140 AMQP or "Broker? Damn near killed 'er!"
141 ----------------------------------------
270e92d @ralphbean Diagrams, more writing.
authored
142
143 When discussions on the `Fedora Messaging SIG
144 <http://fedoraproject.org/wiki/Messaging_SIG>`_ began, AMQP was the choice by
145 default. Since then members of the SIG have become attracted to an alternative
146 messaging interface called `0mq <http://www.zeromq.org>`_.
147
148 Recommended reading:
149
150 - `What's wrong with AMQP
151 <http://www.imatix.com/articles:whats-wrong-with-amqp>`_
152
153 The following is recreated from J5's Publish/Subscribe Messaging Proposal
154 as an example of how Fedora Infrastructure could be reorganized with AMQP
155 and a set of federated AMQP brokers (qpid).
156
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
157 .. image:: https://github.com/fedora-infra/fedmsg/raw/develop/doc/_static/reorganize-amqp-j5.png
270e92d @ralphbean Diagrams, more writing.
authored
158
159 The gist is that each service in the Fedora Infrastructure would have the
160 address of a central message broker on hand. On startup, each service would
161 connect to that broker, ask the broker to establish its outgoing queues, and
162 begin publishing messages. Similarly, each service would ask the broker to
163 establish incoming queues for them. The broker would handle the routing of
164 messages based on ``routing_keys`` (otherwise known as `topics`) from each
165 service to the others.
166
167 The downshot, in short, is that AMQP requires standing up a single central
168 broker and thus a single-point-of-failure. In the author's work on `narcissus
169 <http://narcissus.rc.rit.edu>`_ I found that for even the most simple of AMQP
170 configurations, my qpid brokers' queues would bloat over time until \*pop\*,
171 the broker would fall over.
172
173 0mq or "Going for Broke(rless)"
174 -------------------------------
175
176 0mq is developed by a team that had a hand in the original development of AMQP.
177 It claims to be a number of things: an "intelligent transport layer",
178 a "socket library that acts as a concurrency framework", and the `sine qua non`
179 "Extra Spicy Sockets!"
180
181 Recommended reading:
182 - `The Z-guide <http://zguide.zeromq.org/page:all>`_
183
184 The following depicts an overview of a subset of Fedora Infrastructure
185 organized with a decentralized 0mq bus parallel to the spirit of J5's
186 recreated diagram in the AMQP section above.
187
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
188 .. image:: https://github.com/fedora-infra/fedmsg/raw/develop/doc/_static/reorganize-0mq-overview.png
270e92d @ralphbean Diagrams, more writing.
authored
189
190 No broker. The gist is that each service will open a port and begin
191 publishing messages ("bind to" in zmq-language). Each other service will
192 connect to that port to begin consuming messages. Without a central broker
193 doing `all the things
194 <http://www.imatix.com/articles:whats-wrong-with-amqp>`_, 0mq can afford a high
195 throughput. For instance, in initial tests of a 0mq-enabled `moksha hub
196 <http://moksha.fedorahosted.org>`_, the Fedora Engineering Team achieved a
197 100-fold speedup over AMQP.
198
199 Service discovery
200 ~~~~~~~~~~~~~~~~~
201
202 Shortly after you begin thinking over how to enable Fedora Infrastructure to
203 pass messages over a `fabric` instead of to a `broker`, you arrive at the
204 problem we'll call "service discovery".
205
206 In reality, (almost) every service both `produces` and `consumes` messages. For
207 the sake of argument, we'll talk here just about a separate `producing
208 service` and some `consuming services`.
209
210 Scenario: the producing service starts up, producing socket (with a hidden
211 queue), and begins producing messages. Consuming services `X`, `Y`, and `Z`
212 are interested in this and they would like to connect.
213
214 With AMQP, this is simplified. You have one central broker and each consuming
215 service need only know it's one address. They connect and the match-making is
216 handled for them. With 0mq, each consuming service needs to somehow
217 `discover` its producer(s) address(es).
218
eb8228c @ralphbean Finished section on service discovery.
authored
219 There are a number of ways to address this:
220
221 - *Write our own broker*; this would not be that difficult. We could (more
222 simply) scale back the project and write our own directory lookup service
223 that would match consumers with their providers. This could be done in
224 surprisingly few lines of python. This issue is that we re-introduce the
225 sticking point of AMQP, a single point of failure.
226
227 - *Use DNS*; There is a helpful `blog post
228 <http://www.ceondo.com/ecte/2011/12/dns-zeromq-services>`_ on how to do this
229 with `djbdns`. DNS is always there anyways: if DNS goes down, we have bigger
230 things to worry about than distributing updates to our messaging topology.
231
232 - *Share a raw text file*; This at first appears crude and cumbersome:
233
234 - Maintain a list of all `fedmsg`-enabled producers in a text file
235 - Make sure that file is accessible from every consuming service.
236 - Have each consuming service read in the file and connect to every
237 (relevant) producer in the list
238
239 In my opinion, using DNS is generally speaking the most elegant solution.
240 However, for Fedora Infrastructure in particular, pushing updates to DNS and
241 pushing a raw text file to every server involves much-the-same workflow:
242 `puppet`. Because much of the overhead of updating the text file falls in-line
243 with the rest of Infrastructure work, it makes more sense to go with the third
244 option. Better not to touch DNS when we don't have to.
245
755b07a @ralphbean Added example config to the documentation.
authored
246 That file is ``/etc/fedmsg-config.py``. It should define a python dict called
247 ``config`` which may look something like the following in a development
248 environment::
249
d5ae6ed @ralphbean Another config format change.
authored
250 # TODO -- update this. It is out of date.
755b07a @ralphbean Added example config to the documentation.
authored
251 config = dict(
252 # This is a dict of possible addresses from which fedmsg can send
253 # messages. fedmsg.init(...) requires that a 'name' argument be passed
254 # to it which corresponds with one of the keys in this dict.
255 endpoints=dict(
256 # For other, more 'normal' services, fedmsg will try to guess the
257 # name of it's calling module to determine which endpoint definition
258 # to use. This can be overridden by explicitly providing the name in
259 # the initial call to fedmsg.init(...).
260 bodhi="tcp://*:3001",
261 fas="tcp://*:3002",
262 fedoratagger="tcp://*:3003",
263
264 # This is the output side of the relay to which all other
265 # services can listen.
266 relay_outbound="tcp://*:4001",
267 ),
268
269 # This is the address of an active->passive relay. It is used for the
270 # fedmsg-logger command which requires another service with a stable
271 # listening address for it to send messages to.
272 relay_inbound="tcp://127.0.0.1:2003",
273
274 # Set this to dev if you're hacking on fedmsg or an app.
275 # Set to stg or prod if running in the Fedora Infrastructure
276 environment="dev",
277
278 # Default is 0
279 high_water_mark=1,
280
281 io_threads=1,
282 )
270e92d @ralphbean Diagrams, more writing.
authored
283
356b059 @ralphbean Added note about multiple fedmsg-config.py files.
authored
284 ``fedmsg`` will look for a config file in ``/etc/``, ``$HOME``, and ``.`` (the
285 current working directory). If it finds multiple files, it will read all of
286 them but overwrite values from the system (``/etc/``) file with the more local
287 file (``$HOME``).
288
270e92d @ralphbean Diagrams, more writing.
authored
289 Different buses
290 ---------------
291
2cb6e73 @ralphbean More clearly marked TODO items as such.
authored
292 TODO -
293
294 - critical and statistical buses (critical is subset of statistical).
270e92d @ralphbean Diagrams, more writing.
authored
295
296 Authn, authz
297 ------------
298
2cb6e73 @ralphbean More clearly marked TODO items as such.
authored
299 TODO -
300
301 - (func has certs laying around already).
bcba6da @ralphbean Added note from comphappy about pubsub security.
authored
302 - Read http://www.zeromq.org/topics:pubsub-security. ``comphappy`` reports
303 that it has some interesting points.
270e92d @ralphbean Diagrams, more writing.
authored
304
305 network load
306 ------------
307
2cb6e73 @ralphbean More clearly marked TODO items as such.
authored
308 TODO -
309
310 - calculate network load -
270e92d @ralphbean Diagrams, more writing.
authored
311 http://lists.zeromq.org/pipermail/zeromq-dev/2010-August/005254.html
312
313 fringe services
314 ---------------
315
2cb6e73 @ralphbean More clearly marked TODO items as such.
authored
316 TODO -
270e92d @ralphbean Diagrams, more writing.
authored
317
2cb6e73 @ralphbean More clearly marked TODO items as such.
authored
318 - example of building a relay that condenses messages from `n`
319 proxies and re-emits them.
320 - example of bridging amqp and 0mq
321 - bugzilla-push - https://github.com/LegNeato/bugzilla-push
270e92d @ralphbean Diagrams, more writing.
authored
322
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
323 Namespace considerations
324 ------------------------
325
326 In the above examples, the topic names are derived from the service names.
327 For instance, pkgdb publishes messages to
1815373 @ralphbean Refine hypothetical topics to match proposed topic conventions.
authored
328 ``org.fedoraproject.service.pkgdb*``, AutoQA presumably publishes messages
329 to ``org.fedoraproject.service.autoqa*``, and so on.
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
330
331 This convention, while clear-cut, has its limitations. Say we wanted to
332 replace pkgdb whole-sale with our shiney new `threebean-db` (tm). Here,
333 all other services are subscribed to topics that mention pkgdb explicitly.
334 Rolling out threebean-db will require patching every other service; we find
335 ourselves in a new flavor of the same complexity/co-dependency trap
336 described in the first section.
337
338 The above `service-oriented` topic namespace is one option.
339 Consider an `object-oriented` topic namespace where the objects are things
340 like users, packages, builds, updates, tests, tickets, and composes. Having
1815373 @ralphbean Refine hypothetical topics to match proposed topic conventions.
authored
341 bodhi subscribe to ``org.fedoraproject.object.tickets`` and
342 ``org.fedoraproject.object.builds`` leaves us less tied down to the current
cfcf56f @ralphbean Began adding comments to the top of tiling.coffee.
authored
343 implementation of the rest of the infrastructure. We could replace `bugzilla`
344 with `pivotal` and bodhi would never know the difference - a ticket is a
345 ticket.
346
d1bd732 @ralphbean And back to a service-oriented topic namespace.
authored
347 That would be nice; but there are too many objects in Fedora Infrastructure that
348 would step on each other. For instance, Koji **tags** packages and Tagger
349 **tags** packages; these two are very different things. Koji and Tagger cannot
350 **both** emit events over ``org.fedoraproject.package.tag.*`` without widespread
351 misery.
352
353 Consequently, our namespace follows a `service-oriented` pattern.
354
270e92d @ralphbean Diagrams, more writing.
authored
355 The scheme
356 ----------
357
d1bd732 @ralphbean And back to a service-oriented topic namespace.
authored
358 Event topics will follow the rule::
613b5d7 @ralphbean Decided on object-oriented namespace.
authored
359
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
360 org.fedoraproject.ENV.SERVICE.OBJECT[.SUBOBJECT].EVENT
613b5d7 @ralphbean Decided on object-oriented namespace.
authored
361
270e92d @ralphbean Diagrams, more writing.
authored
362 Where:
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
363
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
364 - ``ENV`` is one of `dev`, `stg`, or `production`.
095dae7 @ralphbean Examples of emitting events.
authored
365 - ``SERVICE`` is something like `koji`, `bodhi`, or `fedoratagger`
270e92d @ralphbean Diagrams, more writing.
authored
366 - ``OBJECT`` is something like `package`, `user`, or `tag`
367 - ``SUBOBJECT`` is something like `owner` or `build` (in the case where
368 ``OBJECT`` is `package`, for instance)
6c79b07 @ralphbean New -> Create in topic names.
authored
369 - ``EVENT`` is a verb like `update`, `create`, or `complete`.
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
370
1815373 @ralphbean Refine hypothetical topics to match proposed topic conventions.
authored
371 All 'fields' in a topic **must**:
372
373 - Be `singular` (Use `package`, not `packages`)
374 - Use existing fields as much as possible (since `complete` is already used
375 by other topics, use that instead of using `finished`).
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
376
377
270e92d @ralphbean Diagrams, more writing.
authored
378 Code Examples - 0mq and ``fedmsg``
379 ==================================
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
380
095dae7 @ralphbean Examples of emitting events.
authored
381 This package (the `package containing the docs you are reading right now
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
382 <http://github.com/fedora-infra/fedmsg>`_) is ``fedmsg``. It aims to be a wrapper
095dae7 @ralphbean Examples of emitting events.
authored
383 around calls to the `moksha hub <http://moksha.fedorahosted.org>`_ API that:
384
385 - Handles Fedora-Infra authn/authz
386 - Handles Fedora-Infra service discovery
387 - Helps you avoid topic and message content typos.
388 - Gets in your way as little as possible
389
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
390 Examples of emitting events
391 ---------------------------
392
095dae7 @ralphbean Examples of emitting events.
authored
393 Here's a real dummy test::
394
395 >>> import fedmsg
aec60ea @ralphbean Deprecate .send_message in favor of .publish. Fixes #28.
authored
396 >>> fedmsg.publish(topic='testing', modname='test', msg={
31d3d75 @ralphbean No longer using fedmsg schema/validation.
authored
397 ... 'test': "Hello World",
095dae7 @ralphbean Examples of emitting events.
authored
398 ... })
399
400 The above snippet will send the message ``'{test: "Hello World"}'`` message
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
401 over the ``org.fedoraproject.dev.test.testing`` topic.
8a27880 @ralphbean Fixed doc ref to guess_modname.
authored
402 The ``modname`` argument will be omitted in most use cases. By default,
403 ``fedmsg`` will try to guess the name of the module that called it and use
404 that to produce an intelligent topic.
405 Specifying ``modname`` argues that ``fedmsg`` not be `too smart`.
095dae7 @ralphbean Examples of emitting events.
authored
406
407 Here's an example from
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
408 `fedora-tagger <http://github.com/fedora-infra/fedora-tagger>`_ that sends the
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
409 information about a new tag over
410 ``org.fedoraproject.{dev,stg,prod}.fedoratagger.tag.update``::
095dae7 @ralphbean Examples of emitting events.
authored
411
412 >>> import fedmsg
aec60ea @ralphbean Deprecate .send_message in favor of .publish. Fixes #28.
authored
413 >>> fedmsg.publish(topic='tag.update', msg={
31d3d75 @ralphbean No longer using fedmsg schema/validation.
authored
414 ... 'user': user,
415 ... 'tag': tag,
095dae7 @ralphbean Examples of emitting events.
authored
416 ... })
417
418 Note that the `tag` and `user` objects are SQLAlchemy objects defined by
aec60ea @ralphbean Deprecate .send_message in favor of .publish. Fixes #28.
authored
419 tagger. They both have ``.__json__()`` methods which ``.publish``
095dae7 @ralphbean Examples of emitting events.
authored
420 uses to convert both objects to stringified JSON for you.
421
422 ``fedmsg`` has also guessed the module name (``modname``) of it's caller and
423 inserted it into the topic for you. The code from which we stole the above
424 snippet lives in ``fedoratagger.controllers.root``. ``fedmsg`` figured that
425 out and stripped it down to just ``fedoratagger`` for the final topic of
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
426 ``org.fedoraproject.{dev,stg,prod}.fedoratagger.tag.update``.
095dae7 @ralphbean Examples of emitting events.
authored
427
d953b5e @ralphbean Docs for fedmsg-logger --json-input.
authored
428 ----
429
430 You could also use the ``fedmsg-logger`` from a shell script like so::
431
432 $ echo "Hello, world." | fedmsg-logger --topic testing
433 $ echo '{"foo": "bar"}' | fedmsg-logger --json-input
434
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
435 Examples of consuming events
436 ----------------------------
437
d5a0839 @ralphbean Documentation for consuming messages.
authored
438 Consuming events is accomplished by way of the fedmsg-hub. For example,
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
439 in the `busmon <https://github.com/fedora-infra/busmon>`_ app, all messages from
d5a0839 @ralphbean Documentation for consuming messages.
authored
440 the hub are processed to be formatted and displayed on a client's browser. We
441 mark them up with a pretty-print format and use pygments to colorize them.
442
443 Here are the *important* parts: you must define a new class which extends
444 ``moksha.api.hub:Consumer``, declares a ``topic`` attribute and a ``consume``
445 method. The topic is used soley for constraining what messages make their way
446 to the consumer; the consumer can *send* messages on any topic. You may use
447 'splats' ('*') in the topic and subscribe to ``'org.fedoraproject.stg.koji.*'``
448 to get all of the messages from koji in the staging environment. In the example
449 below, the ``MessageColorizer`` consumer simply subscribes to '*'; it will
450 receive every message that hits it's local fedmsg-hub.
451
58e7c85 @ralphbean s/ralphbean/fedora-infra/g
authored
452 Here's the full example from `busmon <https://github.com/fedora-infra/busmon>`_, it
d5a0839 @ralphbean Documentation for consuming messages.
authored
453 consumes messages from every topic, formats them in pretty colored HTML and then
454 re-sends them out on a new topic::
455
456 import pygments.lexers
457 import pygments.formatters
458 from moksha.api.hub import Consumer
459
460 import fedmsg
e4e48e5 @ralphbean simplejson -> json.
authored
461 import fedmsg.encoding
d5a0839 @ralphbean Documentation for consuming messages.
authored
462
463 class MessageColorizer(Consumer):
464 topic = "*"
465 jsonify = False
466
467 destination_topic = "colorized-messages"
468
469 def consume(self, message):
470 # Just so we don't create an infinite feedback loop.
471 if self.destination_topic in message.topic:
472 return
473
474 # Format the incoming message
475 code = pygments.highlight(
e4e48e5 @ralphbean simplejson -> json.
authored
476 fedmsg.encoding.pretty_dumps(fedmsg.encoding.loads(message.body)),
d5a0839 @ralphbean Documentation for consuming messages.
authored
477 pygments.lexers.JavascriptLexer(),
478 pygments.formatters.HtmlFormatter(full=False)
479 ).strip()
480
481 # Ship it!
aec60ea @ralphbean Deprecate .send_message in favor of .publish. Fixes #28.
authored
482 fedmsg.publish(
d5a0839 @ralphbean Documentation for consuming messages.
authored
483 topic=self.destination_topic,
484 msg=code,
485 )
486
487 Now, just defining a consumer isn't enough to have it picked up by the ``fedmsg-hub`` when it runs. You must also declare the consumer as an entry-point in your app's ``setup.py``, like this::
488
489 setup(
490 ...
491 entry_points={
492 'moksha.consumer': (
493 'colorizer = busmon.consumers:MessageColorizer',
494 ),
495 },
496 )
497
498 At initialization, ``fedmsg-hub`` looks for all the objects registered
499 on the ``moksha.consumer`` entry point and loads them
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
500
e939fff @ralphbean Added a note to the docs about console scripts.
authored
501 Console Scripts
502 ---------------
503
504 It makes sense for ``fedmsg`` to also provide a number of console scripts for
505 use with random shell scripts or with nagios, for instance.
506
507 Currently we have implemented:
508
abf4f6c @ralphbean Added new console scripts to the docs.
authored
509 - ``fedmsg-tail`` - watches all endpoints on the bus and prints each message to
510 stdout.
3f4935f @ralphbean Added FI environment name as a field to the topic namespace.
authored
511 - ``fedmsg-logger`` - sends messages over the ``org.fedoraproject.dev.logger``
abf4f6c @ralphbean Added new console scripts to the docs.
authored
512 topic. This requires that an instance of ``fedmsg-relay`` be running
513 *somewhere* and that it's inbound address be listed in ``fedmsg-config.py``.
514 - ``fedmsg-relay`` - a service which binds to two ports, listens for messages
515 on one and emits them on the other. ``fedmsg-logger`` requires that an
516 instance of ``fedmsg-relay`` be running *somewhere* and that it's inbound
517 address be listed in ``fedmsg-config.py``.
d5a0839 @ralphbean Documentation for consuming messages.
authored
518 - ``fedmsg-hub`` - the all-purpose daemon. This should be run on every host
519 that has services which declare their own consumers. ``fedmsg-hub`` will
520 listen to every endpoint defined in ``/etc/fedmsg-config.py`` and forward
521 messages in-process to the locally-declared consumers.
e939fff @ralphbean Added a note to the docs about console scripts.
authored
522
feaeee4 @ralphbean Elaboration on topics.
authored
523 Systems and Events
524 ==================
525
526 All messages will be transmitted as stringified JSON.
527
528 List of systems, their events, and associated fields
529 ----------------------------------------------------
530
531 Each item here is a service followed by the list of events that it emits. Each
532 event is followed by a list of services that will likely consume that event.
533
992f378 @ralphbean Added a status table to the docs.
authored
534 See also :doc:`status`.
535
feaeee4 @ralphbean Elaboration on topics.
authored
536 ----
537
d83cf65 @ralphbean Added AskBot TODO to the docs.
authored
538 - AskBot
539
540 - TODO - Brainstorm a list of potential message topics.
541
feaeee4 @ralphbean Elaboration on topics.
authored
542 - AutoQA
543
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
544 - TODO - Add these hooks. j_dulaney is working on this.
545
546 - ``org.fedoraproject.{stg,prod}.autoqa.package.tests.complete`` -> koji, bodhi, fcomm
feaeee4 @ralphbean Elaboration on topics.
authored
547
548 - Bodhi
549
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
550 - This is done in a branch in git.
551 https://fedorahosted.org/bodhi/browser/bodhi/model.py?rev=1712d35e79ea3c27b7134006f0afa62ffd7f1769#L446
552 TODO - merge and push to stg then prod
553
554 - ``org.fedoraproject.{stg,prod}.bodhi.update.request{.TYPE}`` -> fcomm, autoqa
555 - ``org.fedoraproject.{stg,prod}.bodhi.update.complete{.TYPE}`` -> fcomm, autoqa
556
557 - TODO - These hooks still need to be added.
558
559 - ``org.fedoraproject.{stg,prod}.bodhi.update.push`` -> fcomm
560 - ``org.fedoraproject.{stg,prod}.bodhi.update.remove`` -> fcomm
feaeee4 @ralphbean Elaboration on topics.
authored
561
562 - Bugzilla
563
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
564 - TODO - get AMQP messages from redhat. Run a service to translate.
565
566 - ``org.fedoraproject.{stg,prod}.bugzilla.bug.create`` -> fcomm
567 - ``org.fedoraproject.{stg,prod}.bugzilla.bug.update`` -> fcomm
feaeee4 @ralphbean Elaboration on topics.
authored
568
569 - Compose
570
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
571 - TODO - Add the hooks
572
573 - ``org.fedoraproject.{stg,prod}.compose.compose.complete`` -> mirrormanager, autoqa
feaeee4 @ralphbean Elaboration on topics.
authored
574
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
575 - Elections (TODO -- what is the app called?)
576
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
577 - TODO - Add the hooks
578
579 - ``org.fedoraproject.{stg,prod}.elections...`` <-- TODO. Objects and events?
580
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
581
feaeee4 @ralphbean Elaboration on topics.
authored
582 - FAS
583
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
584 - All of these hooks have been added.
585 TODO - merge and push to stg then prod.
586
587 - ``org.fedoraproject.{stg,prod}.fas.user.create`` -> fcomm
588 - ``org.fedoraproject.{stg,prod}.fas.user.update`` -> fcomm
589 - ``org.fedoraproject.{stg,prod}.fas.group.update`` -> fcomm
590 - ``org.fedoraproject.{stg,prod}.fas.group.member.apply`` -> fcomm
591 - ``org.fedoraproject.{stg,prod}.fas.group.member.sponsor`` -> fcomm
592 - ``org.fedoraproject.{stg,prod}.fas.group.member.sponsor`` -> fcomm
593 - ``org.fedoraproject.{stg,prod}.fas.group.create`` -> fcomm
594 - ``org.fedoraproject.{stg,prod}.fas.group.update`` -> fcomm
595 - ``org.fedoraproject.{stg,prod}.fas.role.update`` -> fcomm
feaeee4 @ralphbean Elaboration on topics.
authored
596
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
597 - Koji
feaeee4 @ralphbean Elaboration on topics.
authored
598
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
599 - TODO - Add the hooks
600
601 - ``org.fedoraproject.{stg,prod}.koji.tag.build`` -> secondary arch koji
602 - ``org.fedoraproject.{stg,prod}.koji.tag.create`` -> secondary arch koji
603 - ``org.fedoraproject.{stg,prod}.koji.package.build.complete`` -> fcomm,
604 secondary arch koji, SCM, autoqa, sigul
605 - ``org.fedoraproject.{stg,prod}.koji.package.build.start`` -> fcomm
606 - ``org.fedoraproject.{stg,prod}.koji.package.build.fail`` -> fcomm
feaeee4 @ralphbean Elaboration on topics.
authored
607
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
608 - MeetBot (supybot?)
609
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
610 - TODO - Add the hooks
611
612 - ``org.fedoraproject.{stg,prod}.irc.meeting.start``
613 - ``org.fedoraproject.{stg,prod}.irc.meeting.complete``
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
614
613b5d7 @ralphbean Decided on object-oriented namespace.
authored
615 - NetApp -- FIXME, the topics from netapp should be reviewed. They seem
616 ambiguous.
feaeee4 @ralphbean Elaboration on topics.
authored
617
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
618 - TODO - Add the hooks
619
620 - ``org.fedoraproject.{stg,prod}.netapp.sync.stop`` -> mirrormanager
621 - ``org.fedoraproject.{stg,prod}.netapp.sync.resume`` -> mirrormanager
feaeee4 @ralphbean Elaboration on topics.
authored
622
623 - PkgDB
624
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
625 - TODO - Add the hooks
626
627 - ``org.fedoraproject.{stg,prod}.pkgdb.package.create`` -> koji, secondary arch koji, bugzilla
628 - ``org.fedoraproject.{stg,prod}.pkgdb.package.remove`` -> koji, secondary arch koji,
629 - ``org.fedoraproject.{stg,prod}.pkgdb.package.rename`` -> bugzilla
630 - ``org.fedoraproject.{stg,prod}.pkgdb.package.retire`` -> SCM
631 - ``org.fedoraproject.{stg,prod}.pkgdb.package.owner.update`` -> koji, secondary arch koji, bugzilla
632 - TODO - lots of ``org.fp.user...`` events to detail here.
feaeee4 @ralphbean Elaboration on topics.
authored
633
634 - SCM
635
0d8933c @ralphbean Stub of a post-receive hook and a note in the docs about next steps.
authored
636 - TODO - Add the hooks. This is blocking on getting an instance of
637 fedmsg-relay stood up in production. That, on the other hand, is blocking
638 on getting the fedmsg wrapper around moksha done so that the relay doesn't
639 eat up 100% CPU.
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
640
641 - ``org.fedoraproject.{stg,prod}.scm.repo.checkin`` -> fcomm, autoqa
feaeee4 @ralphbean Elaboration on topics.
authored
642
cb95033 @ralphbean Reorganized sections. Added tagger events.
authored
643 - Tagger
644
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
645 - These hooks have been added. Need to push to stg then prod.
646
647 - ``org.fedoraproject.{stg,prod}.fedoratagger.tag.create`` -> fcomm, pkgdb
648 - ``org.fedoraproject.{stg,prod}.fedoratagger.tag.remove`` -> fcomm, pkgdb
649 - ``org.fedoraproject.{stg,prod}.fedoratagger.tag.update`` -> fcomm, pkgdb
650 - ``org.fedoraproject.{stg,prod}.fedoratagger.user.rank.update`` -> fcomm, (pkgdb?)
651 - ``org.fedoraproject.{stg,prod}.fedoratagger.login`` -> ??
feaeee4 @ralphbean Elaboration on topics.
authored
652
9573fba @ralphbean Note about the mediawiki messages in the docs.
authored
653 - Wiki. This is implemented as a mediawiki plugin in
654 ``extras/mediawiki/fedmsg-mediawiki-emit.php``.
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
655
9573fba @ralphbean Note about the mediawiki messages in the docs.
authored
656 - ``org.fedoraproject.{stg,prod}.wiki.article.edit``
657 - ``org.fedoraproject.{stg,prod}.wiki.upload.complete``
bd7709c @ralphbean Added some ideas from #fedora-apps
authored
658
feaeee4 @ralphbean Elaboration on topics.
authored
659 - Zabbix
660
a3baf25 @ralphbean Reorganized list of topics by done and not-done.
authored
661 - TODO - Add the hooks
662
663 - ``org.fedoraproject.{stg,prod}.zabbix.service.update`` -> fcomm
a79853b @ralphbean Documented a few more ideas.
authored
664
665 Other Ideas
666 -----------
667
668 - Error messages from cron jobs
669 - The Nag-once script could be enhanced to send output to the bus
670 - Nagios alerts
Something went wrong with that request. Please try again.