Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 252 lines (195 sloc) 11.634 kb
875a86f Ralph Bean Fix broken section ordering.
authored
1 ========
74ad769 Ralph Bean Updates to overview.
authored
2 Overview
3 ========
c2f526e Ralph Bean Copy proposal -> overview.
authored
4
5 Description of the problem
fb77c06 Ralph Bean table of contents.
authored
6 ~~~~~~~~~~~~~~~~~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
7
8 Fedora Infrastructure is composed of a number of services (koji, fedpkg, pkgdb,
9 etc..) some of which are maintained outside the Fedora Project and some of which
10 were built in-house by the infrastructure team. These are strung together in
11 a pipeline. Think "how an upstream release becomes a package update", "How a
12 new source distribution becomes a package."
13
14 At present, many of the steps in this process require the maintainer to wait and
15 watch for a previous step to complete. For instance once a branch of a
16 package is successfully built in koji, the maintainer must `submit their
17 update to bodhi
18 <http://fedoraproject.org/wiki/PackageMaintainers/UpdatingPackageHowTo#Submit_your_update_to_Bodhi>`_
19 (See the `new package process
20 <http://fedoraproject.org/wiki/New_package_process_for_existing_contributors>`_
21 for more details).
22
23 Other progressions in the pipeline are automated. For instance, `AutoQA
24 <http://fedoraproject.org/wiki/AutoQA_architecture>`_ defines a `set of
25 watchers
26 <http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=watchers;h=af4f6d5e68e9dfcff938d0481ac65fa52bcd1d17;hb=HEAD>`_.
27 Most watchers are run as a cron task. Each one looks for `certain events
28 <http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=events>`_ and fires off
29 tests when appropriate.
30
31 At LinuxFest Northwest (2009), jkeating gave `a talk
32 <http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf>`_ on the problem of
33 complexity in the Fedora infrastructure and how this might be addressed with a
34 message bus architecture. Each service in the infrastructure depends on
35 many of the others. Some pieces directly poke others: git (fedpkg) currently
36 pokes AutoQA from a post-update hook. Other pieces poll others' status: koji
37 scrapes pkgdb for package-owner relationships and email aliases.
38
39 This dense coupling of services makes changing, adding, or replacing services
40 more complicated: commits to one project require a spidering of code changes
41 to all the others.
42
43 How messaging might address the problem
fb77c06 Ralph Bean table of contents.
authored
44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
45
46 jkeating's `talk on messaging in the Fedora Instructure
47 <http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf>`_ proposed the
48 adoption of a unified message bus to reduce the complexity of multiple
49 interdependent services. Instead of a service interfacing with its
50 dependencies' implementations, it could subscribe to a `topic`, provide a
51 callback, and respond to events.
52
53 For instance, instead of having koji scrape pkgdb on an interval for changed
54 email addresses, pkgdb could emit messages to the
55 ``org.fedoraproject.service.pkgdb`` topic whenever an account's email address
56 changes. koji could subscribe to the same topic and provide a callback that
57 updates its local email aliases when invoked.
58
59 In another case, the git (fedpkg) post-update hook could publish messages to
60 the ``org.fedoraproject.service.fedpkg.post-update`` topic. AutoQA could
61 subscribe to the same. Now if we wanted to enable another service to act when
62 updates are pushed to fedpkg, that service need only subscribe to the topic and
63 implement its own callback instead of appending its own call to fedpkg's
64 post-update hook (instead of coupling its own implementation with fedpkg's).
65
66 A message bus architecture, once complete, would dramatically reduce the work
67 required to update and maintain services in the Fedora infrastructure.
68
69 Other benefits
fb77c06 Ralph Bean table of contents.
authored
70 ~~~~~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
71
72 By adopting a messaging strategy for Fedora Infrastructure we could gain:
73
74 - A stream of data which we can watch and from which we can garner statistics
75 about infrastructure activity.
76 - The de-coupling of services from one another.
77 - libnotify notifications to developers' desktops.
78 - jquery.gritter.js notifications to web interfaces.
79
80 - this could be generalized to a ``fedmsg.wsgi`` middleware layer that
81 injects a fedora messaging dashboard header into every page served by apps
82 `X`, `Y`, and `Z`.
83
1293ddd Ralph Bean Updates to overview.
authored
84 - An irc channel, ``#fedora-fedmsg`` that echoes every message on the bus.
c2f526e Ralph Bean Copy proposal -> overview.
authored
85 - An identi.ca account, @fedora-firehose, that echoes every message on the bus.
86
87 AMQP, and 0mq
fb77c06 Ralph Bean table of contents.
authored
88 -------------
c2f526e Ralph Bean Copy proposal -> overview.
authored
89
90 AMQP or "Broker? Damn near killed 'er!"
fb77c06 Ralph Bean table of contents.
authored
91 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
92
93 When discussions on the `Fedora Messaging SIG
94 <http://fedoraproject.org/wiki/Messaging_SIG>`_ began, AMQP was the choice by
95 default. Since then members of the SIG have become attracted to an alternative
96 messaging interface called `0mq <http://www.zeromq.org>`_.
97
98 Recommended reading:
99
100 - `What's wrong with AMQP
101 <http://www.imatix.com/articles:whats-wrong-with-amqp>`_
102
103 The following is recreated from J5's Publish/Subscribe Messaging Proposal
104 as an example of how Fedora Infrastructure could be reorganized with AMQP
105 and a set of federated AMQP brokers (qpid).
106
1293ddd Ralph Bean Updates to overview.
authored
107 .. image:: _static/reorganize-amqp-j5.png
c2f526e Ralph Bean Copy proposal -> overview.
authored
108
109 The gist is that each service in the Fedora Infrastructure would have the
110 address of a central message broker on hand. On startup, each service would
111 connect to that broker, ask the broker to establish its outgoing queues, and
112 begin publishing messages. Similarly, each service would ask the broker to
113 establish incoming queues for them. The broker would handle the routing of
114 messages based on ``routing_keys`` (otherwise known as `topics`) from each
115 service to the others.
116
117 The downshot, in short, is that AMQP requires standing up a single central
118 broker and thus a single-point-of-failure. In the author's work on `narcissus
119 <http://narcissus.rc.rit.edu>`_ I found that for even the most simple of AMQP
120 configurations, my qpid brokers' queues would bloat over time until \*pop\*,
121 the broker would fall over.
122
123 0mq or "Going for Broke(rless)"
fb77c06 Ralph Bean table of contents.
authored
124 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
125
126 0mq is developed by a team that had a hand in the original development of AMQP.
127 It claims to be a number of things: an "intelligent transport layer",
128 a "socket library that acts as a concurrency framework", and the `sine qua non`
129 "Extra Spicy Sockets!"
130
131 Recommended reading:
132 - `The Z-guide <http://zguide.zeromq.org/page:all>`_
133
134 The following depicts an overview of a subset of Fedora Infrastructure
135 organized with a decentralized 0mq bus parallel to the spirit of J5's
136 recreated diagram in the AMQP section above.
137
74ad769 Ralph Bean Updates to overview.
authored
138 .. image:: _static/reorganize-0mq-overview.png
c2f526e Ralph Bean Copy proposal -> overview.
authored
139
140 No broker. The gist is that each service will open a port and begin
141 publishing messages ("bind to" in zmq-language). Each other service will
142 connect to that port to begin consuming messages. Without a central broker
143 doing `all the things
144 <http://www.imatix.com/articles:whats-wrong-with-amqp>`_, 0mq can afford a high
145 throughput. For instance, in initial tests of a 0mq-enabled `moksha hub
146 <http://moksha.fedorahosted.org>`_, the Fedora Engineering Team achieved a
147 100-fold speedup over AMQP.
148
149 Service discovery
150 ~~~~~~~~~~~~~~~~~
151
152 Shortly after you begin thinking over how to enable Fedora Infrastructure to
153 pass messages over a `fabric` instead of to a `broker`, you arrive at the
154 problem we'll call "service discovery".
155
156 In reality, (almost) every service both `produces` and `consumes` messages. For
157 the sake of argument, we'll talk here just about a separate `producing
158 service` and some `consuming services`.
159
1293ddd Ralph Bean Updates to overview.
authored
160 Scenario: the producing service starts up a producing socket (with a hidden
161 queue) and begins producing messages. Consuming services `X`, `Y`, and `Z`
c2f526e Ralph Bean Copy proposal -> overview.
authored
162 are interested in this and they would like to connect.
163
164 With AMQP, this is simplified. You have one central broker and each consuming
165 service need only know it's one address. They connect and the match-making is
166 handled for them. With 0mq, each consuming service needs to somehow
167 `discover` its producer(s) address(es).
168
169 There are a number of ways to address this:
170
171 - *Write our own broker*; this would not be that difficult. We could (more
172 simply) scale back the project and write our own directory lookup service
173 that would match consumers with their providers. This could be done in
174 surprisingly few lines of python. This issue is that we re-introduce the
175 sticking point of AMQP, a single point of failure.
176
177 - *Use DNS*; There is a helpful `blog post
178 <http://www.ceondo.com/ecte/2011/12/dns-zeromq-services>`_ on how to do this
179 with `djbdns`. DNS is always there anyways: if DNS goes down, we have bigger
180 things to worry about than distributing updates to our messaging topology.
181
182 - *Share a raw text file*; This at first appears crude and cumbersome:
183
184 - Maintain a list of all `fedmsg`-enabled producers in a text file
185 - Make sure that file is accessible from every consuming service.
186 - Have each consuming service read in the file and connect to every
187 (relevant) producer in the list
188
189 In my opinion, using DNS is generally speaking the most elegant solution.
190 However, for Fedora Infrastructure in particular, pushing updates to DNS and
191 pushing a raw text file to every server involves much-the-same workflow:
192 `puppet`. Because much of the overhead of updating the text file falls in-line
193 with the rest of Infrastructure work, it makes more sense to go with the third
194 option. Better not to touch DNS when we don't have to.
195
1293ddd Ralph Bean Updates to overview.
authored
196 That configuration is kept in ``/etc/fedmsg.d/``, is read by the code in
bca44c1 Ralph Bean Typofix.
authored
197 :mod:`fedmsg.config`. The config value of interest is :term:`endpoints`.
c2f526e Ralph Bean Copy proposal -> overview.
authored
198
199 Namespace considerations
fb77c06 Ralph Bean table of contents.
authored
200 ------------------------
c2f526e Ralph Bean Copy proposal -> overview.
authored
201
202 In the above examples, the topic names are derived from the service names.
203 For instance, pkgdb publishes messages to
204 ``org.fedoraproject.service.pkgdb*``, AutoQA presumably publishes messages
205 to ``org.fedoraproject.service.autoqa*``, and so on.
206
207 This convention, while clear-cut, has its limitations. Say we wanted to
208 replace pkgdb whole-sale with our shiney new `threebean-db` (tm). Here,
209 all other services are subscribed to topics that mention pkgdb explicitly.
210 Rolling out threebean-db will require patching every other service; we find
211 ourselves in a new flavor of the same complexity/co-dependency trap
212 described in the first section.
213
214 The above `service-oriented` topic namespace is one option.
215 Consider an `object-oriented` topic namespace where the objects are things
216 like users, packages, builds, updates, tests, tickets, and composes. Having
217 bodhi subscribe to ``org.fedoraproject.object.tickets`` and
218 ``org.fedoraproject.object.builds`` leaves us less tied down to the current
219 implementation of the rest of the infrastructure. We could replace `bugzilla`
220 with `pivotal` and bodhi would never know the difference - a ticket is a
221 ticket.
222
223 That would be nice; but there are too many objects in Fedora Infrastructure that
224 would step on each other. For instance, Koji **tags** packages and Tagger
225 **tags** packages; these two are very different things. Koji and Tagger cannot
226 **both** emit events over ``org.fedoraproject.package.tag.*`` without widespread
227 misery.
228
229 Consequently, our namespace follows a `service-oriented` pattern.
230
231 The scheme
fb77c06 Ralph Bean table of contents.
authored
232 ~~~~~~~~~~
c2f526e Ralph Bean Copy proposal -> overview.
authored
233
234 Event topics will follow the rule::
235
236 org.fedoraproject.ENV.SERVICE.OBJECT[.SUBOBJECT].EVENT
237
238 Where:
239
240 - ``ENV`` is one of `dev`, `stg`, or `production`.
241 - ``SERVICE`` is something like `koji`, `bodhi`, or `fedoratagger`
242 - ``OBJECT`` is something like `package`, `user`, or `tag`
243 - ``SUBOBJECT`` is something like `owner` or `build` (in the case where
244 ``OBJECT`` is `package`, for instance)
245 - ``EVENT`` is a verb like `update`, `create`, or `complete`.
246
1293ddd Ralph Bean Updates to overview.
authored
247 All 'fields' in a topic **should**:
c2f526e Ralph Bean Copy proposal -> overview.
authored
248
249 - Be `singular` (Use `package`, not `packages`)
250 - Use existing fields as much as possible (since `complete` is already used
251 by other topics, use that instead of using `finished`).
Something went wrong with that request. Please try again.