Skip to content
Browse files

A good deal more code examples.

  • Loading branch information...
1 parent 9c06387 commit 25498e5b6a242d2d7d77fa6e60d7112b35d842fc @gtaylor committed Apr 11, 2012
15 doc_src/access.rst
@@ -0,0 +1,15 @@
+.. _access:
+.. include:: global.txt
+Getting access to the EMDR network
+In order to get access to the EMDR network, you will merely need to connect
+to a relay. For your convenience, we have listed the relays below that have
+available capacity.
+* These will be added once we go live.
+Once you have chosen a relay, simply adapt the sample in
+:doc:`using` to use the hostname/port of the relay of your choice.
3 doc_src/global.txt
@@ -6,6 +6,9 @@
.. _virtualenv:
.. _virtualenvwrapper:
+.. _ZeroMQ:
+.. _gevent:
+.. _bottle:
.. _git:
.. _Supervisor:
BIN doc_src/images/emdr-daemon-diagram.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 doc_src/index.rst
@@ -5,7 +5,7 @@
EVE Market Data Relay (EMDR)
-EVE Market Data Relay is a super-scalable, super-reliable way to accept a large
+EVE Market Data Relay is a super-scalable, super-available way to accept a large
amount of user-submitted market data (via uploaders), and re-broadcast said
data in near real-time to a large number of subscribers.
@@ -36,6 +36,7 @@ Documentation
+ access
EMDR Developer Documentation
60 doc_src/overview.rst
@@ -1,5 +1,7 @@
.. _overview:
+.. include:: global.txt
A High-Level Overview
@@ -42,33 +44,30 @@ How it all fits together
For any given submitted market order, here is the flow said order goes through::
- (Gateway) -> (Broker) -> (Processor) -> (Relay) -> (Applications, other Relays)
+ (Gateway) -> (Broker) -> (Processor) -> (Announcer) -> (Applications and/or Relays)
First, the order hits the **Gateway**, which is a very light WSGI application.
The only purpose of the gateway is to queue the order for submission to
a **Broker** daemon. Brokers are used to distribute raw order data out to any
of the many **Processor** daemons. The number of processor daemons can scale
up and down without any modifications or restarts of any of the components,
and it is super easy for external people to volunteer their machines by running
-Processor daemons and getting permission to connect ot the Broker.
-Order data gets passed between each component via ZeroMQ_, which is an extremely
-scalable and performant transport/messaging layer.
+Processor daemons and getting permission to connect to the Broker.
-Getting back to our example above, the **Processor** gets raw order data from
-a **Broker**, looks at the raw data, parses it, performs some
-really simple validation/verification, then passes it on to our top level
-**Relay** via ZeroMQ_.
+A **Processor** instance gets raw order data from one of the **Broker** instances,
+looks at the raw data, parses it, performs some really simple
+validation/verification, then passes it on to our top level
+**Announcer** via ZeroMQ_. From there, the **Announcer** passes the data off
+to a **Relay**.
The **Relay** is a dumb repeater daemon. It takes the processed orders and just
spews them out to any subscribers. Subscribers can be other **Relay** daemons,
or actual user sites/applications. In the case of our eventual production
-deployment, the top-level (tier-1) **Relay** will only speak to other relays.
+deployment, the top-level (tier-1) **Announcer** will only speak to **Relays**.
The second level (tier-2) relays and lower are the ones that other
sites/applications can actually use. This keeps the load on our
-top-level (tier-1) relay to a minimum, *meanwhile, allowing anyone to volunteer
-to run tier-2 relays*.
-.. _ZeroMQ:
+top-level (tier-1) Announcer to a minimum, meanwhile, allowing anyone to
+volunteer to run tier-2 relays.
High Availability through shared burden
@@ -77,10 +76,45 @@ EMDR is architected in a way that allows every single component to be replicated
off-site. We can easily add additional daemons at each level of the stack in
order to increase throughput or availability.
+In the diagram below, we see what will be our initial deployment. Site 1 is
+comprised of EMDR running on Greg Taylor's (the project maintainer) machines,
+and Site 2 will be running on a trusted party's machines that are in another
+data center/region.
+At every step of the entire flow, we can afford to lose one of the two
+daemons, with no service interruption. The infrastructure can be scaled well
+out past the initial two sites, if need be.
+.. image:: images/emdr-daemon-diagram.png
EMDR contains no real security features of its own. Access will need to be
restricted at the OS/firewall level, and only trusted individuals should be
added to the stack.
+Security is something we take seriously, but let's consider the current
+reality of market data with EVE sites: *Players upload market data directly
+to market sites.* We are no less secure than that. Uploads can be faked,
+and malicious payloads can be sent, though EMDR will catch most of those
+and prevent them from being relayed.
+Technology Used
+This is the least interesting part of the overview, so it goes towards the
+* EMDR is written in Python.
+* All network-related stuff is handled by ZeroMQ_, which is an incredibly
+ simple and performant networking library.
+* gevent_ is used for their excellent greenlet-based Queue, Workers, and
+ async network I/O.
+* The gateway HTTP servers run bottle_.
+The entire stack is super low overhead, and very fast.
104 doc_src/using.rst
@@ -1,7 +1,107 @@
.. _using:
+.. include:: global.txt
Using data from the EMDR network
-This will contain details for how to connect your application to the EMDR
+Below are a few examples of how to connect to the data feed. If you see
+anything wrong with the examples below, please let us know on the
+`issue tracker`_. The original author of this documentation is only familiar
+with Python.
+.. warning:: Right now, the hostnames you see below are not in action. They
+ will become active once we complete our first deployment.
+The following example uses the pyzmq_ module (available off of PyPi)
+and simplejson_. You can substitute the stdlib's json_ module, if that suits
+your fancy::
+ #
+ # Example Python EMDR client.
+ #
+ import zmq
+ import simplejson
+ def main():
+ context = zmq.Context()
+ subscriber = context.socket(zmq.SUB)
+ # Connect to the first publicly available relay.
+ subscriber.connect('tcp://')
+ # Disable filtering.
+ subscriber.setsockopt(zmq.SUBSCRIBE, "")
+ while True:
+ # Receive raw market JSON strings.
+ market_json = subscriber.recv()
+ # Un-serialize the JSON data to a Python dict.
+ market_data = simplejson.loads(market_json)
+ # Dump the market data to stdout. Or, you know, do more fun
+ # things here.
+ print market_data
+ if __name__ == '__main__':
+ main()
+.. _pyzmq:
+.. _simplejson:
+.. _json:
+PHP accesses EMDR via ZeroMQ's `php-zmq`_ PHP bindings::
+ <?php
+ /*
+ * Example PHP EMDR client.
+ */
+ $context = new ZMQContext();
+ $subscriber = $context->getSocket(ZMQ::SOCKET_SUB);
+ // Connect to the first publicly available relay.
+ $subscriber->connect("tcp://");
+ // Disable filtering.
+ $subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, "");
+ while (true) {
+ $string = $subscriber->recv();
+ // I can't remember how JSON parsing worked with PHP, but you'd
+ // parse $string here and be left with an easy-to-use named array..
+ printf($string)
+ }
+.. _php-zmq:
+Ruby accesses EMDR via ZeroMQ's zmq_ Ruby bindings::
+ #
+ # Synchronized subscriber
+ #
+ require 'rubygems'
+ require 'ffi-rzmq'
+ context =
+ subscriber = context.socket(ZMQ::SUB)
+ // Connect to the first publicly available relay.
+ subscriber.connect("tcp://")
+ subscriber.setsockopt(ZMQ::SUBSCRIBE,"")
+ loop do
+ subscriber.recv_string(string = '')
+ # I have no idea how JSON parsing works with Ruby, but you'd parse
+ # this string as JSON, and be off to the races.
+ puts string
+ end
+.. _zmq:
3 doc_src/volunteering.rst
@@ -3,4 +3,5 @@
Volunteering computing resources
-This will contain details on how to volunteer your computing resources.
+This will contain details on how to volunteer your computing resources as we
+see a need for them.
@@ -22,6 +22,7 @@
scripts = [
+ 'bin/emdr-announcer',

0 comments on commit 25498e5

Please sign in to comment.
Something went wrong with that request. Please try again.