Merge branch 'master' of github.com:gtaylor/EVE-Market-Data-Relay

README.rst

@@ -4,14 +4,18 @@ EVE Market Data Relay
 :Author: Greg Taylor
 :License: BSD
 
-This project is a proof-of-concept for a super-scalable, affordable way to
+This project is a super-scalable, affordable way to
 accept a large amount of user-submitted market data (via uploaders), and
 re-broadcast said data in realtime to a number of subscribers.
 
-The end result is that those writing market-data driven applications could
+The end result is that those writing market-data driven applications can
 simply subscribe to a "firehose" of market data, and get going, without having
 to hassle with uploaders or data submission APIs.
 
+Additionally, the consumers may accept very large amounts of data without the
+overhead associated with a ton of HTTP connections. EMDR's ZeroMQ underpinnings
+are hugely more efficient.
+
 Documentation
 -------------
 

doc_src/global.txt

@@ -1,8 +1,11 @@
 .. _GitHub project: https://github.com/gtaylor/EVE-Market-Data-Relay
 .. _issue tracker: https://github.com/gtaylor/EVE-Market-Data-Relay/issues
 .. _mailing list: https://groups.google.com/forum/#!forum/eve-emdr
+.. _EMDR map: http://map.eve-emdr.com/
+.. _@gctaylor Twitter: https://twitter.com/#!/gctaylor
 
 .. _Unified Uploader Interchange Format: http://dev.eve-central.com/unifieduploader/start
+.. _Clients supporting UUIF: http://dev.eve-central.com/unifieduploader/implementations
 
 .. _Python: http://python.org
 .. _nose: http://somethingaboutorange.com/mrl/projects/nose/

doc_src/index.rst

@@ -5,33 +5,46 @@
 EVE Market Data Relay (EMDR)
 ============================
 
-EVE Market Data Relay is  a super-scalable, super-stable way to accept a large
-amount of user-submitted market data (via uploaders), and re-broadcast said
-data in near real-time to a large number of subscribers.
+EVE Market Data Relay (EMDR) is a super scalable, highly available firehose of
+real-time market data. For those that wish to record price and history data
+as it comes in, EMDR will help you do so as efficiently and reliably as
+possible. EMDR's data feed is open to the public, and is developed as an
+open source project.
 
-The end result is that those writing market-data driven applications can
-simply subscribe to a "firehose" of market data, without having
-to hassle with writing uploaders, data submission APIs, or scraping data from
-other market sites.
+EMDR may appeal to you if:
 
-For a more complete run-down, see :doc:`overview`.
+* You need real-time access to market data, as soon as possible. Perhaps for
+  sending out price/inventory level alerts, notification of lucrative
+  trade routes, or real-time charts and graphs.
+* You want to record prices over time.
+* You want the absolutely most complete set of data that you can get.
+* The effort and overhead of getting large amounts of direct player uploads to
+  your site is too much to bear.
 
-Learning more
--------------
+EMDR's primary goals are:
 
-To learn more about EMDR, see the :doc:`overview`.
+* Ensuring that all market sites have access to player-uploaded market data.
+* Extremely high reliability.
+* Minimize expense to those running EMDR (shared burden).
+* Minimize expense to those consuming the feed (bandwidth).
+
+For a more complete run-down, see :doc:`overview`.
 
 **License:** EVE Market Data Relay is licensed under the `BSD License`_.
 
-These links may also be useful to you.
+Assorted Info
+-------------
 
-* Source repository: https://github.com/gtaylor/EVE-Market-Data-Relay
-* Issue tracker: https://github.com/gtaylor/EVE-Market-Data-Relay/issues
-* IRC Room: irc.coldfront.net #emdr
-* Mailing list: https://groups.google.com/forum/#!forum/eve-emdr
-* Thread on EVE Gate: https://forums.eveonline.com/default.aspx?g=posts&t=95454
-* Real-time data monitoring map: http://map.eve-emdr.com/
-* @gctaylor on Twitter: https://twitter.com/#!/gctaylor
+* `Mailing list`_ - If you are consuming the feed, make sure
+  to subscribe to this for important announcements. This is also one of the
+  best places to ask questions or discuss EMDR stuff.
+* IRC Room - irc.coldfront.net #emdr, an excellent place for getting quick
+  help, or hanging out with other developers and consumers.
+* `Issue tracker`_ - Report bugs here.
+* `Source code <GitHub project>`_ - For those wanting to hack on EMDR.
+* `EMDR map`_ - See the solar systems light up as
+  market data arrives.
+* `@gctaylor Twitter`_ - Tweets from the maintainer.
 
 General Topics
 --------------

doc_src/overview.rst

@@ -23,18 +23,14 @@ A developer would probably need to:
   a ways down the road to burnout.
 
 None of these tasks are fun, they all involve re-inventing the wheel. By the
-time the developer gets through these, burnout may be a real possibility.
+time the developer gets through all of this (if they do), burnout is a distinct
+possibility. All before getting to the fun stuff!
 
-**For current market-driven sites**, the need to patch together scripts to
-pull data from the other various market sites is tedious, boring, and will
-often resource-intensive for both parties (downloader and the API being
-downloaded from).
-
-**In both cases (current, or prospective developers/sites)**, EVE Market
-Data Relay (EMDR) allows you to forgo all of this drudgery, and instead,
-connect to a firehose of data in the standardized
-`Unified Uploader Interchange Format`_ format. Additionally, EMDR is distributed
-and highly available, making the likelihood of failure very slim.
+EVE Market Data Relay (EMDR) allows you to forgo all of this drudgery, and
+instead, connect to a firehose of data in the standardized
+`Unified Uploader Interchange Format`_ format. EMDR's ZeroMQ_ underpinnings also
+make it easier, and exponentially more efficient than accepting HTTP
+uploads directly.
 
 Core Principles
 ---------------
@@ -64,25 +60,27 @@ For any given submitted market order, here is the flow said order goes through::
     (Gateway) -> (Announcer) -> (Relays) -> (Applications)
 
 First, the order hits the **Gateway**, which is a simple HTTP application
-that parses the message. Incoming messages can be in a number of different
-formats (EVE Marketeer/EVE Marketdata, Unified Uploader Format, etc).
+that parses the message. Incoming messages are in
+`Unified Uploader Interchange Format`_.
 
-The Gateway interprets the message, then converts it to
-`Unified Uploader Interchange Format`_. The converted message is then handed
-off to all of the **Anouncers** in the network.
+The Gateway interprets the message, validates it, normalizes anything weird,
+then pipes it to all of the root-level **Announcers** in the network.
 
 The **Announcer** is the first tier of our market data distribution.
-As market messages arrive, they are sent out to all **Relays** that are
+Announcers relay any data they receive to **Relays** that are
 connected to the Announcer. There are only a few Announcers, and these only
-accept connections from approved Relays.
+accept connections from approved Relays. Most relays connect to multiple
+announcers for added redundancy.
+
+The **Relay**, like the Announcer, is a dumb repeater of everything it
+receives. Relays receive data from their Announcers, then pipe it out to any
+subscribers that are connected to them. Subscribers can be other **Relays**,
+or actual user sites/applications.
 
-The **Relay** is a dumb repeater daemon. It takes the processed orders and just
-spews them out to any subscribers. Subscribers can be other **Relay** daemons,
-or actual user sites/applications. By hanging Relays below Announcers, and
-having applications connect to the Relays, we keep bandwidth usage and costs
+By using our system of Relays, we keep bandwidth usage and costs
 lower on the top-level Announcers. We are also able to keep "fanning out" to
-allow more and more consumers to get the data without breaking the bank, or
-putting massive load on a single server.
+improve redundancy and serve greater numbers of consumers without large
+increases in bandwidth utilization.
 
 We are left with a very efficient, very sturdy data relay network. The next
 section goes into detail about fault-tolerance.
@@ -92,25 +90,25 @@ High Availability through shared burden
 
 EMDR is architected in a way that allows every single component to be
 replicated. We can easily add additional daemons at each level of the stack in
-order to increase throughput or availability.
+order to improve availability, or to spread costs.
 
-Uploads are dispersed via Round-Robin DNS, which is a
-simple way to distribute load across multiple machines. For each Gateway
-in the DNS rotation, incoming bandwidth consumption drops for the whole pool
-as the load is divided. If at any time one of the gateways becomes unreachable,
-it is automatically removed from the DNS rotation.
+HTTP Uploads are dispersed to Gateways via Round-Robin DNS, which is a
+simple way to distribute the traffic across multiple machines. For each additional
+Gateway added to DNS rotation, incoming bandwidth consumption drops for the
+whole pool as the load is divided. If at any time one of the gateways becomes
+unreachable, it is automatically removed from the DNS rotation.
 
-In the diagram below, we see what will be our initial deployment. Site 1 is
-comprised of EMDR running on Greg Taylor's (the project maintainer) machines,
-and Site 2 will be running on a trusted party's machines that are in another
-data center/region.
+In the diagram below, we see a rough representation of our current deployment.
+Site 1 is comprised of EMDR running on Greg Taylor's (the project maintainer)
+machines, and Site 2 is a separate copy running in another data center. The
+relays are all ran by different volunteers.
 
 .. note:: We are not limited to just two instances of EMDR, there is no hard
     limit. Additionally, we'll mostly scale by adding more Gateways, since
     additional Announcers are only for redundancy.
 
 At every step of the entire flow, we can afford to lose one of the two
-daemons, with no service interruption. The infrastructure can be scaled well
+daemons without a service interruption. The infrastructure can be scaled well
 out past the initial two sites, if need be.
 
 .. image:: images/emdr-daemon-diagram.png

doc_src/uploading.rst

@@ -16,35 +16,20 @@ Market data is uploaded to EMDR by default.
 
 .. _EVEMon: http://evemon.battleclinic.com/
 
-With the EVE Marketeer Client
------------------------------
+With other clients
+------------------
 
-While we prefer EVEMon, you can use the EVE Marketeer Client as well:
+While we prefer EVEMon, you can use any client that supports the
+`Unified Uploader Interchange Format`_. An up to date list is maintained
+here: `Clients supporting UUIF`_.
 
-* Download/install the `EVE Marketeer Uploader`_.
-* Run the application.
-* Go to the Endpoints tab.
-* Hit 'Add'.
-* Enter 'EMDR' for the name.
-* Leave data type as EVE Marketeer & Marketdata.
+Steps vary from client to client, but here is the typical process:
+
+* Open the dialog that lets you specify where to send market data.
+* Create a new endpoint. Select Unified format if it asks.
 * Set the URL to: http://upload.eve-emdr.com/upload/
-* Enter EMDR for upload key.
-* Hit Save.
-* You're all set. Get uploading!
+* Enter your upload key, if you feel like it. Otherwise, just make something
+  up or leave it blank.
+* Hit save, and start uploading.
 
 You can then use any market service's auto-uploader pages.
-
-.. _EVE Marketeer Uploader: http://www.evemarketeer.com/uploader
-
-Other Clients
--------------
-
-Any client that supports either Unified Uploader Interchange format, or
-EVE Marketeer/EVE Market Data format will also work just fine. The following
-clients fall under this designation:
-
-* Contribtastic for Mac
-
-Simply point your client at: http://upload.eve-emdr.com/upload/
-
-The message format will be auto-detected and parsed accordingly.

doc_src/volunteering.rst

@@ -3,13 +3,30 @@
 Volunteering computing resources
 ================================
 
-This will contain details on how to volunteer your computing resources as we
-see a need for them. We are going to need plenty of regional relays across
-a number of the major hosting providers. In particular, we're interested in:
+EMDR is ran by volunteers who foot the cost in order for everyone to get
+access to market data. While running pieces of EMDR is not something that will
+get you fame or notoriety, it is crucial to the continued survival and
+success of the network.
 
-* Amazon Web Services
-* Rackspace
-* Linode
+If you have idle computing resources, or would like to share capacity on
+a machine, we'd love to have it. ``gtaylor`` is available to assist with setup
+and configuration.
 
-We will, of course, love to have anyone, but these three are likely to have
-a good chunk of our consumers on them, which saves bandwidth for all of us.
+Relays
+------
+
+Relays are our current need. These are at the lowest rung of the ladder, and
+are where consumers connect to get their data. Relay access policy is completely
+up to the volunteer. Relays can be 100% open to the public, or completely
+restricted however you'd like.
+
+Relay characteristics:
+
+* Traffic depends entirely upon how many consumers are connected. However,
+  most don't exceed 500 kb/s.
+* Processor usage isn't huge. We've had great success running on Linodes and
+  other virtual machines without issues.
+* RAM usage is about 8-16 MB.
+
+If you'd like to volunteer a relay, please get in touch with gtaylor via email:
+``gtaylor (at) gc-taylor (dot) com``