Merge pull request #76 from earthgecko/luminosity

Update docs

docs/getting-data-into-skyline.rst

@@ -2,11 +2,8 @@
 Getting data into Skyline
 =========================
 
-You currently have two options to get data into Skyline, via the Horizon
-service:
-
-A note on time snyc
-===================
+Firstly a note on time snyc
+===========================
 
 Although it may seems obvious, it is important to note that any metrics
 coming into Graphite and Skyline should come from synchronised sources.
@@ -17,6 +14,31 @@ Time drift does decrease the accuracy and effectiveness of some
 algorithms. In terms of machine related metrics, normal production grade
 time snychronisation will suffice.
 
+Secondly a note on the reliability of metric data
+=================================================
+
+There are many ways to get data to Graphite and Skyline, however some are better
+than others.  The first and most important point is that your metric pipeline
+should be transported via TCP, from source, to Graphite, to Skyline.  Although
+the original Skyline set up in the days of statsd UDP only and where UDP
+transport was seen as acceptable (and it possibly is in a LAN environment).
+Data analysis with metrics shipped via any UDP in a distributed and/or cloud
+environments is not as effective in terms of analysis as TCP shipped metrics.
+
+For Skyline to do the full spectrum of analysis both in the real time and
+historic data contexts, it needs **reliable** data, with as few missing data
+points as possible.
+
+Although collectd is great it ships via UDP, which is not great.  So ensure that
+your metric pipeline is fully TCP transported.  statsd now has a TCP listener,
+sensu is good for server metrics and there are lots of options.
+
+Now getting the data in
+=======================
+
+You currently have two options to get data into Skyline, via the Horizon
+service:
+
 TCP pickles
 ===========
 
@@ -77,6 +99,8 @@ code for the exact protocol), you'll be able to stream to this listener.
 UDP messagepack
 ===============
 
+Genreally do not use this.  It is UDP, but has not been removed.
+
 Horizon also accepts metrics in the form of messagepack encoded strings
 over UDP, on port 2025. The format is
 ``[<metric name>, [<timestamp>, <value>]]``. Simply encode your metrics

docs/skyline-and-friends.md

@@ -28,24 +28,44 @@ over time, in terms of:
   - Skyline's own running times, load, algorithm performance, etc
   - being able monitoring the overall performance of your "things" over time
 
-- To the new user these things may seem like uninteresting, probably never to be
-  looked much metrics, however over time they will describe your ups and downs,
+- To the new user these things may seem uninteresting, probably never to be
+  looked metrics, however over time they will describe your ups and downs,
   your highs and lows and hopefully add to your understanding of your "things"
 
 ## Statsd
 
 [Statsd](https://github.com/etsy/stats) feeds graphite so it is quite handy.
+Use TCP, not UDP
 
 ## BuckyServer
 
 Before tcp transport was added to Statsd was
 [BuckyServer](https://github.com/HubSpot/BuckyServer) for long haul TCP
-transport of your metrics to local Statsd -> Graphite.
+transport of your metrics to Statsd -> Graphite.
+
+### Some advantages to using Buckyserver
+
+Although the introduction of another node app may seem like too much effort or
+additional complexity, using Buckyserver also has additional advantages.
+Although Statd does allow for Multi-Metric Packets the total length of the
+payload must be within the MTU of all your hops, a very hard variable to know
+and quite a hassle to calculate dynamic in scripts/apps.
+
+Buckysever has the advantage of being able to ship lots of metrics with a single
+POST, which is useful for shipping custom or app metrics and has no size limits.
+
+Using Statsd and Buckyserver also allows one to describe 2 methods of shipping
+data in scripts/apps, via Buckyserver and via Statsd.  This allows you to ship
+via the Buckyserver method and if that fails failover to the Statsd method,
+where each metric can be iterated through in your script/app and sent
+individually, if you do not want the hassle of dealing with Multi-Metric Packet
+payload size.  Running Buckyserver and Statsd on the same instance allows for
+these types of advanced usage and has proved to be very reliable.
 
 ## Sensu
 
-[Sensu](https://sensuapp.org/) can feed Graphite -
-[Sensu on github](https://github.com/sensu/sensu)
+[Sensu](https://sensuapp.org/) can feed Graphite like collectd but uses TCP not
+UDP for transport - [Sensu on github](https://github.com/sensu/sensu)
 
 ## Riemann
 
@@ -59,5 +79,6 @@ transport of your metrics to local Statsd -> Graphite.
 
 ## Many more
 
-There are a great deal of apps that can feed Skyline, this just just to mention
-a few.
+There are a great deal of apps that can feed Skyline, this is just mentioning
+a few.  However bear in mind, that whatever you use, try and ensure that it is
+TCP and not UDP based.