Design and Architecture (1.x)
In it's current state, the system deals with data in the form of json payloads. Turbine is data agnostic and simply views the json blob as a map of key -> value pairs.
{a:1, b:2, c:'string', d:true}
The only binding Turbine has with the data is the aggregation dimension specified using name and type. Json payloads received from individual instances matching the same aggregation key are combined together. e.g
{type:'weather-data-temp', name:'New York', temp:74}
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:76}
are combined to give
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:75}
Name and type are just data classifiers and either one could suffice as the aggregation key. However, the reason for having a composite aggregation key is to be able to represent multiple sub streams over the same connection. This is what type is used for.
One can send data payloads of multiple types over the same connection for efficiency, hence multiplexing multiple streams over the same persistent Turbine connection. Consider this example.
{type:'weather-data-temp', name:'New York', temp:74}
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:76}
{type:'weather-data-wind-velocity', name:'New York', temp:12}
{type:'weather-data-wind-velocity', name:'Los Angeles', temp:10}
are combined to give
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:75}
{type:'weather-data-wind-velocity', name:'New York', temp:12}
{type:'weather-data-wind-velocity', name:'Los Angeles', temp:10}
The figure below shows the key components of Turbine. Let's do a quick overview of how the major components work with each other and then discuss each of them in detail.
Everything starts at the InstanceDiscovery module which provides all the host information to Turbine. This is then sent periodically (with updates if any) to the ConnectionManager who is responsible for creating connections to the hosts. Once the connection is established, data is streamed down continuously and sent directly to the Aggregator which aggregates the data and forwards the aggregate output to any listeners downstream, such as browsers and other clients.
Read on further if you are interested in the details of the architecture for each of the key components.
This is responsible for giving the aggregator an accurate view of the system being monitored i.e what hosts are currently in rotation and which ones are available for streaming metrics. Turbine uses the notion of a server being UP or DOWN. If a server is DOWN then Turbine does not connect to it even though it is discoverable. One of the key reasons for doing this is that many complex systems at Netflix do go through lengthy bootstrap processes w.r.t seeding caches, priming network connections, running pre-compute algorithms etc, before they advertise themselves as available.
The InstanceDiscovery component is pluggable via a strategy pattern in Turbine. One can provide implementations to vend instance info from a variety or sources. Examples
- Static file where the list of servers does not change
- Configuration properties dictating the server list
- Eureka - currently used by Netflix
- AWS EC2 apis showing you your ASG state
The InstanceObservable runs periodically where it polls the InstanceDiscovery impl for host info and notifies the InstanceObserver of host updates. InstanceObserver is also an interface and one can register their own observer with the InstanceObservable to receive updates and build other functionality. Some examples are
- Host count monitor - track that fleet size should never go below a min threshold.
- Autoscale activity tracker - get data about autoscale events and use this for analysis which can help with tuning Amazon autoscaling policies.
This is the other big component in the system and is responsible for generating the aggregated metrics. The architecture makes heavy re-use of certain components, let's understand them first.
- This is a simple pub-sub style component where multiple publishers or TurbineDataMonitors can send data to multiple listeners or TurbineDataHandlers.
- Note that each listener can register with the dispatcher and indicate the exact set of hosts/pulishers it is interested in receiving data from.
- Each listener is provided with it's own queue of data and hence this isolates publishers from slow listeners. If a listener falls behind, then it's queue will fill up and the dispatcher just drops new data that is intended for it untill more space becomes available.
- Since listeners have their own queue they are also isolated from each other.
- Isolation is not the only benefit here, we also get de-coupling by using a Dispatcher. TurbineDataMonitors do not need to be aware of who is interested in their data and how many of them. Since you can register multiple TurbineDataHandlers for a TurbineDataMonitor, this can help you build multiple aggregators, or even get the individual host stream if needed.
Now that you get what the Dispatcher does, let's have a look at the detail for an Aggregator.
All the plumbing for getting data from multiple hosts to the same place has been encapsulated into a single component called ClusterMonitor, and it's individual sub-components can be re-used to build another different type of ClusterMonitor, if needed. The aggregator is one extension of the ClusterMonitor. It primarily consists of
- TurbineDataHandler - this data handler is registered to the TurbineDataDispatcher described above, and hence this data handler receives all updates from individual TurbineDataMonitors which are just publishers of the data.
- TurbineDataDispatcher - every ClusterMonitor/Aggregator has it's own TurbineDataDispatcher and hence other TurbineDataHandler interested in getting access to the aggregate output can just subscribe to this dispatcher and receive data. Note that this is just another instance of the exact same component described above. It is being re-used here.
