Scaling out streaming topN

At scale, it is not always possible to stream all data through a single Flume agent and flows will typically fan-out to multiple agents. In this case, with the data flowing through multiple agents in parallel, it is difficult to analyse the global dataset and therefore the analysis must also be parallelised.

The following describes how to scale-out the streaming topN example:

Source

• At the first source (i.e. TwitterSource) the flow should be multiplexed (routed to multiple channels) by some distinguishing factor.
• For example, by the country code from which a status update was sent (i.e. GBR, FRA, DEU, ESP).
• With similar events now grouped together in channels, corresponding sinks can send the events to multiple downstream agents.
• For example, the events routed to the GBR channel will be sent to one agent and the events routed to the FRA channel will be sent to another agent, and so on.
• The downstream agents can now process the flows in parallel.

After the first hop

• Now the flow is multiplexed, the agents after the first hop can individually apply HashtagRollingCountInterceptor and HashtagTopNInterceptor to compute a distributed topN.
• If multiplexing by country code, this will give you the topN by country.
• Each of the agents will also run PeriodicEmissionSource to periodically emit their topN events.
• To compute the global topN, the separate flows must be consolidated at a single agent.
• To do this, the distributed PeriodicEmissionSource's should put their topN events in a single channel and a single sink should send them to a common downstream agent (another hop).

After the second hop

• The single, final agent can then merge the individual topN events to compute the global topN.