Architecture
#Concept of Particles
Apache Storm works with tuples every one of which is a data item to be processed. The tuples coming from Storm Spout follow the materialized topology of user by being processed bolt by bolt in their turn (see Figure 1).
Figure 1.Basic Apache Storm user topology
The Particles (that is how we name our primitives) represent a special type of tuple. The features of Particle are:
- Every Particle has a timestamp;
- Every Particle is strongly typed.
The Particles are matched to Apache Storm tuples automatically, therefore, there is no any break of compatibility with the basic Storm functionality.
We distinguish 2(two) different types of Particles: Data Particle and Meta Particle (see Figure 2). They both are processed by Storm cluster, but in a different way.
Figure 2. Data and Meta Particles
##Data Particles Data Particles are the tuples having data. One can see them as typical Apache Storm data streams with the addition of timestamp to every item. They are processed within Storm cluster, following the topology that user started to deal with these data streams. If the functionality of Grouper is used, the Grouper checks whether the incoming Particle is a Data or Meta Particle. If it happens to be Data Particle, the Grouper just groups the Data Particles in the same way as a basic Storm Grouper does and sends it to the next bolt(s) within the topology.
##Meta Particles Metaparticles do not carry measurements, i.e., data items. They are special Particles that are injected into the Apache Storm data stream(s) in order to trigger a specific behavior to that certain stream(s), for example, to adjust time for the replay functionality for the dataset from the past or to slow down the data processing.
###Grouper for Metaparticles Assuming that there could be more than one bolt instance for every type of bolt, the Metaparticles are always broadcast to every bolt instance throughout the whole topology to guarantee that every bolt instance is aware of this exact Metaparticle (see Figure 3).
Figure 3. Parallelism of Particles processing: Metaparticles are always broadcast to every type of bolt and to every instance of bolt.
Every spout which has received the Metaparticle, broadcasts it to every known bolt instance. Every bolt that receives the Metaparticle, propagates it by broadcast to all bolt instances at the next step. The process repeats itself until the Metaparticle reaches the end of topology (Fig.3).
##Synchronization buffer In a case of receiving many data streams from multiple spouts/bolts at the same time by one bolt instance, some issues may arise that should be dealt with before the streams follow the bolt processing logic. The issues are:
- Data from one spout/bolt could be delivered faster than from other spout/bolt, however, regards, to the receiving bolt processing logic, the merged data stream should be ordered in time. Therefore, the need of ordering regards time synchronization between multiple data streams can arise before the merged stream comes to bolt.
- One bolt instance can receive the multiple copies of the same Metaparticle. It could influence the behavior of the system, especially in case of changing the global state of system (like, shutdown tuple executed two or more times) which would lead to system state inconsistency. To avoid it, every Metaparticle should be delivered Once and only once to every bolt instance.
To deal with these issues the new system component is introduced: synchronization buffer.
It is put before the certain grouper bolt, and it is responsible:
- for applying time ordering for multiple data streams, i.e., provide time synchronization functionality; -for filter out the duplicates of the same Metaparticle, so, that only one Metaparticle is arrived to bolt instance.
-use time window (now const)
if too late: rejected