Architecture
#Concept of Particles
Apache Storm works with tuples every one of which is a data item to be processed. The tuples coming from Storm Spout follow the materialized topology of user by being processed bolt by bolt in their turn (see Figure 1).
Figure 1.Basic Apache Storm user topology
The Particles (that is how we name our primitives) represent a special type of tuple. The features of Particle are:
- Every Particle has a timestamp;
- Every Particle is strongly typed.
The Particles are matched to Apache Storm tuples automatically, therefore, there is no any break of compatibility with the basic Storm functionality.
We distinguish 2(two) different types of Particles: Data Particle and Meta Particle (see Figure 2). They both are processed by Storm cluster, but in a different way.
Figure 2. Data and Meta Particles
##Data Particles Data Particles are the tuples having data. One can see them as typical Apache Storm data streams with the addition of timestamp to every item. They are processed within Storm cluster, following the topology that user started to deal with these data streams. If the functionality of Grouper is used, the Grouper checks whether the incoming Particle is a Data or Meta Particle. If it happens to be Data Particle, the Grouper just groups the Data Particles in the same way as a basic Storm Grouper does and sends it to the next bolt(s) within the topology.
##Meta Particles Meta Particles do not carry measurements, i.e., data items. They are special Particles that are injected into the Apache Storm data stream(s) in order to trigger a specific behavior to that certain stream(s), for example, to adjust time for the replay functionality for the dataset from the past or to slow down the data processing.
Assuming that there could be more than one bolt instance for every type of bolt, the Meta Particles are always **broadcast **to every bolt instance in a way through the whole topology to guarantee that every bolt is aware of this exact Meta Particle.
###Grouper for MetaParticles
##Sync buffer
- apply ordering -use time window (now const) -duplicate Metaparticles filter if too late: rejected