Micro Pipeline

To build a new micro pipeline the framework provides three different types of elements:

• data source consumers which received events and inject them into the pipeline and
• data stream operators which process a data stream element wise
• data stream emitters that export a stream of data to an arbitrary sink

Data Source Consumer

With respect to its name, an element of this type consumes data from a source and publishes it into the pipeline. It serves as an intermediary between an arbitrary source, eg. Twitter, RDBMS tables, or click stream providers (eg. webtrends), and the configured operator elements.

Every pipeline specification must start with a source configuration to receive any data. A pipeline must provide at least one source per specification. Learn more about sources and how to build them.

Operator

As data source consumers are responsible for reading data from external sources, processor elements provide features to filter or modify transferred data, eg. filter by content or aggregate. Every pipeline is allowed to have any number of operators, it may even have none at all.

Learn more about operators and how to build them.

Emitter

The emitter component allows a pipeline to export its data to any sink type. Therefore it subscribes itself to either a source or an operator. As data may be intended for more than one sink a pipeline is allowed to have any number of emitters.

Lean more about emitters and how to build them.