Pipelines

Introduction

The SPQR framework uses the notion of pipelines to describe stream processing units. Depending on the layer they are deployed at they are differentiated between

• micro pipelines and
• macro pipelines

A micro pipeline is built up from a number of components that are used to form a process which consumes data from any source, operates on the data and writes to any sink. The macro pipeline on the other hand is built from micro pipelines simply by establishing connections between them forming a larger distributed pipeline.

The driving force behind this separation is to use intermediate results in different processes (aka micro pipelines) and to reduce monitoring complexity.