Back Pressure Handling

Back pressure is a serious issue that needs to be solved by all frameworks and applications working on streams. It describes a situation where a producer generates more output than a consumer is able to process.

There are at least two different ways to cope with this problem:

• scale the processing units and redirect work to additional processors
• ensure that producers are never bothered by slow consumers

The SPQR framework (actually) follows the second approach and ensures that no producer ever slows down or is interrupted by slow consumers.

Again there are two different layers where this approach is implemented:

• inter-pipeline communication via macro pipelines which is implemented by relying on Apache Kafka - the more messages the more disk space is consumed but no producer must wait for slow consumers
• inter-component communication inside micro pipelines which is based on OpenHFT chronicle queues where all messages are written to disk (and consumed from disk) - the more messages the more disk space is consumed but no producer must wait for slow consumers