Kafka Source stalls event processing becomes non-interactive

[hrabalp]

#30

when max_partition_fetch_bytes is too high or if there is problem in processing (like slow lookup)

than kafka source is uneable to commit in time, stalls and is uneable to recover.
This way any unexpected small network/db glitch can indirectly kill whole processing and forces BSpump restart

[ateska]

Simulate this by time.sleep(20) in a pipeline that uses Kafka source.

Technically, this is a problem of non-interactive pipeline (processing of one event takes much longer than few milliseconds and the event loop is held for too long).
We can try to optimise the Kafka source behavior but the root cause is different - blocking event processing - that should be detected and notified to a user.

[ateska]

It seems that we need similar mechanism to simulate_event() from FileABCSource to be added to a KafkaSource. Obviously, Kafka chunk is processed in one event loop cycle.