We have some use cases that demand very fresh data. For example, display events aggregates of the last minute for decision making.
However, as shown in the diagram below, the ingestion latency spikes during segment sealing, which breaches the SLA
The segment size is 5MB, so in this particular case, it seals every a couple of hours. We could try smaller segment sizes, but that may result in different issues like too many zookeeper nodes.
The solution to this could be more optimistic on new segment creation: the new segments start ingestion without waiting for the sealing. However, this can complicate the state machine, as the segment cleanup is more complicated when the segment commit fails.
Any other thoughts to mitigate this issue?
We have some use cases that demand very fresh data. For example, display events aggregates of the last minute for decision making.
However, as shown in the diagram below, the ingestion latency spikes during segment sealing, which breaches the SLA
The segment size is 5MB, so in this particular case, it seals every a couple of hours. We could try smaller segment sizes, but that may result in different issues like too many zookeeper nodes.
The solution to this could be more optimistic on new segment creation: the new segments start ingestion without waiting for the sealing. However, this can complicate the state machine, as the segment cleanup is more complicated when the segment commit fails.
Any other thoughts to mitigate this issue?