perf: reduce e2e produce + fetch latency on AWS to < 100ms

[klaudworks]

I think combined with the prior s3 parallelization we can reduce e2e produce + consume p90 tail latency to sub 100ms for 4MB segments, measured from the time the broker receives a message to the consumer starting to read the first byte. I don't see any architectural issues. When utilizing s3 express buckets one could probably reduce p90 to less than 30ms for small segments.

Missing changes:

1. etcd watch that updates local map instead of polling every 10ms for offset key changes
2. in-memory broadcast from producer to consumer to wake it up on offset changes

@novatechflow I know that your main concern is not e2e latency but these are quite unintrusive changes which extend the use cases even further (and probably allow for better marketing as a drop in replacement for all but realtime use cases). Would you be fine with these changes? I'd be happy to benchmark this.

[klaudworks]

First results

Baseline (532d01a before any of my PRs) vs Optimized (includes all my changes so far including the unmerged above mentioned etcd watch mechanism).

The tests just sequentally produced individual messages and fetched them again. Measurement started before producing the message and ended after reading and decoding. Tested in AWS eu-west-1 with an m6g.large machine and a standard s3 bucket in the same region.

1KB payload (1000 individual messages)

Metric	Baseline	Optimized	Improvement
P50	44.22 ms	21.69 ms	-51%
P95	55.37 ms	34.57 ms	-38%
Mean	45.60 ms	23.15 ms	-49%

4KB payload (1000 individual messages)

Metric	Baseline	Optimized	Improvement
P50	44.25 ms	22.14 ms	-50%
P95	56.31 ms	34.41 ms	-39%
Mean	45.68 ms	23.56 ms	-48%

64KB payload (1000 individual messages)

Metric	Baseline	Optimized	Improvement
P50	44.46 ms	24.76 ms	-44%
P95	75.44 ms	49.81 ms	-34%
Mean	50.14 ms	28.88 ms	-42%

When the producer batches messages the improvements are way lower because s3 segment + index parallelization matters less (index is small and fast to upload in comparison), also shaving off 7-12ms by caching offsets in memory and proactively notifying consumers has less relative impact.

4MB batch (1000 x 4KB records), 1000 batches

Metric	Baseline	Optimized	Improvement
P50	132.15 ms	108.79 ms	-18%
P95	165.76 ms	145.48 ms	-12%
Mean	130.51 ms	110.39 ms	-15%

I compared these numbers against raw 4MB uploads via the AWS SDK and they come pretty close. So there is not much room for optimization left on the broker side.

4MB AWS Put Uploads, 1000 Batches

Metric	Latency
P50	85.21 ms
P95	119.35 ms
Mean	86.31 ms

Using s3 express buckets reduces e2e latency by another 40-50% compared to the above optimized version:

4MB batch (1000 x 4KB records), 1000 batches

Metric	Standard S3	S3 Express	Improvement
P50	108.79 ms	62.18 ms	-43%
P95	145.48 ms	67.50 ms	-54%
Mean	110.39 ms	62.28 ms	-44%

[klaudworks]

Given that we probably have to tie brokers to specific partitions in #120, a watch-notify mechanism via etcd for cross-broker offset updates has little value anymore. The whole problem doesn't exist in the first place.

I'll leave this issue open until the discussion around #120 resolved.

[novatechflow]

@klaudworks - nice! Let's do it!