Kpipe on Raspberry Pi / Edge Computing: Backpressure and Slow Downstream Scenarios

[igfasouza]

Hi everyone,

I've recently been exploring Kpipe and I think it could be a very interesting fit for edge computing workloads running on constrained hardware such as Raspberry Pi.

One of the scenarios I would like to test is a slow downstream dependency:

Load Generator
    ↓
Kafka
    ↓
Kpipe Consumer
    ↓
Slow REST Service (5s delay)
    ↓
Kafka

The idea is to generate events at a much higher rate than the downstream service can process and observe how Kpipe's backpressure mechanisms behave under sustained pressure.

My hypothesis is that this type of setup could be a good demonstration of one of Kpipe's key strengths: allowing lag to accumulate in Kafka while keeping resource usage on the consumer side predictable and stable, especially on small ARM-based devices.

A few questions for the community:

• Has anyone already deployed or tested Kpipe on Raspberry Pi, Orange Pi, or similar ARM64 devices?
• Are there any known limitations or considerations for running Kpipe on low-resource environments?
• Would the project be interested in example applications focused on edge computing use cases?

I'm considering building a small demo application that simulates a slow downstream service and demonstrates how Kpipe handles backpressure while running on constrained hardware.

If that sounds useful, I'd be happy to contribute it as an example or benchmark project.

I'd love to hear your thoughts and feedback.

[eschizoid]

Hi @igfasouza, this is a great fit and I'd genuinely like to see it happen.

A bit of context that should shape how you scope it. kpipe's current benchmark suite tops out at workMicros=1000,
which is 1 ms of simulated downstream work. @chuck-alt-delete from Conduktor recently pointed out that 1 ms is way
too low to be representative of real workloads; most production Kafka consumers do HTTP hops or DB writes that cost
10-100 ms, and llingr's bench already covers that range. What you're describing (a 5s downstream call) sits well past that, in the regime where backpressure actually starts driving the whole behaviour of the consumer. That's the gap I know about and haven't filled.

On ARM64 and Raspberry Pi: honestly, no, we have zero deployment data on aarch64. Java 25 + Loom on aarch64 should
work in principle (OpenJDK's aarch64 build is solid), but I haven't measured it. The two unknowns that would worry me
are JIT warm-up on slower cores and allocation pressure on a 4GB Pi running sustained throughput. The framework arms in
our current bench sit at ~1.7 KB/op on Intel.

At 5s downstream, commit cadence becomes load-bearing. The default KafkaOffsetManager commits safe offsets on every
poll cycle, but you'll want to think about what re-processing means if a 5s record fails mid-call. KEY_ORDERED is also
worth a look if your workload has key affinity (per-key serial, parallel across keys).

Yes to the example. The measurements I'd most want from a Pi run:

1. Per-record latency at p50/p95/p99, not just throughput. kpipe's pinned-VT model should give predictable tail
   latency, and I'd like data rather than theory on that.
2. Heap allocation rate sustained over 30+ minutes via JFR or -XX:+PrintGCDetails. The 1.7 KB/op figure is JMH on
   Intel and aarch64 may differ.
3. SEQUENTIAL vs PARALLEL backpressure at the same workMicros. I'm honestly curious which one wins at 5s, the
   bottlenecks are different.
4. JIT warm-up time on aarch64. First five minutes vs steady state. Loom-on-Pi isn't really documented anywhere I've
   seen, and that's the data point I'd value most.

Practical suggestion for the harness: MockConsumer for the bench arm (measure framework cost in isolation),
Testcontainers for the integration smoke (prove end-to-end). That split has worked well for the existing JMH suite under
benchmarks/.

One optional stretch if you have appetite for it: the harness probably handles 10ms / 35ms / 50ms / 100ms with one
parameter change. Those are the production-realistic values Chuck was pointing at, and seeing the curve from there up to
your 5s edge case would be much more valuable than any single point. Bonus points if you cover them, but I'd rather see
a tight 5s edge story than a half-finished spread.

Happy to look at the design before you go deep. A short README plus a BenchmarkPlan.md with those measurements is
probably enough to align on shape before you commit code.

[igfasouza]

Thanks a lot for the detailed feedback ... this is exactly the kind of direction I was hoping for.

Before implementing the benchmark suite, I'd like to spend some time running Kpipe locally and playing with the different processing modes to get a better feel for the behavior under load and backpressure scenarios.
By local I mean in a Raspberry PI.

My initial plan is:

1. Run Kpipe and experiment with different configurations (PARALLEL, SEQUENTIAL, KEY_ORDERED) to understand the behavior and identify the most interesting scenarios.
2. Add a short README.md describing the edge/backpressure scenario.
3. Add a BenchmarkPlan.md covering:
   • p50/p95/p99 per-record latency
   • sustained heap allocation / GC over 30+ minutes
   • SEQUENTIAL vs PARALLEL
   • first 5 minutes vs steady state on aarch64
   • commit/reprocessing considerations for failed 5s calls
4. Use MockConsumer for the isolated benchmark arm.
5. Use Testcontainers only for the end-to-end smoke test.
6. Keep 10ms / 35ms / 50ms / 100ms as optional stretch cases after the 5s case is solid.

I'll start by exploring the runtime behavior, then put together the README + BenchmarkPlan and share the overall shape before going too deep into code.

Thanks again ...

[eschizoid]

🚢🇮🇹