Quick architecture question on iceoryx2 for token streaming

[julienzarka]

Hi team,

I'm building an on-device LLM kiosk, and iceoryx2 is already in our stack as the zero-copy SHM transport between micro services. Before I commit to a design, I wanted to run a streaming use case past you and get your read.

Context. On a Cortex-A78 class, one service runs Gemma 3 270M locally and emits tokens at roughly 50 to 100 tokens per second, about 256 tokens per answer. Two peer subscribers consume the stream in parallel: a TTS component that turns tokens into
speech, and a UI front-end that renders the live transcript. I do not want our rule engine to see every token. It only watches lifecycle events (stream started, stream complete).

What I am leaning towards. A per-session iceoryx2 service, named something like ai.llm.stream.{session_id}, created when a rule activates and torn down when the stream completes. The intuition is that per-session services give us natural cleanup, no cross-session token leakage, and per-session ACL via a subscriber token issued at activation time.

Where I am not sure. Three alternatives I am also considering:

1. A single global service ai.llm.stream with session_id in the payload, subscribers filter. Simpler, no dynamic services, but fan-out width includes every subscriber even when they don't care about that session.
2. RPC server-streaming straight from my service. Standard pattern, but we lose the zero-copy benefit that pulled us to iceoryx2 in the first place.
3. A pool of K pre-allocated session services, round-robin assigned. Bounds resource cost if dynamic creation turns out to be expensive.

The questions I'd love your view on:

1. Does iceoryx2 support cheap dynamic service creation and teardown at session boundaries, where a session lives 5 to 30 seconds? Or is service creation amortized enough that we should keep a warm pool?
2. What is the realistic resource cost per service (SHM segment, port count, file descriptors), assuming maybe 10 concurrent sessions worst case?
3. Is per-service ACL feasible, or is access control effectively process-level, meaning we'd need to layer auth somewhere else?
4. Backpressure semantics when a subscriber is slow: drop oldest, drop newest, block producer? Any recommended pattern for a streaming case like this?
5. Typical discovery latency from "publisher creates service" to "subscriber attaches"? We have a budget of around 100 ms for first token, so anything in the tens of ms is fine, anything past 200 ms starts to hurt.

If the answer to question 1 is "expensive", I would probably fall back to option 1 (single global service plus payload session_id) and accept the wider fan-out. Curious whether you'd steer us the same way, or if there's a better idea I haven't thought of.

No rush, but if you have 15 minutes for a written reply or a quick call this week or next, that would save us from a wrong turn before I commit code.

Thanks,
Julien

[elBoberido]

Hi @julienzarka,

we are currently quite busy to get iceoryx2 v0.9.0 out. We are definitely interested in a quick call but it will most probably be early next week.