Streaming responses and cache write-behind — when does the cache entry become available?

[emgeisler2]

Following up on the streaming + cache interaction. I confirmed that LLMix does cache streaming responses after the stream completes. But I have a timing question.

In this scenario:

1. Request A starts streaming (cache miss)
2. Request B arrives with the same key while A is still streaming
3. Request A completes, cache entry written

Does request B:

• (a) Wait for A to complete and then get the cached result (singleflight-like)?
• (b) Start its own stream in parallel (no dedup for streaming)?
• (c) Get a cache miss and start a new stream, but both results end up identical in cache?

If it's (b) or (c), we're paying for duplicate tokens during the streaming window. For long responses (4K+ tokens), that's a significant cost multiplier under concurrent load.

async def handle_request(messages):
    # 10 concurrent users asking the same question
    async for chunk in llmix.dispatch_stream(config, messages):
        yield chunk
    # Does only 1 upstream call happen, or 10?

Our P95 streaming response takes 8 seconds. During that window, we often get 5-10 identical requests. Understanding this interaction is critical for our cost model.