supportsStreaming() sends a live prompt to the provider on every call

[jasperblues]

Goal

Asking whether a model supports streaming should not cost a request to that model.

User value

Every supportsStreaming() call currently sends a real completion request to the provider. It is billable, it is on the path immediately before streaming, and it is invisible — it does not look like a model call from the caller's side, so nobody attributes latency or spend to it. A deployment that checks before each stream pays for two requests per streamed turn and sees one.

Current state

StreamingCapabilityVerifier (in SpringAiLlmService.kt) determines support behaviourally, by streaming a real prompt:

private const val TEST_PROMPT_MESSAGE = "Say 'test' to confirm streaming works"
private const val STREAMING_TEST_TIMEOUT_MS = 100L

fun supportsStreaming(chatModel: ChatModel): Boolean {
    return try {
        val testRequest = Prompt(listOf(UserMessage(TEST_PROMPT_MESSAGE)))
        val stream = chatModel.stream(testRequest)
        canConsumeStream(stream)      // stream.hasElements().timeout(100ms).block()
        true
    } catch (e: UnsupportedOperationException) {
        false
    } catch (e: Exception) {
        false
    }
}

SpringAiLlmService delegates straight to it with no memoisation:

override fun supportsStreaming(): Boolean = StreamingCapabilityVerifier.supportsStreaming(chatModel)

Callers reach it per operation, not once per model — StreamingPromptRunner.asStreamingWithValidation(), DelegatingStreamingPromptRunner, StreamingPromptRunnerBuilder, and AbstractLlmOperations.supportsStreaming. PromptRunner's own KDoc tells applications to Check supportsStreaming() before calling streaming(), so the documented usage is the expensive one.

Gaps

• Not cached. The answer is a property of the model and cannot change between calls for a given ChatModel, but it is recomputed every time.
• Costs a real completion. For a provider that does support streaming, the probe is an ordinary billable request. The blocking timeout(100ms) bounds the wait, not the spend — the request has been sent.
• The prompt reaches real providers. "Say 'test' to confirm streaming works" appears in production traffic and in whatever request logging or observability the deployment has.
• catch (e: Exception) -> false conflates "does not support streaming" with "rate limited", "network blip", "bad key". A transient failure silently reports the model as non-streaming, and the caller then takes a non-streaming path for reasons that never surface.

Direction

Cheapest correct fix is to memoise per ChatModel instance — the value cannot change for the life of one — which makes the cost once per model rather than once per call.

Better, if it can be known statically: derive support from the model or provider rather than probing at all. The probe exists because Spring AI's ChatModel has a default stream implementation that throws, so the interface alone does not answer the question — but the provider adapters generally do know, and an unsupported model throws UnsupportedOperationException immediately and cheaply on the first attempt anyway.

Separately worth splitting the two catch clauses so that "provider was unreachable" does not present as "model cannot stream".

Outcome

supportsStreaming() is free to call, does not put a synthetic prompt in front of a real model, and does not report a transient provider failure as a missing capability.

Notes

Pre-existing on main, introduced 2026-04-21 in #1600 ("Deep refactoring of streaming area"). Untouched by #1883, #1888, #1889 and #1891 — found while reviewing how the BYOK embedding probe caches its result, which stamps the observed value onto the returned service and so probes exactly once per service built. Raising rather than copying the pattern.