Auth callout under connection storms: a blocking wait in the broker, and a documentation gap for auth-callout implementers

[cfedersp]

Two separate things came up while looking at how auth callout behaves under a burst of simultaneous connection attempts — a broker-side implementation detail worth reconsidering, and an auth-callout implementer footgun worth documenting. Neither is a defect.
I'm opening this as a discussion because one part is a concrete proposal I'd like feedback on before it becomes a PR, and the other is guidance I think belongs in the docs, not a behavior change to argue for.

1. The broker blocks a goroutine per pending connection

processClientOrLeafCallout (server/auth_callout.go) waits on the callout reply like this:

go
reply := s.newRespInbox()
respCh := make(chan string, 1)
sub, err := acc.subscribeInternal(reply, processReply)
...
select {
case errStr = <-respCh:
    authorized = errStr == _EMPTY_
case <-time.After(authTimeout):
    ...
}
return authorized, errStr

That's a genuine goroutine park. It ties up one goroutine per in-flight, unauthenticated connection for up to authTimeout. The cost scales with how many connections are simultaneously mid-auth, not with how much real work is happening — which is exactly the shape of a connection storm (a server restart, a network partition healing, a client fleet reconnecting together after a blip).

Proposed Direction:
Each request already gets its own unique inbox subject (newRespInbox()) and its own private subscription (processReply), so a reply is correlated to its pending connection by NATS's own subject routing before any application code runs. Building on that directly:

1. processReply runs the connection-setup continuation inline, the moment the reply arrives, in place of respCh <- errStr.
2. The time.After(authTimeout) timer becomes a context.WithTimeout deadline, whose ctx.Done() path runs that same continuation (reject and finish) if it fires first. Deriving that context from the connection's own lets a disconnect mid-callout cancel cleanly too, rather than leaving anything waiting out the full timeout regardless.
3. A sync.Once (or equivalent) shared between those two paths ensures exactly one of them completes the connection, covering the case where a reply and the deadline land at the same instant.

The result reuses the per-request subscription-and-closure pattern already in place — each pending request still owns its own inbox and closure, just acting immediately on whichever event reaches it first instead of relaying that event to a parked goroutine.

2. The auth-callout side has the same problem, and the docs don't warn about it

This isn't a nats-server defect, and I'm not proposing nats-server enforce anything here — a callout impl is arbitrary user code, always has been. But the default, most obvious way to write one hits the same failure mode as (1): nc.Subscribe's async handler delivers serially, one dedicated goroutine per subscription calling the callback inline, one message at a time. An auth-callout built the straightforward way — a single Subscribe with a synchronous handler doing JWKS verification and signing — queues every request behind whichever one is running, and under a burst, requests near the back of that queue can exceed authTimeout before the auth-callout even looks at them, rejecting clients for reasons that have nothing to do with their credentials. excalibase/excalibase PR #36 is a real, independent instance of exactly this.

Given how common connection storms are in practice, I'd strongly recommend the auth-callout docs and ADR-26 say explicitly that a auth-callout needs async design or bounded concurrency to discourage the naive path.

Why I'm raising both together

This came up while looking at a larger, separate idea: extending NATS to support genuinely asynchronous client-side authenticators (a browser client using WebCrypto with a non-extractable key, where producing a signature is a Promise, not a synchronous call). Auth callout is the one place today where the server already tolerates a connection staying unauthenticated longer than one round trip, so it's the natural reference point for what a client-side equivalent might look like — but composing an async protocol for clients on top of an auth design that's blocking on the broker side, and commonly blocking on the auth-callout side too, seemed like the wrong foundation to build on. I'd rather see both of these addressed — (1) as a real fix, (2) as documentation — before that conversation goes further, though I recognize that's a separate scope from either of these on their own.