v0.36.0 — token introspection, egress accounting, CORP

Token introspection

A new off-by-default endpoint, POST {prefix}/__introspect_token__, resolves an opaque bearer credential to a principal — for a reverse proxy that terminates the only public listener and must know which principal a credential authenticates as before it can authorize anything.

POST /__introspect_token__
Authorization: Bearer <introspector credential>
{"token": "<opaque subject credential>"}

200  {"principal": "alice@example.com", "token_name": "laptop", "ttl_seconds": 300}

The response is an identity assertion made by the thing being protected, and the asker acts on it using credentials the worker does not hold. It has to be trusted more than the worker, which is why the guards are requirements rather than options:

Guard
Route absent unless enabled	No worker grows a credential→identity oracle by upgrading a dependency
Introspector allowlist, no permissive default	Authentication and introspection are different capabilities. Where any valid credential may introspect, any user can test guesses of any other user's credential and resolve a stolen one to its owner
JWS-shaped subjects refused before the resolver	Routing one through hands a third party a token the asker may itself have rejected — an expired access token is still live at its issuer
Byte-identical rejections	Unknown, expired and malformed are one answer; anything else confirms that a guessed credential exists
Credential digested, never logged	Asserted against captured log output, not review

The response never carries claims. It is a closed set of three keys, asserted as a set so a future addition has to come through a test. A pass-through claims field would let a worker choose its caller's tenant routing, row scope and policy branch.

ttl_seconds exists so the caller caches; the endpoint holds no cache. Treat it as an authorization window.

Outage is not rejection

AuthUnavailableError is new, and deliberately not a ValueError — chain_authenticate advances on ValueError, so an outage raised as one reads as "not my credential, try the next" and emerges as a 401 from the end of the chain. Raised as this type it propagates and surfaces as 503 + Retry-After.

That distinction is what a caller's negative cache depends on. Without it a sidecar restart 401s every caller at once, and clients that treat a second 401 after a refresh as fatal turn a thirty-second blip into a fleet-wide re-login storm.

Egress accounting

Access-log records gain request_bytes, response_bytes and externalized_bytes.

The existing input_bytes / output_bytes measure logical Arrow buffers — what the worker processed. They miss compression in one direction and externalisation in the other, which makes them the wrong number for anything that costs money:

output_bytes  (logical Arrow): 200008
response_bytes (on the wire):     183

A factor of ~1,000 on one compressible result. In the other direction, a call that externalises 10 GB leaves a pointer batch of a few hundred bytes in the HTTP body, with the rest previously invisible.

response_bytes could not be measured where records were written — handlers finish before response compression runs. Emission now defers through a per-request sink drained by middleware once the final body exists. The cost is that a crash between handler and response loses that request's records; the alternative was a permanently wrong number.

Trace correlation and sampling

trace_id / span_id (W3C hex) join a record to the span it ran under. request_id only ever correlated within one service.

--access-log-sample RATE keeps a fraction of successful calls. Three properties, all specified and tested:

• Errors are never sampled. A rate below 1 exists because successes repeat, which is exactly what failures don't.
• The decision is per call, not per record — keyed on stream_id then request_id, so a stream's continuations share its init's fate. Random per-record sampling shreds multi-record calls into fragments that read as data loss.
• sample_rate rides on every kept record. A rate discoverable only from a deployment's flags gets guessed wrong.

--access-log-async moves writes to a listener thread. Bounded queue, never blocks; full means drop, and the next record through carries dropped_records.

⚠️ Claim redaction is on by default

The one behavioural change in this release. Claim values whose names match credential patterns (*_token, *_key, password) or standard OIDC PII (email, phone_number, given_name, …) are replaced with "[redacted]".

Keys survive — which claims a token carried is a question an audit log exists to answer; what they contained is not. An access log outlives its token by months and ships to systems chosen for searchability rather than for holding personal data.

from vgi_rpc.logging_utils import set_claim_redactor, no_redaction
set_claim_redactor(no_redaction)   # restores the previous behaviour

Matching is key-based, like vgi_rpc.sentry's existing kwarg redactor — free text holding an address is not caught, and cannot be without guessing.

CORS: Cross-Origin-Resource-Policy

A CORS-enabled server now sends Cross-Origin-Resource-Policy: cross-origin. Correct CORS is not sufficient for a caller that has opted into cross-origin isolation: a page sending Cross-Origin-Embedder-Policy: require-corp has its own fetches blocked without CORP, and the server sees an ordinary successful response — so it fails invisibly from the operator's side. Narrow with cors_resource_policy="same-site", or None to omit.

Conformance also closes two blind spots the Go and Rust ports found:

• Headers a plain worker never advertises. The derived exposure check reads OPTIONS /health, so conditional capability headers (upload URLs, size caps) were never seen. The CORS fixture now points at a storage-enabled worker.
• Headers that only ride failures. /health is a success-path surface, so nothing advertises X-VGI-RPC-Error, VGI-Auth-Reason or X-Request-ID — a derived check cannot reach them. Now named explicitly and asserted one at a time.

For porters

Two new optional conformance fixtures; omit either and its group skips.

Fixture	Group
conformance_http_introspect_port	TestTokenIntrospection (TestTokenIntrospectionOffMode runs ungated)
conformance_http_cors_port	TestCors — point it at a storage-enabled worker

The CORP assertion does not skip when the header is absent. Reaching it means the port supplied the CORS fixture, i.e. declared browser support, so a missing CORP is a real gap — expect TestCors to fail on first run against this suite until the one-line change lands.

Both docs/WIRE_PROTOCOL.md (§16) and docs/porting-guide.md mark the definitive-vs-transient distinction normative.

---

Upgrading: no action required unless you read claim values out of access logs — see the redaction section. Access-log consumers should note that payload omission now reports truncated: "payload_omitted" rather than true, which frees true to mean genuine size-driven loss again; a normally-configured server previously set true on nearly every unary record, leaving nothing to filter on. No wire-protocol changes to the RPC surface; protocol_hash is unchanged.