fix(x402): inject agent upstream auth

[bussyjd]

Summary

Fixes the obol sell agent paid path where the x402 verifier successfully verifies payment, forwards the request to the seller's per-agent Hermes upstream, and then receives 401 Invalid API key because Hermes requires API_SERVER_KEY but the verifier had no way to inject it.

The fix keeps the current security boundary: Hermes remains bearer-token protected, buyers never see the key, and the verifier injects Authorization: Bearer <API_SERVER_KEY> only after x402 verification has accepted the request.

This PR also pins the embedded verifier deployment to a branch-built multi-arch image containing this source change:

ghcr.io/obolnetwork/x402-verifier:46e63fd@sha256:a8cd7946884c9a702b5cfcfad28d1f5eac1037899303eb4e0157e3ffab7a572c

Architecture

Before this change, agent offers produced a valid paid route, but the route source only knew how to read LiteLLM upstream credentials:

sequenceDiagram
    participant Buyer
    participant Verifier as x402-verifier
    participant SO as ServiceOffer route source
    participant Hermes as per-agent Hermes

    SO->>SO: Watch ServiceOffers
    SO->>SO: Watch litellm-secrets only
    SO->>Verifier: RouteRule{UpstreamURL: Hermes, UpstreamAuth: empty}
    Buyer->>Verifier: POST /services/agent/... + X-PAYMENT
    Verifier->>Verifier: Verify x402 payment
    Verifier->>Hermes: Forward request without Authorization
    Hermes-->>Verifier: 401 Invalid API key
    Verifier-->>Buyer: 401, no settlement

Loading

After this change, the verifier watches the per-agent API secret and uses the existing RouteRule.UpstreamAuth injection mechanism:

sequenceDiagram
    participant Buyer
    participant Verifier as x402-verifier
    participant SO as ServiceOffer route source
    participant Secret as hermes-api-server Secret
    participant Hermes as per-agent Hermes

    SO->>Secret: Watch hermes-api-server in agent namespaces
    SO->>SO: Decode API_SERVER_KEY
    SO->>Verifier: RouteRule{UpstreamURL: Hermes, UpstreamAuth: Bearer key}
    Buyer->>Verifier: POST /services/agent/... + X-PAYMENT
    Verifier->>Verifier: Verify x402 payment
    Verifier->>Hermes: Forward request with Authorization
    Hermes-->>Verifier: 200 response
    Verifier->>Verifier: Settle after upstream success
    Verifier-->>Buyer: 200 + X-PAYMENT-RESPONSE

Loading

Changes

• Split the verifier secret watch into scoped informers for litellm-secrets and hermes-api-server.
• Preserve LiteLLM auth behavior unchanged.
• For type=agent ServiceOffers, resolve UpstreamAuth from spec.agent.ref.namespace so cross-namespace references use the agent namespace, not the offer namespace.
• Extend the x402-verifier ClusterRole to read/watch only the additional hermes-api-server Secret name.
• Pin the embedded verifier image to the branch image that includes this fix.
• Add regression coverage for normal and cross-namespace agent auth routing, RBAC, and the production image pin.

Validation

go test ./internal/x402 -run 'TestRoutesFromStore|TestRouteRuleFromOffer_Agent' -count=1
go test ./internal/embed -run 'TestX402VerifierRBAC_CanReadAgentAPISecrets|TestX402VerifierImage_CarriesAgentAuthFix|TestEmbeddedImages_NamedImagesAreDigestPinned' -count=1

Image publication:

gh workflow run docker-publish-x402.yml --ref fix/agent-seller-upstream-auth
docker buildx imagetools inspect ghcr.io/obolnetwork/x402-verifier:46e63fd --format '{{ .Manifest.Digest }}'
# sha256:a8cd7946884c9a702b5cfcfad28d1f5eac1037899303eb4e0157e3ffab7a572c

Remaining smoke gate

Full flow smoke is running from a combined branch with forced local dev images, but it is currently blocked on the local sudo prompt. The LLM-gated dual-stack flows are also blocked because silvermesh.v1337.lan:8081 is reachable on the LAN but refusing TCP connections.

[OisinKyne]

I hope those service offers really can't leak those secrets.... maybe lets talk about it on sync

[bussyjd]

@OisinKyne — chased your concern and posting the answer here so we have it on the record.

Can a ServiceOffer leak the API_SERVER_KEY value to a buyer? No. For type=agent offers, UpstreamURL comes from offer.Status.AgentResolution.Endpoint, which the controller deterministically derives from the in-cluster Hermes Service DNS (internal/serviceoffercontroller/agent.go:206). The Agent CRD has a status subresource and hermes-agent-factory-write does not grant agents/status, so an attacker can't redirect the bearer at an external host by editing agent.Status.Endpoint. The verifier proxies via httputil.ReverseProxy and doesn't echo Authorization back to the buyer. So the literal worry — secret value leaking out — is fine.

But there's a related confused-deputy issue worth fixing now. resolveAgentOffer doesn't enforce that spec.agent.ref.namespace == metadata.namespace, and the verifier route source keys the bearer lookup directly off offer.Spec.Agent.Ref.Namespace (serviceoffer_source.go:114). openclaw-monetize-write is a ClusterRoleBinding, so the Hermes obol-agent SA can create a ServiceOffer in any namespace pointing `agent.ref` at any other namespace's Agent. The route gets published with the victim's `API_SERVER_KEY` as the upstream bearer, an attacker-chosen `path`, and an attacker-chosen `payment.payTo`. Any buyer who pays gets x402-gated, fully-authenticated proxy access to the victim Hermes `/api/*` (chat, skills, sub-agent management, wallet ops) as the victim.

Not exploitable today (single Hermes mother → no second tenant to confuse the deputy against; mother already has cluster-wide `agents`/`secrets` write over its children). It becomes exploitable as soon as we split sub-agent SAs out of the mother or land multi-mother marketplace mode.

Proposed fix — single-line invariant in `resolveAgentOffer`, fail-closed:

```go
// internal/serviceoffercontroller/agent_resolver.go
func (c *Controller) resolveAgentOffer(ctx context.Context, offer *monetizeapi.ServiceOffer, status *monetizeapi.ServiceOfferStatus) (ok bool, err error) {
ref := offer.Spec.Agent.Ref
if ref.Name == "" || ref.Namespace == "" {
return false, fmt.Errorf("type=agent offer %s/%s missing spec.agent.ref", offer.Namespace, offer.Name)
}
if ref.Namespace != offer.Namespace {
// Confused-deputy guard: the verifier injects API_SERVER_KEY from
// ref.Namespace into the upstream request. Allowing a cross-namespace
// ref would let any principal with serviceoffers write in namespace A
// expose Hermes /api in namespace B as an x402-gated route under
// attacker-controlled path + payTo.
return false, fmt.Errorf(
"type=agent offer %s/%s: spec.agent.ref.namespace %q must equal offer namespace",
offer.Namespace, offer.Name, ref.Namespace,
)
}
// ...
}
```

Plus one regression test in `agent_resolver_test.go` asserting the cross-namespace ref returns an error and leaves `status.AgentResolution` nil, and one in `serviceoffer_source_test.go` asserting an offer with mismatched ref produces zero routes. Pushing this as a follow-up PR — calling it out here first so we can agree on the invariant on the sync.

[bussyjd]

Aggregated into #566 (release/v0.10.0-rc8 → main). The discussion thread above — in particular @OisinKyne's approval review and the follow-up analysis on the API_SERVER_KEY leak question — is the canonical review record for the upstream-auth fix and the confused-deputy guard in #564.