feat(x402): chain label, last-settlement gauge, PurchaseAutoRefill sync + verifier PodMonitor + obol-frontend RBAC for purchaserequests

[bussyjd]

Summary

Powers the new My Listings → EarningsStrip and My Purchases → expansion drawer surfaces in obol-stack-front-end#parity/results-bar by closing three data-pipeline gaps:

1. chain label is now on every obol_x402_buyer_* and obol_x402_verifier_* series — required for per-chain aggregation in the EarningsStrip / WalletStrip.
2. obol_x402_verifier_last_payment_success_seconds gauge stamps the most recent settled payment per route — powers the "Last settlement · 2m ago" cell.
3. PurchaseAutoRefill Go-type sync with the CRD spec — adds MaxTotal + MaxSpendPerDay fields that already existed in purchaserequest-crd.yaml but were unread by the controller / frontend.

Plus the x402-verifier PodMonitor the namespace was missing (only litellm-x402-buyer was scraped), and obol-frontend RBAC grows read access to purchaserequests so the new /api/marketplace/purchases route can list them.

What changed

#	File	Why
1	internal/x402/buyer/metrics.go	Add chain to all 9 counter/gauge label sets
2	internal/x402/buyer/proxy.go	Thread cfg.Network through prometheusLabels() at all 3 callsites
3	internal/x402/metrics.go	Add lastPaymentSuccess GaugeVec + register it
4	internal/x402/verifier.go	chain label in prometheusLabels(rule); SetToCurrentTime() on success in both ForwardAuth + proxy-mode paths
5	internal/monetizeapi/types.go	PurchaseAutoRefill.MaxTotal int + MaxSpendPerDay string
6	internal/x402/verifier_test.go	Update label maps for chain; new TestVerifier_LastPaymentSuccessGauge (3 subtests)
7	internal/x402/buyer/proxy_test.go	Update 4 existing TestProxy_* label-map assertions for the new chain label
8	internal/x402/buyer/metrics_test.go (new)	TestPrometheusLabels_ChainPropagation (3) + TestMetrics_ChainLabelScrapeRoundtrip (2)
9	internal/monetizeapi/types_test.go (new)	TestPurchaseAutoRefill_JSONRoundTrip (5) + TestPurchaseAutoRefill_UnmarshalAcceptsCRDForm
10	internal/embed/infrastructure/helmfile.yaml	Add x402-verifier PodMonitor (release: monitoring) + grant obol-frontend ClusterRole get/list/watch on purchaserequests

Live validation (base-sepolia)

Funded the default obol-agent wallet (0xBb0a70F713334401063c9A8519014F6F026E1c5e) with 0.005 ETH and 3 USDC from the smoke-test seller key, then drove two paid requests through buy.py pay against demo-hello:

Settlement tx 1   0x52140ce18c64cb96170dc54bccbbdfcbe8e0426a161318c8adf687639ee3ac31
Settlement tx 2   (second buy.py pay)

obol_x402_verifier_charged_requests_total{
  chain="eip155:84532",
  offer_namespace="demo", offer_name="demo-hello",
  route="/services/demo-hello/*"
}  =  2

obol_x402_verifier_last_payment_success_seconds{same labels}
  =  1779554031.04   (matches the on-chain block timestamp)

increase(charged_requests_total[7d]){chain="eip155:84532"}  ≈  1.08
   (Prometheus extrapolates from 2 samples; close enough for the
    EarningsStrip's per-chain × price display)

The verifier-side new image was rebuilt locally (localhost:54103/x402-verifier:dev, sha256:af11911f…), imported into the k3d cluster, and the PodMonitor confirmed up{job="x402/x402-verifier"} = 1 for both replicas before traffic flowed.

CRD-cleanliness notes

• No CRD schema change. Pause/resume continues to ride the existing metadata.annotations["obol.org/paused"] mechanism (already honored by the controller at internal/serviceoffercontroller/controller.go:458–463). The frontend's new PATCH route writes the annotation; the controller does the rest.
• PurchaseAutoRefill schema in purchaserequest-crd.yaml already declared maxTotal (integer) and maxSpendPerDay (string) — the Go types just hadn't been updated. This commit closes the drift, which TestPurchaseAutoRefill_UnmarshalAcceptsCRDForm now guards against re-occurring.
• Earnings / 24h-traffic / last-settlement intentionally stay in Prometheus, not on the ServiceOffer / PurchaseRequest status. The recommendation from the up-front mapping work was that CR status should not be a metrics store; the new obol_x402_verifier_last_payment_success_seconds gauge is the cleanest place for the one timestamp the design canvas asks for.

Test plan

• go test ./internal/x402/... ./internal/monetizeapi/... — green (14 new subtests, 4 pre-existing buyer-proxy assertions updated, no skips)
• go vet ./internal/x402/... ./internal/monetizeapi/... — clean
• go build ./... — clean
• Live: paid request through buy.py pay against demo-hello on base-sepolia — verifier metrics scraped, frontend /api/sell/list returns non-zero chargedByChain + lastSettlementUnix
• Release-smoke (flow-08/14/15) — out of scope for this PR; the buyer-proxy test update means flow-08 should now pass label assertions without manual edits

Related

Pairs with frontend PR https://github.com/ObolNetwork/obol-stack-front-end/pull/new/parity/results-bar which consumes these labels via three new PrometheusClient methods (chargedSalesByOfferAndChain, chargedRequests24hByOffer, lastSettlementByOffer).

[bussyjd]

Phase 1+2 hardening pushed (commit 27e1ac5)

Picked up findings from a 4-agent K8s architecture review (observability, RBAC, helmfile, frontend boundary). What landed in this commit:

RBAC trims (defense-in-depth — the obol-stack frontend is local-only behind the hostnames: ["obol.stack"] HTTPRoute restriction):

• Drop secrets get/list from obol-frontend-openclaw-discovery ClusterRole. Pre-existing dangling grant; no code reads them.
• Drop /status subresource from purchaserequests rule. Frontend never writes status; only the controller does.

Monitoring + RBAC co-location (kills 3 bedag/raw helmfile releases):

• x402-verifier: PodMonitor → ServiceMonitor in base/templates/x402.yaml. Verifier has a stable Service on http:8080; scrapes the endpoint cleanly across replicas.
• litellm-x402-buyer: PodMonitor moved into base/templates/llm.yaml. Stays PodMonitor (sidecar port not in a Service).
• obol-frontend-rbac ClusterRole + binding moved into new base/templates/obol-frontend-rbac.yaml.

Label cardinality:

• Drop route label from verifier metrics. (offer_namespace, offer_name, chain) already uniquely scopes a paid route; route (= rule.Pattern) was redundant and unbounded by path fragments.

PrometheusRule (new base/templates/x402-prometheus-rules.yaml):

• Recording: x402:revenue:{24h,7d}_by_offer_chain, x402:revenue:lifetime_by_offer, x402:settlement_rate:1h_by_offer_chain. Frontend reads these so renaming raw metrics no longer breaks the UI; increase() 2-sample minimum no longer leaves cold offers at "0" for first 30s.
• Alerting: X402PaymentFailureRateHigh (>10% over 1h), X402NoSettlementsAfterChallenge (402s issued, no settlements).

Live-validated: 2 paid requests against demo-hello survive RBAC trims + the ServiceMonitor swap. x402:revenue:7d_by_offer_chain returns 1.0076 for chain=eip155:84532 (matches the underlying counter). /api/marketplace/purchases still 200 after the /status grant trim.

Explicitly deferred (out of scope, documented inline):

• Frontend-egress NetworkPolicy: tried and reverted. On k3s + Flannel the apiserver Service Endpoints point at the host process, outside the cluster pod/service CIDRs. A clean allowlist can't target the apiserver portably without an install-specific ipBlock. Tracked for the day obol-stack ships a non-k3s deployment surface.
• obol-marketplace-api Go aggregator: overkill for the local single-operator context.
• Three-deployment-paths consolidation: larger refactor, separate workstream.

Pairs with the frontend hardening commit on https://github.com/ObolNetwork/obol-stack-front-end/pull/330

[bussyjd]

Follow-up: GC for stale verifier metric series (commit 0fbb99a)

Closes the last deferred item from the hardening pass.

Problem: deleting an offer left its per-label series in the verifier registry forever — most notably obol_x402_verifier_last_payment_success_seconds, which would keep advertising the deleted offer's last-success timestamp and falsely satisfy "recent activity" dashboards/alerts.

Fix: Verifier.load() now diffs the incoming route set against the live (offer_namespace, offer_name, chain) tuples in the registry and DeletePartialMatches each vec for every triple no longer served. Both reload paths (file-config watcher and the ServiceOffer informer via ConfigAccumulator.SetRoutes) funnel through load(), so one hook covers everything — no separate OnDelete handler needed in the informer.

Test: TestVerifier_Reload_PrunesDeletedOfferSeries stamps two offers' worth of metrics with paid requests, reloads with one removed, asserts the removed offer is gone from all six vecs (requests_total, payment_required_total, payment_verified_total, payment_failed_total, charged_requests_total, last_payment_success_seconds) while the kept offer survives.

Also: fixed a guard test (TestX402Manifest_UsesServiceOfferControllerModel) that was still asserting "no ServiceMonitor in this manifest" after the prior commit intentionally relocated one in. Flipped to assert presence so a future cleanup can't silently drop it.

[TateLyman]

One accounting nit before merge: routes that inherit the global chain still emit chain="".

In this PR, matchPaidRouteFull correctly resolves chainName := cfg.Chain when rule.Network == "", but the metric label still comes from prometheusLabels(rule), which returns "chain": rule.Network at internal/x402/verifier.go:464-473. The reload keep-set uses the same raw value at internal/x402/verifier.go:72-78, and the new prune test currently asserts chain: "" for default-chain routes (internal/x402/verifier_test.go:953-954).

That means the new per-chain recordings can produce a blank-chain bucket for the normal/default route case instead of base-sepolia / eip155:84532 / whichever canonical label the frontend expects. It also makes dashboards mix "route inherited default chain" with "route genuinely has no chain", which weakens the EarningsStrip/WalletStrip story in the PR description.

Suggested shape: thread the resolved chainName (or chain.CAIP2Network, if the UI wants CAIP-2) into the labels and the prune keep-key, then flip the regression test from chain: "" to the resolved default chain. The buyer-side proxy already passes cfg.Network into its label helper, so this is mostly a verifier-side inherited-chain case.

[bussyjd]

Bundle PR #536 includes this PR's commits along with the full marketplace architecture-review roadmap (#515-#535, minus #526 obsoleted by #535, minus #514 already merged). Leaving #513 open per request — close manually if the bundle merge satisfies it.

[bussyjd]

Closing as superseded by #536. The head of feat/x402-marketplace-metrics is already included in feat/marketplace-bundle, and #536 is now the single review/merge target for the marketplace bundle.