Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/docker-publish-x402.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ on:
push:
branches:
- main
- feat/secure-enclave-inference
tags:
- 'v*'
paths:
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/lint-test.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
name: Lint and Test Charts

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

Expand Down
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,11 @@ Obol Stack: framework for AI agents to run decentralised infrastructure locally.
## Conventions

- **Commits**: Conventional commits — `feat:`, `fix:`, `docs:`, `test:`, `chore:`, `security:` with optional scope
- **Branches**: `feat/`, `fix/`, `research/`, `codex/` prefixes
- **GitHub branch policy**: never push `codex/`-prefixed branches to GitHub from this repository; use `feat/`, `fix/`, `research/`, or another non-codex branch name before pushing
- **Branches**: `feat/`, `fix/`, `research/`, `docs/`, `chore/` prefixes
- **GitHub branch policy**: never push `codex/`-prefixed branches to GitHub from this repository; rename to `feat/`, `fix/`, `research/`, `docs/`, `chore/`, or another non-codex branch name before pushing
- **Detailed architecture reference**: `@.claude/skills/obol-stack-dev/SKILL.md` (invoke with `/obol-stack-dev`)
- **Review scope**: Avoid broad, vague review/delegation boundaries. State the exact files, invariants, and expected evidence before reviewing or spawning agents. Prefer concrete checks such as "controller cannot access signer/Secrets", "agent write RBAC is namespace-scoped", and "flow uses real obol CLI path" over generic "review architecture".
- **Planning / report docs**: Do not commit plan, roadmap, install-report, or PR-review writeups to the repo (`plans/*.md`, `docs/plans/*.md`, `docs/pr-review-*.md`, `docs/*-testing-log.md`, `docs/*-test-plan.md`, etc.). PR bodies, GitHub issues/discussions, and issue comments are the right home for ephemeral planning artifacts. Only durable, user-facing documentation belongs in `docs/`.

## Build, Test, Run

Expand Down
132 changes: 132 additions & 0 deletions cmd/obol/sell.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ func sellCommand(cfg *config.Config) *cli.Command {
sellStatusCommand(cfg),
sellTestCommand(cfg),
sellStopCommand(cfg),
sellUpdateCommand(cfg),
sellDeleteCommand(cfg),
sellPricingCommand(cfg),
sellRegisterCommand(cfg),
Expand Down Expand Up @@ -1334,6 +1335,137 @@ func sellStopCommand(cfg *config.Config) *cli.Command {
}
}

// ---------------------------------------------------------------------------
// sell update
// ---------------------------------------------------------------------------

func sellUpdateCommand(cfg *config.Config) *cli.Command {
return &cli.Command{
Name: "update",
Usage: "Update pricing or wallet on an existing ServiceOffer in place",
ArgsUsage: "<name>",
Description: `Patches a live ServiceOffer without deleting it. Only the fields you pass
are changed; everything else is preserved. The serviceoffer-controller will
reconcile the new payment config automatically.

Switching price models (e.g. per-request → per-mtok) nulls the previous keys
so the controller picks up the new model.

Examples:
obol sell update my-api -n llm --per-request 0.002
obol sell update my-api -n llm --per-mtok 5.0
obol sell update my-api -n llm --wallet 0xNew... --chain base`,
Flags: []cli.Flag{
&cli.StringFlag{
Name: "namespace",
Aliases: []string{"n"},
Usage: "Namespace of the ServiceOffer",
Required: true,
},
&cli.StringFlag{
Name: "wallet",
Aliases: []string{"w"},
Usage: "New USDC recipient wallet address",
},
&cli.StringFlag{
Name: "chain",
Usage: "New payment chain (base, base-sepolia, ethereum)",
},
&cli.StringFlag{
Name: "price",
Usage: "New per-request price in USDC (alias for --per-request)",
},
&cli.StringFlag{
Name: "per-request",
Usage: "New per-request price in USDC",
},
&cli.StringFlag{
Name: "per-mtok",
Usage: "New per-million-tokens price in USDC",
},
&cli.StringFlag{
Name: "per-hour",
Usage: "New per-compute-hour price in USDC",
},
},
Action: func(ctx context.Context, cmd *cli.Command) error {
u := getUI(cmd)
if cmd.NArg() == 0 {
return errors.New("name required: obol sell update <name> -n <ns> [--per-request N | --per-mtok N | --per-hour N] [--wallet 0x...] [--chain base]")
}

name := cmd.Args().First()
if err := validate.Name(name); err != nil {
return err
}
ns := cmd.String("namespace")

if _, err := kubectlOutput(cfg, "get", "serviceoffers.obol.org", name, "-n", ns, "-o", "name"); err != nil {
return fmt.Errorf("ServiceOffer %s/%s not found: %w", ns, name, err)
}

payment := map[string]any{}

if wallet := strings.TrimSpace(cmd.String("wallet")); wallet != "" {
if err := x402verifier.ValidateWallet(wallet); err != nil {
return err
}
payment["payTo"] = wallet
}

if chain := strings.TrimSpace(cmd.String("chain")); chain != "" {
payment["network"] = chain
}

priceSet := cmd.String("price") != "" || cmd.String("per-request") != "" || cmd.String("per-mtok") != "" || cmd.String("per-hour") != ""
if priceSet {
priceTable, err := resolvePriceTable(cmd, true)
if err != nil {
return err
}

price := map[string]any{
"perRequest": nil,
"perMTok": nil,
"perHour": nil,
}
switch {
case priceTable.PerRequest != "":
price["perRequest"] = priceTable.PerRequest
case priceTable.PerMTok != "":
price["perMTok"] = priceTable.PerMTok
case priceTable.PerHour != "":
price["perHour"] = priceTable.PerHour
}
payment["price"] = price
}

if len(payment) == 0 {
return errors.New("nothing to update: pass at least one of --per-request / --per-mtok / --per-hour / --wallet / --chain")
}

patch := map[string]any{
"spec": map[string]any{
"payment": payment,
},
}
patchBytes, err := json.Marshal(patch)
if err != nil {
return fmt.Errorf("marshal patch: %w", err)
}

if err := kubectlRun(cfg, "patch", "serviceoffers.obol.org", name, "-n", ns, "--type=merge", "-p", string(patchBytes)); err != nil {
return fmt.Errorf("failed to patch serviceoffer: %w", err)
}

u.Successf("ServiceOffer %s/%s updated", ns, name)
u.Info("The controller will reconcile the new payment config.")
u.Infof("Check status: obol sell status %s -n %s", name, ns)
return nil
},
}
}

// ---------------------------------------------------------------------------
// sell delete
// ---------------------------------------------------------------------------
Expand Down
61 changes: 28 additions & 33 deletions docs/guides/monetize-inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,27 @@ This guide walks you through exposing a local LLM as a paid API endpoint using t
> [!NOTE]
> `--per-mtok` is supported for inference pricing, but phase 1 still charges an
> approximate flat request price derived as `perMTok / 1000` using a fixed
> `1000 tok/request` assumption. Exact token metering is deferred to the
> follow-up `x402-meter` design described in
> [`docs/plans/per-token-metering.md`](../plans/per-token-metering.md).
> `1000 tok/request` assumption. Exact token metering is not implemented yet.

> [!IMPORTANT]
> The monetize subsystem is alpha software on the `feat/secure-enclave-inference` branch.
> The monetize subsystem is alpha software.
> If you encounter an issue, please open a
> [GitHub issue](https://github.com/ObolNetwork/obol-stack/issues).

> [!IMPORTANT]
> The current implementation is event-driven. `ServiceOffer` is the source of truth, `serviceoffer-controller` owns reconciliation, `RegistrationRequest` isolates registration side effects, and `x402-verifier` derives live routes directly from published ServiceOffers.
> Older references below to the obol-agent reconcile loop, heartbeat polling, or direct `x402-pricing` route mutation are historical.
> `ServiceOffer` is the source of truth. `serviceoffer-controller` owns
> reconciliation, `RegistrationRequest` isolates registration side effects, and
> `x402-verifier` derives live routes directly from published ServiceOffers.

## System Overview

```
SELLER (obol stack cluster)

obol sell http --> ServiceOffer CR --> Agent reconciles:
obol sell http --> ServiceOffer CR --> serviceoffer-controller reconciles:
1. ModelReady (pull model in Ollama)
2. UpstreamHealthy (health-check Ollama)
3. PaymentGateReady (create x402 Middleware + pricing route)
3. PaymentGateReady (create x402 Middleware)
4. RoutePublished (create HTTPRoute -> Traefik gateway)
5. Registered (ERC-8004 on-chain, optional)
6. Ready (all conditions True)
Expand Down Expand Up @@ -195,7 +194,7 @@ Ready [check] All required conditions True
Watch the progress:

```bash
# Check conditions (wait ~60s for agent heartbeat)
# Check conditions
obol sell status my-qwen --namespace llm

# Verify Kubernetes resources
Expand Down Expand Up @@ -538,7 +537,7 @@ obol sell status

### Pausing

Stop serving an offer without deleting it. This removes the pricing route so requests pass through without payment:
Pause an offer without deleting it:

```bash
obol sell stop my-qwen --namespace llm
Expand All @@ -560,7 +559,6 @@ Deletion:

- Removes the ServiceOffer CR
- Cascades Middleware and HTTPRoute via OwnerReferences
- Removes the pricing route from the x402 verifier
- Deactivates the ERC-8004 registration (sets `active=false`)

Verify cleanup:
Expand Down Expand Up @@ -611,7 +609,7 @@ Traefik Gateway
+--------+---------+
|
+----------v-----------+
| PaymentGateReady | (create Middleware + pricing route)
| PaymentGateReady | (create Middleware)
+----------+-----------+
|
+---------v----------+
Expand All @@ -629,20 +627,20 @@ Traefik Gateway

### Kubernetes Resources per ServiceOffer

When the agent reconciles a ServiceOffer named `my-qwen` in namespace `llm`:
When `serviceoffer-controller` reconciles a ServiceOffer named `my-qwen` in namespace `llm`:

| Resource | Kind | Namespace | Name |
|----------|------|-----------|------|
| ServiceOffer | `obol.org/v1alpha1` | `llm` | `my-qwen` |
| Middleware | `traefik.io/v1alpha1` | `llm` | `x402-my-qwen` |
| HTTPRoute | `gateway.networking.k8s.io/v1` | `llm` | `so-my-qwen` |
| ConfigMap patch | `v1` | `x402` | `x402-pricing` (route added) |

The Middleware and HTTPRoute have `ownerReferences` pointing at the ServiceOffer, so they are garbage-collected on deletion.

### Pricing Configuration

The x402 verifier reads its config from the `x402-pricing` ConfigMap:
The x402 verifier reads cluster-wide payment defaults from the
`x402-pricing` ConfigMap:

```yaml
wallet: "0x70997970C51812dc3A010C7d01b50e0d17dc79C8"
Expand All @@ -657,39 +655,35 @@ routes:
network: "base-sepolia"
```

This configuration is used by the `litellm-config` ConfigMap in the `llm` namespace, which LiteLLM reads for model_list configuration.

Per-route `payTo` and `network` override the global values, enabling multiple ServiceOffers with different wallets or chains.
Published offer routes are derived from `ServiceOffer` resources rather than
being maintained manually in this ConfigMap. Per-offer `payTo` and `network`
can still override the cluster defaults.

---

## Troubleshooting

### Agent not reconciling
### Offer not reconciling

The agent reconciles on a heartbeat (~60 seconds). Check agent logs:
Check ServiceOffer conditions and controller logs:

```bash
obol kubectl logs -n openclaw-* -l app=openclaw --tail=50
obol sell status my-qwen --namespace llm
obol kubectl logs -n x402 -l app=serviceoffer-controller --tail=50
```

### x402 verifier returning 200 instead of 402

The pricing route may not have been added, or was overwritten. Check the ConfigMap:

```bash
obol kubectl get cm x402-pricing -n x402 -o jsonpath='{.data.pricing\.yaml}'
```

Ensure a route matching your path exists in the `routes` list. The verifier logs its route count at startup:
The ServiceOffer may not be `Ready`, or the request path may not match the
published offer. Check the offer and the resources it owns:

```bash
obol sell status my-qwen --namespace llm
obol kubectl get middleware x402-my-qwen -n llm
obol kubectl get httproute so-my-qwen -n llm
obol kubectl logs -n x402 -l app=x402-verifier --tail=10
# Look for: "routes: 1" (or however many you expect)
```

If routes are missing, the agent may not have reconciled yet (heartbeat is ~60s). You can also re-trigger reconciliation by deleting and re-creating the ServiceOffer.

### Facilitator unreachable from cluster

If using a self-hosted facilitator on the host, verify the k3d bridge:
Expand Down Expand Up @@ -787,7 +781,7 @@ Replace `openclaw-obol-agent` with your actual OpenClaw namespace if different.
| `obol sell http <name> --wallet ... --chain ... --per-request ... --upstream ... --port ...` | Create a ServiceOffer and register by default |
| `obol sell list` | List all ServiceOffers |
| `obol sell status <name> -n <ns>` | Show conditions for an offer |
| `obol sell stop <name> -n <ns>` | Pause an offer (remove pricing route) |
| `obol sell stop <name> -n <ns>` | Pause an offer without deleting it |
| `obol sell delete <name> -n <ns>` | Delete an offer and cleanup |
| `obol sell status` | Show cluster pricing and registration |
| `obol sell register --private-key-file ...` | Advanced/manual registration or repair path |
Expand All @@ -796,9 +790,10 @@ Replace `openclaw-obol-agent` with your actual OpenClaw namespace if different.

| Resource | Namespace | Purpose |
|----------|-----------|---------|
| `x402-pricing` ConfigMap | `x402` | Pricing routes and wallet config |
| `x402-pricing` ConfigMap | `x402` | Cluster-wide wallet, chain, and facilitator settings |
| `x402-secrets` Secret | `x402` | Wallet address |
| `x402-verifier` Deployment | `x402` | Shared seller-owned x402 gateway and legacy `/verify` endpoint |
| `serviceoffer-controller` Deployment | `x402` | Reconciles ServiceOffers into published resources |
| `serviceoffers.obol.org` CRD | (cluster) | ServiceOffer custom resource definition |
| `traefik-gateway` Gateway | `traefik` | Main ingress gateway |

Expand Down
Loading