Skip to content

Release v0.16.0-rc2

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 14 Oct 12:33
· 7 commits to main since this release
v0.16.0-rc2
This tag was signed with the committer’s verified signature.
OisinKyne Oisín Kyne
SSH Key Fingerprint: kFIvk1BVuGQBCOUSc8koJUxrf04M4KO7iSYIrwK8fzo
Verified
Learn about vigilant mode.
51ffc20
This commit was signed with the committer’s verified signature.
OisinKyne Oisín Kyne
SSH Key Fingerprint: kFIvk1BVuGQBCOUSc8koJUxrf04M4KO7iSYIrwK8fzo
Verified
Learn about vigilant mode.

Obol banner

v0.16.0-rc2 - Lighter agents, watched nodes

The third release candidate for v0.16. rc2 fixes Grafana not starting on fresh installs and tidies obol model and obol network status output; see New since rc1 below. rc1 added safer paid offers, Grafana and LLM performance metrics on top of rc0. OpenClaw is gone, Hermes carries fewer built-in skills, and a new obol doctor checks the whole stack and prints the fix for anything wrong. Local Ethereum nodes are easier to keep an eye on: obol network status reports execution and consensus sync, and Hermes can now read workloads, logs and events across the cluster, so it can watch your nodes for you.

This is a pre-release for testing. Please report anything that breaks.

Warning

This software is early alpha, you could lose what you put in. Please use caution when it comes to non-testnet assets. Back up before upgrading: obol stack export --file backup.tar.gz. If you still run an OpenClaw agent, back up its wallet with v0.15 first (see Breaking changes).

Note

Testnets have reached Gloas. Sepolia forked on 2026-10-06 and Hoodi forks on 2026-10-26 at 17:42 UTC. The stable releases of Lighthouse (the default) and Grandine don't support Gloas yet, so on Sepolia and Hoodi install with --consensus-client teku, lodestar, nimbus or prysm. Every execution client in this release is Gloas-ready.

Install / Upgrade

# Fresh install
curl -fsSL https://stack.obol.org/v0.16.0-rc2 | bash
obol stack init && obol stack up

# Upgrade an existing stack
curl -fsSL https://stack.obol.org/v0.16.0-rc2 | bash
obol upgrade
obol network sync --all   # apply the new client versions, metrics and RBAC to existing nodes

# Check everything
obol doctor

New since rc1

  • Grafana starts again. On rc1, monitoring-grafana stayed in Init:0/1 forever. The Grafana chart bundled with kube-prometheus-stack 91.9.0 switched its datasource init container from load-once to watch mode by default, so it never exited. It's set back to load once, and a test checks it. Upgrading from rc1 fixes the stuck pod.
  • Quieter model commands. obol model setup, remove, sync and prefer no longer print the full "Hermes installed successfully!" banner after every change. They print one line, so "Primary model is now …" is easy to see. First installs and obol agent sync still show the banner.
  • Plainer obol network status:
    • A node API that isn't answering (for example while its pod restarts) now reads not answering (pod starting or restarting?) instead of a raw error with a temporary port-forward address.
    • An OOM-killed pod is reported once, and the warning says "out of memory".
    • The eRPC row reads pod Running · ready instead of raw kubectl columns.
    • The consensus line says "synced" or "syncing, N slots behind" instead of syncing=/optimistic=/el_offline= flags.
    • Client versions are shortened.
    • --verbose shows the full detail, and -o json output is unchanged.

New since rc0

Safer paid offers

  • Selling LiteLLM no longer exposes admin access. The seller gateway attaches the LiteLLM master key to paid requests, and every path was forwarded, so one payment could reach admin endpoints such as /model/new or /key/generate. Offers on the litellm upstream now reach only chat, completions and embeddings. Anything else, including encoded or .. paths, gets a 404 before payment.
  • The seller picks the model. obol sell http --upstream litellm now requires --model, and obol sell update <name> --model <model> changes it. The gateway overwrites the buyer's model, and strips the other ways a buyer could switch models: fallbacks, extra_body.model, the ?model= query parameter and x-litellm-* headers.
  • Agent offers reach only POST /v1/chat/completions. The Hermes API key the gateway attaches would otherwise let a paying buyer read or delete other buyers' sessions and stored responses, create jobs, or steer runs. Each paid request now also gets its own Hermes session, so buyers with the same opening message no longer share one.
  • No charge for an empty reply (#821). On inference, agent and LiteLLM offers, a request is settled only if the reply delivered content, reasoning or tool calls. Non-streaming replies are held until they finish, and streaming replies are watched as they go out without being slowed. Unsettled replies carry X-Payment-Settled: false.
  • One completion per payment: n and best_of are removed from paid requests.

Grafana and better metrics

  • Grafana at http://obol.stack:8080/grafana, with no login and read-only access. It is only reachable on obol.stack, never through the tunnel. It comes with three dashboards:
    • Stack overview;
    • Ethereum node, for all supported clients (/grafana/d/obol-eth-node?var-namespace=ethereum-<id>);
    • Charon.
  • Prometheus keeps 8 days of data on disk (5 Gi, capped at 4 GB). Before, it kept 2 days in memory.
  • eRPC, Traefik and cloudflared metrics are now collected.
  • LLM performance: LiteLLM's Prometheus metrics are on, and the new obol-llm dashboard shows time to first token (p50/p95/p99), generation tokens per second, latency and error rates by model and upstream. New rules give one-number answers: obol:llm_ttft_seconds:p95_5m and obol:llm_output_tokens_per_second:mean_5m. flows/llm-bench.sh sends the same requests to each model, so comparisons are fair. Time to first token is only recorded for streaming requests.

LiteLLM v1.105.0

The Obol LiteLLM fork moves from v1.87 to v1.105.0. That fixes two upstream security advisories: buyer-supplied api_base sending provider keys elsewhere, and a credentials file read. Obol's no-database model management is kept.

Upgrade notes for rc1

  • Existing sell http --upstream litellm offers stay not-ready until you set a model: obol sell update <name> -n <ns> --model <model>. obol sell status shows the exact command.
  • Agent clients calling any Hermes path other than chat completions now get a 404.
  • Prometheus moves onto a volume, which recreates it once, so up to 2 days of earlier metrics are lost.
  • Per-request prices don't limit max_tokens; price your offers with that in mind.

Release Highlights

obol doctor

One command checks the things that usually go wrong:

  • Host: Docker memory, disk space, the obol.stack name and the ingress ports.
  • Versions: obol and its pinned tools.
  • Cluster: the cluster and its kubeconfig, node pressure, unhealthy pods and Helm releases.
  • Services: models, LiteLLM drift, Ollama, the x402 CA bundle, the tunnel (including a rejected token or an HTTP 525) and wallet addresses.

Every problem comes with the command that fixes it. --fix runs only the safe repairs (missing tools, a stale kubeconfig, an empty CA bundle). --offline skips network checks, --strict fails on warnings too, and -o json gives machine-readable output.

obol doctor
obol doctor --fix

Is my node synced?

obol network list now shows your installed nodes, with each one's network, execution and consensus clients, and mode. It reads them from disk, so it works with the cluster down.

obol network status keeps the eRPC summary and adds a section for each local Ethereum node:

  • pod state;
  • execution sync, head block, peers and client version;
  • consensus sync, peers and version;
  • a one-line verdict: synced, syncing, waiting, degraded, el_offline, unreachable or not_deployed.

It reaches the node APIs through short-lived port-forwards, so nothing new is exposed on obol.stack.

obol network install ethereum --network hoodi --consensus-client teku
obol network list
obol network status ethereum/hoodi
obol -o json network status ethereum/hoodi

Execution and consensus client metrics are now scraped by the stack's Prometheus as well.

Hermes can watch the stack

The default Hermes agent can now read pods, logs, events, statefulsets, PVCs and jobs in every namespace, so it can answer "is my node synced?" or "why is this pod restarting?" on its own. Secrets and ConfigMaps stay out of reach, because several stack ConfigMaps hold credentials. The kube.py helper in its obol-stack skill gained -n/-A/-l, statefulset, PVC and job listings, and logs --since/--previous.

Network namespaces now grant ConfigMap read to Hermes. Before, they were bound to an agent service account that no longer existed. Run obol network sync --all to apply this to existing nodes.

Leaner Hermes skills

Hermes used to load 22 built-in skills on every turn. It now has 11 that operate the stack:

  • agent-factory, buy-x402, discovery, ethereum-local-wallet, ethereum-networks, inspect, obol-stack, sell, sub-agent-business and swap;
  • ethereum-reference, which merges the ten former reference skills (addresses, gas, L2s, standards, wallets and so on) into one, with a guide per topic.

Everything else is opt-in from ObolNetwork/skills. The release (v1.8.0) is pinned and checked by content digest.

obol agent skills list --available
obol agent skills add distributed-validators
obol agent skills add dv-steward       # Hermes watches a DV node running on the stack

Sub-agents take them through obol agent new … --skills or obol agent update <name> --skills +<skill>.

Node client updates

obol network sync now renders each node from the templates built into this binary, not from the copy made at install. That means version bumps and template fixes reach existing nodes. Your values.yaml is kept, and a failed sync leaves the deployment as it was.

New versions:

  • geth v1.17.8;
  • Nethermind 2.1.0;
  • Teku 26.10.0;
  • Nimbus v26.10.0;
  • Lodestar v1.50.0, the first stable release that schedules Gloas on Hoodi.

The stack's node doesn't run MEV-boost: validators using it build blocks locally until Gloas, and receive builder bids over p2p after.

Smaller wins

  • Grandine installs - the Grandine image is now pinned. Before, installing it failed with a nil-pointer error.
  • Old nodes sync - Ethereum nodes installed before v0.10 get a default mode on sync.
  • Faster turns - fewer built-in skills means a smaller system prompt for every agent turn.
  • Safe skill syncs - a sync removes only skills that obol installed. Skills you added by hand stay, and offline syncs keep your opt-in copies.

Breaking changes / Migration notes

  • OpenClaw is removed: obol openclaw … and --runtime openclaw now fail with a migration hint.
    • Before upgrading, back up an OpenClaw wallet with v0.15: obol agent wallet backup --runtime openclaw <id> --file w.json.
    • That backup restores into Hermes on v0.16: obol agent wallet restore --runtime hermes --input w.json --force. Back up the Hermes wallet first.
    • obol stack up lists any leftover openclaw-* namespaces, with their keystore paths.
  • obol domain and obol tunnel login are removed, along with tunnel setup --management local. An existing locally managed tunnel keeps running, but you can't add hostnames to it. Move to a dashboard-managed tunnel with obol tunnel setup <connector-token>.
  • ethereumjs is removed as an execution client: it only publishes nightly images and isn't Gloas-ready. Syncing an existing ethereumjs node now stops with a message. Delete it and reinstall with --execution-client reth, geth, nethermind, besu or erigon.
  • Skills: the ten reference skills are now the single ethereum-reference skill. distributed-validators is now opt-in, so agents that used it need obol agent skills add distributed-validators.
  • Nethermind 1.39 → 2.1: the next obol network sync moves Nethermind nodes to a new major version.

Known issues

  • Gloas on testnets: see the note at the top. Lighthouse and Grandine move to Gloas-ready versions once their stable releases ship.
  • dv-pod 0.22 isn't published yet: the opt-in dv-steward skill, and the DV pod flow it watches, depend on the dv-pod 0.22 chart.

What's Changed

  • fix(network): sync renders the built-in network files by @OisinKyne in #885

Full Changelog: v0.16.0-rc1...v0.16.0-rc2

Contributors

OisinKyne
Assets 11
Loading