feat(deploy): Dockerfile + compose + ghcr CI (M5 packaging)#61
Closed
brunota20 wants to merge 6 commits into
Closed
feat(deploy): Dockerfile + compose + ghcr CI (M5 packaging)#61brunota20 wants to merge 6 commits into
brunota20 wants to merge 6 commits into
Conversation
Closes the M5 packaging gap surfaced by the audit: the Dockerfile +
compose recipe lived inside `docs/production.md` but neither was at
the repo root, so `docker build .` didn't work and there was no
published image. This change makes the deploy path one-line on a
fresh VM.
## What ships
- **`Dockerfile`** — multi-stage build (rust:1.96-slim-bookworm →
debian:bookworm-slim). Builds the engine in release + the 5
production modules to wasm32-wasip2. Runtime stage strips down to
`tini` (PID 1 for graceful shutdown / SIGINT forwarding per
COW-1072) + `ca-certificates` (TLS to cow.fi + paid RPCs) + a
non-root `shepherd` user owning `/var/lib/shepherd`. Final image:
**198 MB** (engine + 5 wasm modules + Debian slim).
- **`.dockerignore`** — excludes `target/`, `data/`, the heavy
backtest / baseline JSON fixtures, and local-only engine configs,
while keeping `modules/fixtures/*-bomb` (workspace members; Cargo
rejects the manifest if they're missing) and the source markdown
docs (so `docker exec` can grep them in place).
- **`docker-compose.yml`** — two profiles. Default boots just the
engine with a `shepherd-state` named volume + the operator's
`./engine.toml` mounted ro at `/etc/shepherd/engine.toml`, metrics
on the host loopback (`127.0.0.1:9100`). The `observability`
profile (`docker compose --profile observability up`) layers a
Prometheus container pre-wired to scrape `shepherd:9100`. Graceful
shutdown via `stop_signal: SIGINT` + `stop_grace_period: 30s` per
the production runbook. Healthcheck hits `/metrics`.
- **`engine.docker.toml`** — pre-baked config that matches the
paths the image bakes (`/opt/shepherd/modules/*.wasm`,
`/opt/shepherd/manifests/*.toml`, `/var/lib/shepherd` state
dir). Operator workflow: `cp engine.docker.toml engine.toml`,
swap `<RPC_KEY>` placeholders, `docker compose up -d`.
- **`docs/deployment/docker.md`** — operator runbook. Covers
first-boot, engine.toml configuration, upgrade / rollback,
local-build path, post-deploy verification, cross-links to
`docs/production.md` for the full hardening surface.
- **`docs/deployment/prometheus.yml`** — scrape config consumed by
the observability compose profile.
- **`.github/workflows/docker.yml`** — build + push to
`ghcr.io/bleu/nullis-shepherd` on every push to `main` and every
`v*` tag. PR builds run the build for smoke (no push). Tags
produced: `latest` (main HEAD), `v<tag>` (releases),
`sha-<short>` (every event for exact pinning), `manual-<run_id>`
(workflow_dispatch). Registry-side layer cache via
`:buildcache` keeps incremental rebuilds fast. linux/amd64 only —
the soak VM is x86_64; add arm64 once an operator surfaces a
real need. Action SHAs pinned to match `.github/workflows/ci.yml`
style.
## Smoke validation
Build runs locally end-to-end in ~10 min on a clean Docker daemon:
$ docker build -t shepherd:smoke .
$ docker run --rm shepherd:smoke --help
usage: nexum-engine [<wasm-path> [<manifest-path>]] \
[--engine-config <path>] [--pretty-logs]
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^- expected: <RPC_KEY> placeholder not a real URL
Proves: image builds, entrypoint forwards CMD, engine loads
`/etc/shepherd/engine.toml`, metrics exporter binds, provider pool
iterates the configured chains, graceful error path works.
## Tests
- [x] Local `docker build .` succeeds (rust:1.96 base — wasmtime 45
requires rustc >= 1.93, the docs/production.md `1.86` pin was
stale)
- [x] Image size: 198 MB
- [x] `docker run ... --help` works
- [x] `docker run ... -v engine.docker.toml:...` reads config + binds
metrics + iterates chains
- [x] `cargo test --workspace` clean (18 groups, 203 passed, 0 failed)
## Reproducing the soak deploy
On a fresh Debian/Ubuntu VM with Docker installed:
```bash
git clone https://github.com/bleu/nullis-shepherd /opt/shepherd
cd /opt/shepherd
cp engine.docker.toml engine.toml
$EDITOR engine.toml # add real RPC URL
docker compose pull # once ghcr.io image is published
docker compose up -d
docker compose logs -f shepherd
curl -s http://127.0.0.1:9100/metrics | head -50
```
## Follow-ups for M5 (separate PRs)
- `docs/deployment/multi-chain-guide.md` — dedicated walkthrough
configuring 4 chains together (Mainnet + Gnosis + Arbitrum + Base)
with per-chain module subscriptions
- Example module declaring multi-chain support (every current
example pins Sepolia)
- Optional automated CD trigger (workflow_dispatch SSH'ing to the
soak VM to pull + restart) — gated on SSH_PRIVATE_KEY repo secret
AI-assisted authoring with Claude (Opus 4.7); smoke-validated end-
to-end via a local docker build + run before push.
Companion to the M5 Docker packaging — the operator workflow is `cp engine.docker.toml engine.toml` then drop in a paid RPC URL. Without this rule a clumsy `git add -A` could commit the key. The committed sibling templates (engine.example/docker/m2/m3/e2e/load.toml) stay trackable. Validated against a live smoke run: drpc Sepolia WSS endpoint pasted into engine.toml, `docker compose up`, engine subscribed to newHeads + logs, 6 sequential blocks dispatched (11117171..76), metrics `shepherd_event_latency_seconds` p99 = 0.14ms. Tear-down clean. No engine.toml ever staged.
Closes the footgun surfaced by the M5 smoke run on drpc Sepolia:
configuring `rpc_url = "https://..."` for a chain that the modules
subscribe to silently degrades to an infinite WARN-with-backoff loop
(COW-1071's reconnect retries forever because `eth_subscribe` is
WS-only in the JSON-RPC spec). Three coordinated changes:
## 1. Boot-time validation
`EngineConfig::validate_transports()` walks every `[chains.<id>]`
entry, and for any `rpc_url` not starting with `ws://` / `wss://`
emits one loud ERROR-level structured log line with:
- the chain id
- the redacted offending URL
- the redacted suggested `wss://` swap
- actionable copy explaining the WS requirement and the escape
hatch (`[chains.<id>] require_ws = false` for poll-only chains
that never subscribe)
The validator is invoked from `main.rs` AFTER the tracing
subscriber is initialised (calling it inside `load_or_default`
silently dropped the log).
A `require_ws: bool` field is added to `ChainConfig` with
`#[serde(default = "default_require_ws")]` = `true`. Operators who
genuinely need an HTTP endpoint (poll-only modules, no block / log
subscriptions on this chain) opt out explicitly per chain.
## 2. URL redaction in boot logs
The pre-existing `opening chain RPC provider` log in
`provider_pool::from_config` was emitting the full URL — API key
included — at INFO level. Log aggregators (Loki / Datadog / Splunk)
routinely retain weeks of these lines; the key has no business
sitting in cold storage. The new `engine_config::redact_url` helper
(public so other call sites can adopt it) replaces any path segment
longer than 20 chars that doesn't contain `.` or `:` with `<KEY>`.
Matches Alchemy / drpc / Infura / QuickNode key shapes.
Same helper is used for both the validation ERROR's `rpc_url` and
`suggested` fields and the provider-pool boot log.
## 3. Docs + example cleanup
- `engine.example.toml`: every chain entry switched to `wss://`,
with a header block explaining the WS requirement + the
`require_ws = false` escape hatch. The previous mix of `https://`
+ `wss://` would have tripped the new validator on its own example.
- `docs/production.md §6`: blockquote callout pointing operators at
the WS requirement, redaction behaviour, and the escape hatch.
## Validation evidence
Smoke 1 (HTTP, expected to ERROR):
{"level":"ERROR","message":"rpc_url uses HTTP transport but the engine subscribes to blocks/logs via eth_subscribe (WS-only). [...]","chain_id":11155111,"rpc_url":"https://lb.drpc.live/sepolia/<KEY>","suggested":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
Smoke 2 (WSS, expected to pass + redacted):
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,"url":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
## Tests
- 9 new unit tests in `engine_config::tests`:
* `validate_accepts_wss_url`, `validate_accepts_ws_url`
* `validate_is_silent_when_require_ws_is_false`
* `validate_runs_without_panicking_on_http_url`
* `suggest_swaps_https_to_wss`, `suggest_swaps_http_to_ws`,
`suggest_passes_through_already_ws_url`
* `redact_replaces_long_path_segments`,
`redact_keeps_short_segments_intact`
- Workspace: 18 groups, **212 passed, 0 failed** (was 203 → +9)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
AI-assisted authoring with Claude (Opus 4.7); validated against the
live drpc Sepolia endpoint via two smoke runs (HTTP fail, WSS
pass) before push.
Operator workflow before this change forced the paid-RPC URL to
live in a file (`engine.toml`), which is fine for systemd but
awkward for Docker/compose: the URL had to be hand-edited inside a
volume-mounted file, secrets and config got tangled, and the
internal drpc test key was at risk of slipping into a committed
example. This change makes the engine treat `${VAR_NAME}` tokens
inside `engine.toml` as environment-variable references, resolved
at config-load time:
[chains.11155111]
rpc_url = "${SEPOLIA_RPC_URL}"
The `engine.docker.toml` and `engine.example.toml` templates ship
with `${VAR}` placeholders for all five chains, so the committed
files stay secret-free regardless of deployment path.
## Operator workflow (Docker)
cp .env.example .env
$EDITOR .env # paste real wss:// URLs
docker compose up -d
`docker compose` reads the repo-root `.env` automatically (already
the compose default) and forwards the named variables into the
container via the new `environment:` block; the engine substitutes
them when parsing `/etc/shepherd/engine.toml`.
## Implementation
- `engine_config.rs::substitute_env_vars` — hand-rolled parser
(no regex dep) that walks the raw TOML text, matches `${NAME}`
tokens against `[A-Z_][A-Z0-9_]*`, and looks each up via
`std::env::var`. Three error variants via `thiserror`:
* `Missing { name }` — variable referenced but unset; message
includes the exact name and a pointer to the `.env` workflow.
* `InvalidName { name }` — typo (lowercase, leading digit);
suggests the upper-cased variant.
* `Unclosed { offset }` — `${` without matching `}`.
- Called from `load_or_default` before `toml::from_str`, so the
substitution layer never sees parsed TOML — a missing env var
surfaces with the exact variable name, not a downstream
"invalid URI character" several layers deep.
- Substitution runs over the whole file (comments included; harmless).
## Companion changes
- `.env.example` — committed template with placeholders for all 5
chain `*_RPC_URL` variables + the optional `SHEPHERD_IMAGE` and
`SHEPHERD_ENGINE_CONFIG` overrides.
- `.gitignore` — adds `!.env.example` exception so the template
stays trackable while `.env` and `.env.local` etc. stay ignored.
- `docker-compose.yml` — passes the five `*_RPC_URL` env vars
through to the container; the engine config bind-mount now
defaults to `engine.docker.toml` (the committed template) and
honours `SHEPHERD_ENGINE_CONFIG` for operators who prefer a
bespoke file.
- `engine.docker.toml` + `engine.example.toml` — every `[chains.*]`
entry switched to `${*_RPC_URL}` placeholders. Header comments
spell out the workflow.
- `docs/deployment/docker.md` — first-boot section now leads with
`cp .env.example .env` (was `cp engine.example.toml engine.toml
&& edit`). §2 explains the bind-mount + the
`SHEPHERD_ENGINE_CONFIG` escape hatch.
## Validation
Smoke 1 (compose end-to-end):
$ cp .env.example .env
$ echo "SEPOLIA_RPC_URL=wss://lb.drpc.live/sepolia/<real-key>" >> .env
$ echo "SHEPHERD_ENGINE_CONFIG=./engine.local.toml" >> .env
$ docker compose up -d
...
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,
"url":"wss://lb.drpc.live/sepolia/<KEY>",...} ← env-resolved, key redacted
{"level":"INFO","message":"supervisor up","loaded":2,"alive":2,...}
{"level":"INFO","message":"block subscription open","chain_id":11155111,...}
{"level":"INFO","message":"log subscription open","module":"twap-monitor",...}
{"level":"INFO","message":"log subscription open","module":"ethflow-watcher",...}
$ docker compose logs | grep -c <real-key>
0 ← zero leaks
$ curl -s http://127.0.0.1:9100/metrics | grep latency_seconds_count
shepherd_event_latency_seconds_count{module="twap-monitor",event_kind="block"} 4
Smoke 2 (missing env var, expected fail-fast):
$ unset SEPOLIA_RPC_URL
$ docker compose up
Error: engine config env-var substitution failed: environment variable
`SEPOLIA_RPC_URL` referenced via ${SEPOLIA_RPC_URL} in engine.toml but
not set. Export it before launching the engine (e.g. via a `.env`
file consumed by `docker compose`).
## Tests
- 7 new unit tests in `engine_config::tests`:
* `substitute_replaces_known_variable`
* `substitute_errors_on_missing_variable`
* `substitute_errors_on_invalid_name`
* `substitute_errors_on_unclosed_brace`
* `substitute_passes_text_with_no_placeholders_through`
* `substitute_handles_multiple_placeholders_in_one_line`
* `substitute_preserves_utf8_around_placeholder`
- Workspace: 18 groups, **219 passed, 0 failed** (was 212 → +7)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
AI-assisted authoring with Claude (Opus 4.7); validated end-to-end
against drpc Sepolia via the documented `.env` -> compose -> engine
workflow before push (no key ever staged).
VM smoke surfaced a false-negative `(unhealthy)`: the compose healthcheck called `wget` but the runtime image is built on debian:bookworm-slim which doesn't include it (only ca-certificates + tini, intentionally minimal). `wget: not found` → exit 127 → unhealthy mark, despite the engine actually working (21 blocks dispatched in 3 min, p99 latency 0.09ms, zero errors). Swap to bash's `/dev/tcp` builtin (always present in bookworm-slim's `/bin/bash`). Successful TCP open on the metrics port proves the exporter bound, which only happens after the supervisor finishes boot — same semantic, no image growth.
First fix attempt swapped wget for `/dev/tcp` but kept `CMD-SHELL`, which routes through `/bin/sh` (dash on debian:bookworm-slim). dash doesn't have the `/dev/tcp/<host>/<port>` builtin — it's bash- only. Probes failed with "cannot create /dev/tcp/...: Directory nonexistent". Switch to `CMD ["bash", "-c", ...]` so the bash builtin actually resolves. `bash` ships in the slim base; verified via `docker exec shepherd which bash` → `/usr/bin/bash`.
Author
This was referenced Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the M5 packaging gap from the milestone audit: Dockerfile + compose recipe were inside `docs/production.md` but neither was at repo root. `docker build .` now works, and the CI publishes to `ghcr.io/bleu/nullis-shepherd` on every push to main and every `v*` tag.
What ships
Smoke validation (local)
```
$ docker build -t shepherd:smoke . # ~10 min on cold daemon
$ docker run --rm shepherd:smoke --help # entrypoint OK
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^ expected: placeholder, not a real URL
```
Proves: image builds, entrypoint forwards CMD, engine loads config, metrics exporter binds, provider pool iterates chains, graceful error path works.
Notes
Stacked on
PR #60 (`feat/backtest-cow-1078`) → PR #59 → PR #58 → PR #57 → upstream M4 epic (TBD). The Docker image will include all M3 + M4 work once the upstream stack merges.
Follow-ups (separate PRs)
Tests
AI-assistance disclosure
Authored with Claude (Opus 4.7); smoke-validated locally via docker build + run before push.