feat(docker): add din-node containerized run path for dincli

[Santiagocetran]

Summary

Adds a containerized din-node run path for dincli: a Docker image plus a
Compose setup that lets operators run dincli without installing Python locally.
This is Phase 1: Devnet Operator Baseline of the validator-operations roadmap.

din-node is the trusted host process. It holds wallet/config state and uses
the host Docker daemon to spawn the existing per-job din-worker:dev containers
as siblings. The worker sandbox itself is unchanged.

Changes

• Add dincli/docker/node/Dockerfile — the din-node image.
• Add dincli/docker/node/docker-compose.yml — Docker socket mount + persistent
  bind-mounted state.
• Add dincli/docker/node/.env.example — host-specific UID/GID/state config
  (operators copy it to .env, which is git-ignored).
• Add dincli/docker/node/README.md — operator runbook.
• Add .dockerignore — reduces the Docker build context.
• Update .gitignore — ignore local virtualenvs (.venv*/).

Design notes

• Installs dincli as a built, non-editable package (upgrade = rebuild the
  image). The image installs the current source tree as a built package; current
  package metadata is 0.1.0. Python pinned to 3.12-slim.
• Pins dependencies via dincli/requirements.txt (exact ==) for reproducibility.
• Installs the Docker CLI only (docker-ce-cli), not a daemon — it talks to
  the host daemon over the mounted socket.
• Runs non-root (dinuser) by default, with group_add: ${DOCKER_GID} so it
  can access /var/run/docker.sock (DOCKER_GID is host-specific:
  stat -c '%g' /var/run/docker.sock).
• Uses identical-path bind mounts for DIN_STATE_DIR, not named volumes:
  worker containers are created by the host daemon, which resolves the -v
  path against the host filesystem. A named volume's data lives under Docker's
  internal storage, not at the in-container path dincli passes, so the worker
  would get an empty mount and fail silently. Bind-mounting one host dir at the
  identical absolute path makes it valid from both views. (Confirmed with the lead.)

Volume layout (all under one host dir DIN_STATE_DIR):

Path	dincli dir	Survives upgrade	Safe to wipe
config/dincli	CONFIG_DIR (wallet/config)	yes — keep	❌ no (wallet)
cache/dincli	CACHE_DIR (manifests/models/jobs)	yes	⚠️ mostly
cache/dincli-docker	DOCKER_CACHE_DIR (worker pkgs)	yes	✅ yes

Security tradeoff

din-node mounts /var/run/docker.sock, which gives it root-equivalent control
of the host Docker daemon. This is intentional for the devnet baseline and
documented in the runbook. Untrusted worker job containers still run without
the socket, without wallet state, and with the existing sandbox constraints
(--network none, read-only mounts, resource limits). One caveat surfaced while
documenting this: the dependency-install step (pip install of the model owner's
requirements) runs in a networked container by necessity, and pip can run
package build hooks — so install is not network-isolated (existing worker-layer
behaviour, out of scope to change here, flagged in the runbook). Operator
guidance: don't run din-node on a host you don't trust the operator of.

Verification

Tested on Docker 29.x:

• din-node image builds and runs as non-root dinuser; dincli works inside.
• Docker CLI inside din-node reaches the host daemon (socket access works).
• Sibling worker spawn works through the identical-path bind mount — validated with
  a real din-worker:dev run (--network none) that executed and wrote
  result.json (output owned by the host user, not root).

Doc inaccuracies found (per task §5)

• containerization.md "Phase 1: Local Docker Worker Prototype" is written as
  future work, but the worker sandbox already exists in worker.py. The only
  un-built Phase-1 item is the Compose/operator run path — this PR.
• Role coverage is ahead of the doc: aggregator/auditor are listed as Phase 2, but
  all three roles already call the worker layer.
• The doc's worker mount table doesn't match code (proposes /din/data,
  /din/logs; actual: /din/model ro, /din/job/job.json ro, /din/output rw,
  /din/packages ro).
• validator-operations.md lists "no standard Docker Compose run path" — addressed here.
• Minor: dincli/cli/utils.py docstrings reference ~/.din/...; real paths are
  ~/.config/dincli/... via platformdirs. And pyproject.toml
requires-python = ">=3.9" conflicts with py-cid==0.5.0 (>=3.10).

With more time

Pin the base image by digest, add a startup check for XDG_*/socket access, and
run a full train-lms against a local chain (the no-chain worker repro already
proves the spawn mechanism).

[umeradl]

PR Review — Santiago Rodriguez Cetran — feature/din-node-container

PR: #12
Commit: a985ed047537617f1abc24c45eb4fea510eecb9d
Task: task.md — din-node containerized run path for dincli

---

Summary verdict

Strong, complete delivery. Built the image, ran docker compose up, and personally
verified the core risk the task called out (sibling worker spawn through the
mounted docker.sock) — it works exactly as documented. Doc-accuracy pass is
correct and well-cited. Two real gaps surfaced in follow-up discussion (below),
neither a blocker, both good material for review comments / a fast follow-up.

---

What's outstanding

Engineering judgment — the bind-mount-vs-named-volume call

Tested named volumes first, found they break the worker flow, correctly
diagnosed why (Docker-outside-of-Docker path resolution: the host daemon
resolves -v paths against the host filesystem, not the calling
container's filesystem — a named volume's real data lives under
/var/lib/docker/volumes/.../_data, not at the in-container path dincli
passes), and switched to identical-absolute-path bind mounts as the fix. This
is the single most important design decision in the deliverable (the task
explicitly says so), and he got it right with a sound architectural reason,
not a guess.

Code quality

• din-node Dockerfile: pinned Python (3.12-slim), pinned deps
  (dincli/requirements.txt, exact ==), Docker CLI client only (no
  daemon) installed from Docker's official apt repo, non-root dinuser
  (uid/gid 1000, overridable) with group_add: ${DOCKER_GID} as the one
  documented privilege escalation needed for socket access.
• Built (non-editable) install, deliberately — "din-node is a release
  artifact; upgrades rebuild the image rather than mutating mounted source."
  Good call, explicitly justified inline.
• Compose file uses required (:?) variable interpolation for
  DIN_STATE_DIR everywhere it's referenced — if unset, docker compose
  hard-fails instead of silently rendering a broken /config:/cache mount
  that would defeat the identical-path guarantee with no visible error. This
  is a non-obvious defensive detail that shows he was thinking about failure
  modes, not just the happy path.
• Worker container names follow a predictable pattern
  (din-worker-client-model-<id>-gi-<n>, similarly for auditor/aggregator),
  and they run with --rm, so they self-clean after each job — verified in
  code (dincli/cli/worker.py, client.py).

Documentation

• The README (dincli/docker/node/README.md) is genuinely good operator
  documentation: ASCII diagram of the sibling-container architecture, an
  explicit "why bind mounts not named volumes" callout, a volume table with
  survive-upgrade / safe-to-delete columns, and a troubleshooting table
  (socket permission errors, ownership issues, path mismatches) that reads
  like it came from actually hitting those errors, not guessing at them.
• Security tradeoff section is the standout. Calls the docker.sock
  mount "root-equivalent control of the host daemon" in plain language, states
  the operator-trust implication directly ("don't run din-node on a host
  whose operator you do not trust"), and — this is the detail that shows real
  understanding rather than box-ticking — flags that the worker dependency
  install step runs in a networked container (it has to, to pip install)
  and pip can execute arbitrary build hooks from third-party packages, so
  that step is not network-isolated even though the actual job execution is.
  That's an existing property of worker.py (out of scope to fix here), but
  noticing and stating it accurately is the kind of thing that's easy to miss.
• Doc-inaccuracy pass (§5 of the task) is accurate, not padding. Verified
  directly against Developer/issues/containerization.md and
  validator-operations.md:
  • "Phase 1: Local Docker Worker Prototype" is written as future work, but
    the worker sandbox (worker.py) already exists and works — confirmed.
  • Phase 2 lists aggregator/auditor worker coverage as future work; all
    three roles already call the worker layer — confirmed.
  • The doc's suggested worker mount table (/din/data, /din/logs)
    doesn't match the actual mounts in code (/din/model ro,
    /din/job/job.json ro, /din/output rw, /din/packages ro) — confirmed.
  • validator-operations.md's "no standard Docker Compose run path" is now
    stale given this PR — correct to flag.
  • Bonus catch: pyproject.toml pins requires-python = ">=3.9" while
    py-cid==0.5.0 (a real dependency) reportedly needs >=3.10 — a
    pre-existing inconsistency unrelated to his PR, but good that he noticed
    it while in the file.

Verification rigor

The PR's "Verification" section doesn't just assert things work — it names
specific things tested (non-root execution, socket reachability, a real
din-worker:dev spawn producing result.json owned by the host user). I
independently re-ran the same class of test from scratch (built both images,
brought up din-node via compose, wrote a file from inside the container at
a CACHE_DIR-rooted path, spawned a sibling worker via docker run over the
socket, and confirmed the worker saw the real file content and wrote output
back with correct host-user ownership, not root). Everything he claimed
reproduced exactly.

I also had a sub-agent independently trace, for all four worker-spawning
commands (train-lms, auditor evaluate, aggregate-t1, aggregate-t2),
whether every path handed to run_worker_container / ensure_worker_packages_installed
/ write_worker_job is actually rooted under CACHE_DIR/DOCKER_CACHE_DIR
(and therefore covered by the bind mount) — confirmed true for all four
today. This is the kind of thing that's easy to get subtly wrong (one
overlooked call site using a different path source would silently break the
whole guarantee for that command), and it holds.

---

Gaps found (worth a review comment, not blockers)

These came out of operator-experience questions we walked through after the
initial read — re-reading the runbook line by line, they're genuinely not
covered:

1. No migration story from an existing host install. Today, a host
   install of dincli puts state at the platformdirs defaults
   (~/.config/dincli, ~/.cache/dincli, ~/.cache/dincli-docker).
   din-node's compose file redirects XDG_CONFIG_HOME/XDG_CACHE_HOME so
   the same platformdirs calls resolve under $DIN_STATE_DIR instead —
   a different path. The README's "First-time setup" only covers a brand-new
   operator with no prior state; there's no documented step for someone
   moving from a host install to din-node (e.g. cp -r ~/.config/dincli $DIN_STATE_DIR/config/dincli, similarly for the two cache dirs). Getting
   this wrong is exactly the kind of mistake that loses a wallet, so it's
   worth being explicit even though it's a small addition.

2. No way to see which containers belong to DIN. Worker container names
   are predictable (din-worker-<role>-model-<id>-gi-<n>) and they self-clean
   via --rm on normal exit, but none of that is documented. docker compose ps only shows din-node itself — there's no docker ps -a --filter "name=din-" (or similar) for an operator to confirm whether a worker is
   currently running, or to spot anything that didn't exit cleanly.

3. "Remove" is only half-addressed. docker compose down covers
   din-node. There's no guidance for the abnormal case — a worker
   container hung or got orphaned (e.g. daemon restart mid-job) without its
   --rm triggering — on how an operator would notice and clean it up
   (docker rm <name>).

None of these affect correctness of what's shipped; they're operator-runbook
completeness gaps for the "how do I actually live with this day to day"
angle, which is squarely what Phase 1 is supposed to deliver.

---

Questions / discussion points for Santiago

• Can he add a short "migrating from a host install" subsection to the
  README covering the three directory moves into $DIN_STATE_DIR?
• Can he add a one-liner for listing DIN-related containers
  (docker ps -a --filter "name=din-" or similar) and what to do with a
  stuck worker container?
• Confirm/deny the py-cid==0.5.0 / requires-python>=3.9 Python version
  conflict he flagged — is it real, and if so should it be a separate
  one-line fix PR against pyproject.toml?
• Any opinion on whether worker containers should carry a label
  (docker run --label din.role=client ...) rather than relying purely on
  the name prefix, for more robust filtering later (e.g. by dind (din daemon))?

---

Roadmap follow-ups / potential next task

• Operator state-migration + container introspection — turn the three
  gaps above into a small fast-follow PR (likely <1 day): README migration
  section, docker ps filter convention (possibly backed by container
  labels), and orphaned-worker cleanup guidance. Good next task for him
  specifically since he already has full context on this exact area.
• pyproject.toml Python-version pin fix — trivial, one-line, could be
  bundled with the above or split out.
• Phase 2 of validator-operations roadmap (Daemon Operations) assumes a
  persistent host process exists to manage — this PR is exactly that
  prerequisite, now satisfied. He'd be a strong candidate for early dind
  daemon work (health endpoint, structured stdout logs, graceful SIGTERM)
  given he now understands the container/socket architecture end to end.
• Doc fixes — the containerization.md / validator-operations.md
  inaccuracies he found are one-liners; worth either a tiny doc patch in this
  PR or a follow-up doc-only PR so the roadmap docs stop contradicting the
  code.

[umeradl]

Follow-up review — closing

Second commit (9e95bed) addresses all three gaps from the review. Quick pass on each:

Gap 1 — Migration story ✅
Migrating from a host install section is exactly what was asked for: idempotent cp -a . commands for all three dirs, a read-wallet verify step, and the encrypted-wallet password-prompt caveat. The cp -a vs mv advice (keep the host copy until confirmed) is the right call for an operation that touches the wallet.

Gap 2 — Container visibility ✅
docker ps -a --filter "name=din-" is in the day-to-day lifecycle table. The note about the anonymous dependency-install container (won't match the name=din- filter; use --filter "ancestor=din-worker:dev" if you need to catch it mid-install) is a genuinely useful addendum — that's an easy thing to wonder about.

Gap 3 — Orphaned worker cleanup ✅
Cleaning up a stuck or orphaned worker subsection covers all the cases: check, stop (if running), remove, and bulk-clear of exited workers. The warning about not blindly docker stop-ping a running worker (kills the active job, may leave partial output) is the right safety note.

Doc patches ✅
containerization.md and validator-operations.md annotations are minimal and accurate — inline > Status (2026-06) callouts that don't rewrite history, just correct the reader's expectation. The actual mount table note (citing run_worker_container:138-159) is a good pointer.

No new issues introduced. The README is now complete as an operator runbook: first-time setup, migration, day-to-day, state layout, stuck-worker recovery, troubleshooting, and scope/limitations all present. Merging.