Skip to content

One container, one yaml file - #27

Merged
tschm merged 2 commits into
mainfrom
feat/one-container
Aug 31, 2026
Merged

One container, one yaml file#27
tschm merged 2 commits into
mainfrom
feat/one-container

Conversation

@tschm

@tschm tschm commented Aug 31, 2026

Copy link
Copy Markdown
Member

Running the board meant cloning this repo, installing uv, editing two files and running up.sh, which started two containers and installed a launchd agent. Now it is one docker run and a repos.yml.

docker run -d --name jq-fleet \
  -p 127.0.0.1:3000:3000 \
  -v "$PWD/repos.yml:/config/repos.yml:ro" \
  -v "$HOME:/host:ro" \
  -v jq-fleet-data:/data \
  -e GITHUB_TOKEN="$(gh auth token)" \
  ghcr.io/jebel-quant/monitoring:latest

How the working copies get in

The collector lived on the host because it reads your working copies, and a per-repo bind mount can only name a checkout at /repos/<owner>/<name> — which silently loses the four repos in this fleet that live somewhere else.

The fix is to stop mounting repos one at a time. -v "$HOME:/host:ro" mounts the home directory once, whole, and ~/... in repos.yml resolves against it, so ~/repos/tschm/rhiza_projects/cs means exactly what it says. One mount expresses any layout, so the collector comes back inside.

Leave the mount off and the GitHub half still works; the working-copy panels stay empty. That is a supported way to run this, not an error.

The trade: the collector can see everything under $HOME, not only the checkouts you listed — the mount is the same width either way. It still never writes (every git call passes --no-optional-locks), and :ro now means the kernel enforces that rather than it being a property of the code.

repos.yml is read directly

The collector parses it at startup. That deletes the gen-repos.py → environment → collector.sh chain, and with it the three places a repo could be lost between the file and the board. JQ_REPOS / JQ_REPO_PATHS stay as the escape hatch for a deployment with no file to mount.

A broken file is a refusal at startup, not a board that is quietly one repo short — except an unreachable path alongside an explicit repo:, which is what "no $HOME mount" looks like and is logged as a warning.

One container, three processes

Prometheus, Grafana and the collector share a container, with the dashboard, datasource, alert rules and scrape config baked in. docker logs prefixes each process.

Nothing restarts a dead process: the three are one board, so any of them exiting takes the container down and Docker's restart policy handles it. That keeps docker ps honest about whether the board is up, which a supervisor quietly restarting one process in a still-healthy container would not.

Verified end to end

Built and run against the real 25-repo fleet:

  • 25 repos on the board, 24 with checkouts — including jq_repo_cloned{repo="tschm/cs"} 1.0, the nested path the old architecture could not express in a container
  • Prometheus target jq-collector up, Grafana datasource healthy, jq-fleet dashboard provisioned, 1795 metric lines
  • Collector at 100% line coverage, ruff clean, check-dashboard.py passes
  • History migrated from the old volumes: 2d 13h, one unbroken series, 6 alert rules and 364 annotations intact

Two bugs the run caught and this fixes: the dashboard provider still pointed at /var/lib/grafana/dashboards, which is now the /data volume; and Grafana was downloading a Loki plugin at every start.

Notes for review

  • The image does not exist yet. .github/workflows/image.yml publishes ghcr.io/jebel-quant/monitoring (amd64 + arm64) on merge to main. Every doc points at it, so until this merges the recipe is docker build -t ... .
  • First scan is slow — cold Docker Desktop mount: 21s for the very first git call, ~2 min before the GitHub pass started, ~3ms per call once warm. Documented in docs/operations.md.
  • purge-repo now needs a container restart to toggle the admin API, where the old script flipped it live. It prints the exact commands when the API is off.
  • Migrationdocs/operations.md#coming-from-the-two-container-stack carries the Prometheus history over. The instance relabel in prometheus.yml already existed for exactly this move, so the series do not fork.

Gone: scripts/up.sh, down.sh, collector.sh, gen-repos.py, docker-compose.admin.yml. docker-compose.yml is now one optional service; purge-repo.sh moves into the image as docker exec jq-fleet purge-repo.

🤖 Generated with Claude Code

tschm and others added 2 commits August 31, 2026 10:26
Running the board meant cloning this repo, installing uv, editing two files
and running up.sh, which started two containers and installed a launchd agent.
Now it is a docker run and a repos.yml.

The collector moves back into Docker. It was on the host because it reads your
working copies and a per-repo bind mount can only name a checkout at
/repos/<owner>/<name> - which silently loses the four repos in this fleet that
live somewhere else. The fix is to stop mounting repos one at a time: -v
"$HOME:/host:ro" mounts the home directory once, whole, and `~/...` in
repos.yml resolves against it, so ~/repos/tschm/rhiza_projects/cs means exactly
what it says. One mount expresses any layout. Leave the mount off and the
GitHub half still works; the working-copy panels stay empty.

repos.yml is now read by the collector itself, at startup. That deletes the
gen-repos.py -> environment -> collector.sh chain, and with it the three places
a repo could be lost between the file and the board. JQ_REPOS and
JQ_REPO_PATHS stay as the escape hatch for a deployment with no file to mount.

Prometheus, Grafana and the collector share a container, with the dashboard,
datasource, alert rules and scrape config baked into the image. Nothing
restarts a dead process: the three are one board, so any of them exiting takes
the container down and Docker's restart policy handles it, which keeps
docker ps honest about whether the board is up.

The instance relabel in prometheus.yml already existed for exactly this move,
so history carries over unforked - docs/operations.md has the volume migration.

Gone: scripts/up.sh, down.sh, collector.sh, gen-repos.py,
docker-compose.admin.yml. docker-compose.yml is now one optional service.
purge-repo.sh moves into the image as `docker exec jq-fleet purge-repo`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image job asserted repo="Jebel-Quant/actions" appeared in /metrics while
deliberately passing no token. It cannot: metrics.py builds its key set from
remote | local, and a repo with no checkout reaches the board only through the
GitHub half - which 401s unauthenticated on the artifact and dependabot
endpoints, so every repo raised and remote came back empty. The step's own
comment said it did not depend on GitHub having answered; now it does not.

The fleet resolving is asserted on the collector's startup line instead, which
is what that half of the test was actually about.

A repos.yml refusal now names the offending value rather than only the rule -
'entry 7' in a twenty-five entry file means counting lines, and the value is
what you can search the file for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tschm
tschm merged commit 7e49a81 into main Aug 31, 2026
6 checks passed
@tschm
tschm deleted the feat/one-container branch August 31, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant