One container, one yaml file - #27
Merged
Merged
Conversation
Running the board meant cloning this repo, installing uv, editing two files and running up.sh, which started two containers and installed a launchd agent. Now it is a docker run and a repos.yml. The collector moves back into Docker. It was on the host because it reads your working copies and a per-repo bind mount can only name a checkout at /repos/<owner>/<name> - which silently loses the four repos in this fleet that live somewhere else. The fix is to stop mounting repos one at a time: -v "$HOME:/host:ro" mounts the home directory once, whole, and `~/...` in repos.yml resolves against it, so ~/repos/tschm/rhiza_projects/cs means exactly what it says. One mount expresses any layout. Leave the mount off and the GitHub half still works; the working-copy panels stay empty. repos.yml is now read by the collector itself, at startup. That deletes the gen-repos.py -> environment -> collector.sh chain, and with it the three places a repo could be lost between the file and the board. JQ_REPOS and JQ_REPO_PATHS stay as the escape hatch for a deployment with no file to mount. Prometheus, Grafana and the collector share a container, with the dashboard, datasource, alert rules and scrape config baked into the image. Nothing restarts a dead process: the three are one board, so any of them exiting takes the container down and Docker's restart policy handles it, which keeps docker ps honest about whether the board is up. The instance relabel in prometheus.yml already existed for exactly this move, so history carries over unforked - docs/operations.md has the volume migration. Gone: scripts/up.sh, down.sh, collector.sh, gen-repos.py, docker-compose.admin.yml. docker-compose.yml is now one optional service. purge-repo.sh moves into the image as `docker exec jq-fleet purge-repo`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image job asserted repo="Jebel-Quant/actions" appeared in /metrics while deliberately passing no token. It cannot: metrics.py builds its key set from remote | local, and a repo with no checkout reaches the board only through the GitHub half - which 401s unauthenticated on the artifact and dependabot endpoints, so every repo raised and remote came back empty. The step's own comment said it did not depend on GitHub having answered; now it does not. The fleet resolving is asserted on the collector's startup line instead, which is what that half of the test was actually about. A repos.yml refusal now names the offending value rather than only the rule - 'entry 7' in a twenty-five entry file means counting lines, and the value is what you can search the file for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running the board meant cloning this repo, installing
uv, editing two files and runningup.sh, which started two containers and installed a launchd agent. Now it is onedocker runand arepos.yml.How the working copies get in
The collector lived on the host because it reads your working copies, and a per-repo bind mount can only name a checkout at
/repos/<owner>/<name>— which silently loses the four repos in this fleet that live somewhere else.The fix is to stop mounting repos one at a time.
-v "$HOME:/host:ro"mounts the home directory once, whole, and~/...inrepos.ymlresolves against it, so~/repos/tschm/rhiza_projects/csmeans exactly what it says. One mount expresses any layout, so the collector comes back inside.Leave the mount off and the GitHub half still works; the working-copy panels stay empty. That is a supported way to run this, not an error.
The trade: the collector can see everything under
$HOME, not only the checkouts you listed — the mount is the same width either way. It still never writes (every git call passes--no-optional-locks), and:ronow means the kernel enforces that rather than it being a property of the code.repos.ymlis read directlyThe collector parses it at startup. That deletes the
gen-repos.py→ environment →collector.shchain, and with it the three places a repo could be lost between the file and the board.JQ_REPOS/JQ_REPO_PATHSstay as the escape hatch for a deployment with no file to mount.A broken file is a refusal at startup, not a board that is quietly one repo short — except an unreachable
pathalongside an explicitrepo:, which is what "no$HOMEmount" looks like and is logged as a warning.One container, three processes
Prometheus, Grafana and the collector share a container, with the dashboard, datasource, alert rules and scrape config baked in.
docker logsprefixes each process.Nothing restarts a dead process: the three are one board, so any of them exiting takes the container down and Docker's restart policy handles it. That keeps
docker pshonest about whether the board is up, which a supervisor quietly restarting one process in a still-healthy container would not.Verified end to end
Built and run against the real 25-repo fleet:
jq_repo_cloned{repo="tschm/cs"} 1.0, the nested path the old architecture could not express in a containerjq-collectorup, Grafana datasource healthy,jq-fleetdashboard provisioned, 1795 metric linescheck-dashboard.pypassesTwo bugs the run caught and this fixes: the dashboard provider still pointed at
/var/lib/grafana/dashboards, which is now the/datavolume; and Grafana was downloading a Loki plugin at every start.Notes for review
.github/workflows/image.ymlpublishesghcr.io/jebel-quant/monitoring(amd64 + arm64) on merge to main. Every doc points at it, so until this merges the recipe isdocker build -t ... .gitcall, ~2 min before the GitHub pass started, ~3ms per call once warm. Documented indocs/operations.md.purge-reponow needs a container restart to toggle the admin API, where the old script flipped it live. It prints the exact commands when the API is off.docs/operations.md#coming-from-the-two-container-stackcarries the Prometheus history over. Theinstancerelabel inprometheus.ymlalready existed for exactly this move, so the series do not fork.Gone:
scripts/up.sh,down.sh,collector.sh,gen-repos.py,docker-compose.admin.yml.docker-compose.ymlis now one optional service;purge-repo.shmoves into the image asdocker exec jq-fleet purge-repo.🤖 Generated with Claude Code