Read a checkout where it actually is - #22
Merged
Merged
Conversation
localgit.scan derived every path as <JQ_REPO_ROOT>/<owner>/<name>. Inside the
container that is always right, because the generated bind mounts put each
checkout at exactly that path - repos.yml can say anything and the mount
normalises it. Outside the container it is often wrong, and silently so.
Four repos in the fleet this was built against are laid out like this:
- path: ~/repos/tschm/rhiza_projects/cs # a checkout of tschm/cs
No amount of joining owner to name produces that path, so those repos dropped
off the working-copy panels while staying on the GitHub ones: present on the
board, quietly missing half their columns, and nothing said why. Running the
collector on the host reported 20 of 24 checkouts.
JQ_REPO_PATHS now carries the real path per repo, as owner/name=path pairs,
with <repo_root>/<owner>/<name> kept as the fallback. `gen-repos.py --env`
emits it from repos.yml alongside JQ_REPOS, so the layout is stated once, in
the file that already owns it, rather than retyped.
* The container is untouched: it sets no JQ_REPO_PATHS and uses the fallback
exactly as before. Verified - still 24 of 24, 1743 metric lines.
* On the host, 24 of 24 now, with JQ_REPO_ROOT empty and every path coming
from JQ_REPO_PATHS alone.
The origin check matters more now rather than less, since the path is
arbitrary - a path pointing at the wrong repo is still refused rather than
filed under the listed repo's name, and there is a test for it.
Two ways this could have gone quiet, both closed:
* A malformed pair raises at startup instead of being skipped. Dropping one
would take a repo's local panels off the board and say nothing - the exact
failure this module is shaped to avoid.
* A path containing a comma cannot be expressed, comma being the separator.
gen-repos.py refuses to emit one rather than print a line that would be
misread as two repos.
A bad JQ_REPO_ROOT no longer abandons the whole scan either: it logs once and
withdraws the fallback, leaving repos with explicit paths working.
Prometheus and Grafana stay in Docker - they are pinned images with
repo-tracked provisioning, which is what Docker is good at here. The collector
is not; it is a Python package that reads your working copies, and the
container was costing more than it gave:
* bind mounts have to place every checkout at <root>/<owner>/<name>, so a
repo living anywhere else silently lost its local panels. Four in this
fleet did.
* reading thousands of small files back through a macOS bind mount is slow
enough that the line counts needed a cache to stay affordable.
* eight of the Dockerfile's nineteen lines existed only to undo problems the
container created - installing git, and trusting a bind mount owned by a
uid the container does not have.
What changes
* collector/Dockerfile is gone, and so is the collector service.
* prometheus.yml scrapes host.docker.internal:9109. extra_hosts maps it on
Linux, where it is not built in.
* scripts/collector.sh runs it in the foreground - under launchd, in a
terminal, or under tmux. up.sh installs and loads a launchd agent on
macOS and says what to do elsewhere; down.sh unloads it.
* gen-repos.py no longer writes a compose override, because there is no
container to mount anything into. It prints JQ_REPOS and JQ_REPO_PATHS,
and collector.sh runs it at every launch - so repos.yml is the only place
the fleet and the layout are written down, with no generated file in
between to go stale.
Two things this turned up, both fixed here
* A launchd agent inherits /usr/bin:/bin:/usr/sbin:/sbin and nothing else,
so uv under /opt/homebrew was invisible and the agent failed on every
launch. KeepAlive then respawned it in a tight loop - fourteen failures
before it was noticed. up.sh now pins uv's directory into the plist, and
ThrottleInterval turns any future misconfiguration into a slow retry with
a readable log rather than thousands of lines a minute.
* Moving the collector changed its `instance` label from collector:9109 to
the new scrape address, which forked every series in two and showed each
repo on the board twice until the old ones went stale. There is exactly
one collector, so the label carries no information; prometheus.yml now
relabels it to a constant and the history stays continuous across any
future move.
The trade, stated in scripts/collector.sh and docs/operations.md: inside the
container the collector could only see the checkouts mounted into it, so an
unlisted repo was invisible rather than merely filtered out. It now runs as you
and could read anything you can. It still never writes - every git call is
read-only and passes --no-optional-locks - but that is now a property of the
code rather than something the sandbox enforces.
Verified end to end on the real fleet: launchd agent running, 24 of 24
checkouts reporting, Prometheus target up at host.docker.internal:9109, one
instance label, no duplicate series, Grafana serving.
Run the collector on the host, not in a container
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
localgit.scanderived every path as<JQ_REPO_ROOT>/<owner>/<name>. Inside the container that is always right — the generated bind mounts put each checkout at exactly that path, sorepos.ymlcan say anything and the mount normalises it. Outside the container it is often wrong, and silently so.Four repos in the fleet this was built against are laid out like this:
No amount of joining owner to name produces that path. Those repos dropped off the working-copy panels while staying on the GitHub ones — present on the board, quietly missing half their columns, nothing saying why. Running the collector on the host reported 20 of 24 checkouts.
The fix
JQ_REPO_PATHScarries the real path per repo asowner/name=pathpairs, with<repo_root>/<owner>/<name>kept as the fallback.gen-repos.py --envemits it fromrepos.ymlalongsideJQ_REPOS, so the layout is stated once in the file that already owns it rather than retyped.JQ_REPO_PATHS)JQ_REPO_ROOTempty, paths only)The container is genuinely untouched — same fallback, rebuilt and verified at 1,743 metric lines with a clean log.
Two ways this could have gone quiet, both closed
gen-repos.pyrefuses to emit one rather than print a line that would be misread as two repos.The origin check matters more now, not less, since the path is arbitrary: a path pointing at the wrong repo is still refused rather than filed under the listed repo's name. Test included.
A bad
JQ_REPO_ROOTalso no longer abandons the whole scan — it logs once and withdraws the fallback, leaving repos with explicit paths working.Checks
69 tests (10 new; one existing test updated because
--envnow prints two lines — that was pinning the old contract), ruff,check-dashboard.py,mkdocs build --strict, both compose stacks parse.