Run the collector on the host, not in a container - #23
Merged
tschm merged 1 commit intoAug 30, 2026
Conversation
Prometheus and Grafana stay in Docker - they are pinned images with
repo-tracked provisioning, which is what Docker is good at here. The collector
is not; it is a Python package that reads your working copies, and the
container was costing more than it gave:
* bind mounts have to place every checkout at <root>/<owner>/<name>, so a
repo living anywhere else silently lost its local panels. Four in this
fleet did.
* reading thousands of small files back through a macOS bind mount is slow
enough that the line counts needed a cache to stay affordable.
* eight of the Dockerfile's nineteen lines existed only to undo problems the
container created - installing git, and trusting a bind mount owned by a
uid the container does not have.
What changes
* collector/Dockerfile is gone, and so is the collector service.
* prometheus.yml scrapes host.docker.internal:9109. extra_hosts maps it on
Linux, where it is not built in.
* scripts/collector.sh runs it in the foreground - under launchd, in a
terminal, or under tmux. up.sh installs and loads a launchd agent on
macOS and says what to do elsewhere; down.sh unloads it.
* gen-repos.py no longer writes a compose override, because there is no
container to mount anything into. It prints JQ_REPOS and JQ_REPO_PATHS,
and collector.sh runs it at every launch - so repos.yml is the only place
the fleet and the layout are written down, with no generated file in
between to go stale.
Two things this turned up, both fixed here
* A launchd agent inherits /usr/bin:/bin:/usr/sbin:/sbin and nothing else,
so uv under /opt/homebrew was invisible and the agent failed on every
launch. KeepAlive then respawned it in a tight loop - fourteen failures
before it was noticed. up.sh now pins uv's directory into the plist, and
ThrottleInterval turns any future misconfiguration into a slow retry with
a readable log rather than thousands of lines a minute.
* Moving the collector changed its `instance` label from collector:9109 to
the new scrape address, which forked every series in two and showed each
repo on the board twice until the old ones went stale. There is exactly
one collector, so the label carries no information; prometheus.yml now
relabels it to a constant and the history stays continuous across any
future move.
The trade, stated in scripts/collector.sh and docs/operations.md: inside the
container the collector could only see the checkouts mounted into it, so an
unlisted repo was invisible rather than merely filtered out. It now runs as you
and could read anything you can. It still never writes - every git call is
read-only and passes --no-optional-locks - but that is now a property of the
code rather than something the sandbox enforces.
Verified end to end on the real fleet: launchd agent running, 24 of 24
checkouts reporting, Prometheus target up at host.docker.internal:9109, one
instance label, no duplicate series, Grafana serving.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prometheus and Grafana stay in Docker: pinned images with repo-tracked provisioning, which is what Docker is genuinely good at here. The collector is a different case — it's a Python package that reads your working copies, and the container was costing more than it gave:
<root>/<owner>/<name>, so a repo living anywhere else silently lost its local panels. Four in this fleet did.git, and trusting a bind mount owned by a uid the container doesn't have.What changes
collector/Dockerfilecollectorserviceprometheus.ymlhost.docker.internal:9109;extra_hostsmaps it on Linuxscripts/collector.shscripts/up.sh/down.shscripts/gen-repos.pyJQ_REPOSandJQ_REPO_PATHScollector.shrunsgen-repos.pyat every launch, sorepos.ymlis the only place the fleet and the layout are written down — no generated file in between to go stale.Two bugs this turned up, both fixed here
launchd gives an agent almost no PATH —
/usr/bin:/bin:/usr/sbin:/sbinand nothing else — souvunder/opt/homebrewwas invisible and the agent failed on every launch.KeepAlivethen respawned it in a tight loop: 14 failures before I caught it.up.shnow pinsuv's directory into the plist, andThrottleIntervalturns any future misconfiguration into a slow retry with a readable log instead of thousands of lines a minute.Moving the collector forked every series in two.
instancewent fromcollector:9109to the new scrape address, so each repo appeared on the board twice until the old series went stale — the "renaming a label starts a new series" trap the docs already record forrepo. There is exactly one collector, soinstancecarries no information;prometheus.ymlnow relabels it to a constant, and history stays continuous across any future move. Recorded indocs/dashboard.mdnext to the related traps.The trade, stated plainly
Inside the container the collector could only see the checkouts mounted into it, so an unlisted repo was invisible, not merely filtered out. It now runs as you and could read anything you can. It still never writes — every git call is read-only and passes
--no-optional-locks— but that is now a property of the code rather than something the sandbox enforces. Written down inscripts/collector.shanddocs/operations.mdrather than left implicit.Verified end to end on the real fleet
launchd agent running · 24/24 checkouts reporting · Prometheus target
upathost.docker.internal:9109· oneinstancelabel, zero duplicate series · Grafana serving. Plus 69 tests, ruff,check-dashboard.py,mkdocs build --strict, both compose stacks, and every shell script parsing — the last is a new CI step, since these scripts are now the entry points a user actually touches and nothing checked them before.