Skip to content

Run the collector on the host, not in a container - #23

Merged
tschm merged 1 commit into
fix/honour-real-checkout-pathsfrom
feat/collector-runs-natively
Aug 30, 2026
Merged

Run the collector on the host, not in a container#23
tschm merged 1 commit into
fix/honour-real-checkout-pathsfrom
feat/collector-runs-natively

Conversation

@tschm

@tschm tschm commented Aug 30, 2026

Copy link
Copy Markdown
Member

Stacked on #22 — base is fix/honour-real-checkout-paths, which this depends on for JQ_REPO_PATHS. Merge #22 first and this retargets to main on its own.

Prometheus and Grafana stay in Docker: pinned images with repo-tracked provisioning, which is what Docker is genuinely good at here. The collector is a different case — it's a Python package that reads your working copies, and the container was costing more than it gave:

  • Bind mounts must place every checkout at <root>/<owner>/<name>, so a repo living anywhere else silently lost its local panels. Four in this fleet did.
  • Reading thousands of small files back through a macOS bind mount is slow enough that the line counts needed a caching layer to stay affordable.
  • 8 of the Dockerfile's 19 lines existed only to undo problems the container created — installing git, and trusting a bind mount owned by a uid the container doesn't have.

What changes

collector/Dockerfile deleted, along with the collector service
prometheus.yml scrapes host.docker.internal:9109; extra_hosts maps it on Linux
scripts/collector.sh new — runs it in the foreground: launchd, a terminal, or tmux
scripts/up.sh / down.sh install/load and unload a launchd agent on macOS; say what to do elsewhere
scripts/gen-repos.py no longer writes a compose override — there's nothing to mount into. Prints JQ_REPOS and JQ_REPO_PATHS

collector.sh runs gen-repos.py at every launch, so repos.yml is the only place the fleet and the layout are written down — no generated file in between to go stale.

Two bugs this turned up, both fixed here

launchd gives an agent almost no PATH/usr/bin:/bin:/usr/sbin:/sbin and nothing else — so uv under /opt/homebrew was invisible and the agent failed on every launch. KeepAlive then respawned it in a tight loop: 14 failures before I caught it. up.sh now pins uv's directory into the plist, and ThrottleInterval turns any future misconfiguration into a slow retry with a readable log instead of thousands of lines a minute.

Moving the collector forked every series in two. instance went from collector:9109 to the new scrape address, so each repo appeared on the board twice until the old series went stale — the "renaming a label starts a new series" trap the docs already record for repo. There is exactly one collector, so instance carries no information; prometheus.yml now relabels it to a constant, and history stays continuous across any future move. Recorded in docs/dashboard.md next to the related traps.

The trade, stated plainly

Inside the container the collector could only see the checkouts mounted into it, so an unlisted repo was invisible, not merely filtered out. It now runs as you and could read anything you can. It still never writes — every git call is read-only and passes --no-optional-locks — but that is now a property of the code rather than something the sandbox enforces. Written down in scripts/collector.sh and docs/operations.md rather than left implicit.

Verified end to end on the real fleet

launchd agent running · 24/24 checkouts reporting · Prometheus target up at host.docker.internal:9109 · one instance label, zero duplicate series · Grafana serving. Plus 69 tests, ruff, check-dashboard.py, mkdocs build --strict, both compose stacks, and every shell script parsing — the last is a new CI step, since these scripts are now the entry points a user actually touches and nothing checked them before.

Prometheus and Grafana stay in Docker - they are pinned images with
repo-tracked provisioning, which is what Docker is good at here. The collector
is not; it is a Python package that reads your working copies, and the
container was costing more than it gave:

  * bind mounts have to place every checkout at <root>/<owner>/<name>, so a
    repo living anywhere else silently lost its local panels. Four in this
    fleet did.
  * reading thousands of small files back through a macOS bind mount is slow
    enough that the line counts needed a cache to stay affordable.
  * eight of the Dockerfile's nineteen lines existed only to undo problems the
    container created - installing git, and trusting a bind mount owned by a
    uid the container does not have.

What changes

  * collector/Dockerfile is gone, and so is the collector service.
  * prometheus.yml scrapes host.docker.internal:9109. extra_hosts maps it on
    Linux, where it is not built in.
  * scripts/collector.sh runs it in the foreground - under launchd, in a
    terminal, or under tmux. up.sh installs and loads a launchd agent on
    macOS and says what to do elsewhere; down.sh unloads it.
  * gen-repos.py no longer writes a compose override, because there is no
    container to mount anything into. It prints JQ_REPOS and JQ_REPO_PATHS,
    and collector.sh runs it at every launch - so repos.yml is the only place
    the fleet and the layout are written down, with no generated file in
    between to go stale.

Two things this turned up, both fixed here

  * A launchd agent inherits /usr/bin:/bin:/usr/sbin:/sbin and nothing else,
    so uv under /opt/homebrew was invisible and the agent failed on every
    launch. KeepAlive then respawned it in a tight loop - fourteen failures
    before it was noticed. up.sh now pins uv's directory into the plist, and
    ThrottleInterval turns any future misconfiguration into a slow retry with
    a readable log rather than thousands of lines a minute.
  * Moving the collector changed its `instance` label from collector:9109 to
    the new scrape address, which forked every series in two and showed each
    repo on the board twice until the old ones went stale. There is exactly
    one collector, so the label carries no information; prometheus.yml now
    relabels it to a constant and the history stays continuous across any
    future move.

The trade, stated in scripts/collector.sh and docs/operations.md: inside the
container the collector could only see the checkouts mounted into it, so an
unlisted repo was invisible rather than merely filtered out. It now runs as you
and could read anything you can. It still never writes - every git call is
read-only and passes --no-optional-locks - but that is now a property of the
code rather than something the sandbox enforces.

Verified end to end on the real fleet: launchd agent running, 24 of 24
checkouts reporting, Prometheus target up at host.docker.internal:9109, one
instance label, no duplicate series, Grafana serving.
@tschm
tschm merged commit e0e3b45 into fix/honour-real-checkout-paths Aug 30, 2026
5 checks passed
@tschm
tschm deleted the feat/collector-runs-natively branch August 30, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant