fix(ci): rebuild backends when shared build inputs change

[localai-bot]

The incident

PR #10946 fixed a real bug: backend images shipped a partial 4-of-8 cuDNN library set that silently mixed versions with the venv's pip cuDNN, producing CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time on ~10 of 11 backends on an L4T fleet.

It merged 2026-07-19 07:48 UTC. The weekly full-matrix cron (0 6 * * 0) had fired at 06:00 the same day, 1h48m earlier. Every build since was path-filtered.

No backend image contains that fix, and none would until the 07-26 cron. CI was green throughout. The fix looked merged and shipped; it was neither. It was only caught because someone went looking for the rebuilt image to deploy it.

Cause

emitFilteredMatrix filtered with changedFiles.some(f => f.startsWith(backendPath)), where backendPath is the backend's own directory. A file only triggered a rebuild if it lived under that backend's tree.

scripts/build/package-gpu-libs.sh is COPY'd and executed by backend/Dockerfile.python for every Python backend image, but lives under scripts/ — so it matched nothing and produced an empty matrix.

The failure mode is invisible: an empty matrix is not an error.

The same hole applied to backend/Dockerfile.* itself. A change to backend/Dockerfile.llama-cpp triggered nothing, because the Dockerfile does not live under backend/cpp/llama-cpp/. That one was not in the original report and was found while deriving the path set.

Fix

A SHARED_BUILD_INPUTS table maps each shared path to the narrowest set of matrix entries it can honestly invalidate, plus a generic rule for backend/Dockerfile.<x> — exact rather than heuristic, since every matrix entry already names its dockerfile:.

A full matrix is 417 Linux + 56 Darwin builds, so granularity matters. Measured against the real matrix:

Changed path	Rebuilds
backend/backend.proto	everything (all four languages compile or copy it)
backend/Dockerfile.<x>	the Linux entries whose dockerfile: names it
backend/python/common/	Python, Linux + Darwin
scripts/build/package-gpu-libs.sh	Python, Linux only — 176 of 417, not a full matrix
scripts/build/<x>-darwin.sh	the Darwin entries that build target routes to
.github/workflows/backend_build[_darwin].yml	everything on that OS
anything else under scripts/build/ (except *_test.sh)	everything

Each inclusion was verified against the Dockerfiles, the reusable workflows and the Makefile targets rather than assumed — e.g. package-gpu-libs.sh is Linux-only because python-darwin.sh never invokes it.

That last row is deliberate: a new shared script silently shipping to nothing is the exact failure being fixed here. Over-building is recoverable; under-building is invisible.

Deliberately excluded, to keep CI spend honest

• backend/index.yaml — grepped: zero references in any Dockerfile or in backend_build.yml. Gallery metadata read at runtime; never enters an image.
• .github/backend-matrix.yml — would fire all 417 builds on every new-backend PR, and those already trigger via their own directory. Residual gap: editing an existing entry's base-image will not rebuild it.
• backend/Dockerfile.base-grpc-builder — owned by base-images.yml, which has its own path filter; backend builds consume the result by mutable tag.
• Root Makefile — ~11% of recent commits; its backend-relevant edits arrive alongside the backend directory anyway.

Known remaining gap

Go backends genuinely link pkg/ (e.g. backend/go/ced/main.go imports pkg/grpc), so a core change can alter a Go backend binary without rebuilding it. Including pkg/ would fire a Go matrix on nearly every commit, so it is left for a follow-up rather than absorbed here.

Tests

The filtering logic moved to scripts/lib/backend-filter.mjs so it can be unit-tested without bun, js-yaml, or a GitHub API round-trip. 15 tests via node --test (zero dependencies), run from the existing build-scripts job in lint.yml.

Verified red-first against the pre-fix logic, extracted unchanged — 11 failed, 4 passed:

not ok 1 - a change to only package-gpu-libs.sh rebuilds every Python image
  error:  'expected a non-empty Linux matrix'
  actual: 0
ok   2 - docs- and README-only changes never produce a matrix
ok   3 - a backend's own directory still triggers only that backend
ok  14 - tests for the packaging scripts do not rebuild anything
ok  15 - turboquant still retriggers on llama-cpp source changes

The negative pins passed before the fix as well as after — which is what makes them meaningful, since they would have caught an over-broad fix that simply rebuilt everything.

Final: 15/15 pass. make test-build-scripts unchanged and still green (16 PASS lines, no regression). make lint: 0 issues.

Note for reviewers

.agents/adding-backends.md and .github/backend-matrix.yml both point contributors at scripts/changed-backends.js:inferBackendPath, which this change moves. Those docs describe backend registration as "the single most common omission", so a stale pointer there has real cost — hence the doc updates in this diff. Separable if you would rather keep it narrow.

---

🤖 Generated with Claude Code