Skip to content

repository integrity

Jump to bottom
Kadyapam edited this page Oct 4, 2026 · 1 revision

Repository integrity

ci/check-repo.py runs on every pull request to noetl/apt and reads the published tree in the working copy — pool/main/*.deb, dists/*/main/binary-*/Packages{,.gz}, and dists/*/Release. It is the only thing standing between a bad index and an apt-get failure on a user's machine, because the repository is served as static files and nothing validates it at request time.

What it checks

check fails when
1 checksum agreement a .deb's SHA256 differs from its Packages stanza
2 .gz currency Packages.gz does not decompress to the committed Packages
3 Release checksums a Release entry's size or hash does not match the file
4 index completeness a .deb in pool/ appears in no index
5 architecture coverage an architecture present in pool/ is indexed nowhere
6 architecture leakage a binary-<arch> index lists a stanza whose Architecture: is neither <arch> nor all
7 Release identity Origin, Label, Suite, Codename, Architectures or Components is missing
8 no self-entry a Release lists Release in its own checksum table

Everything fails the build. Checks 4–8 were reported rather than failed from the gate's introduction until 2026-10-04, because the release pipeline could not satisfy them; see Why checks 4–8 were advisory first.

Version is deliberately not required

Check 7 does not require a Version field. It is optional in a Debian Release file and apt-ftparchive does not emit one here. Suite and Codename are the fields apt actually matches a sources.list entry against.

Requiring a field that correct output lacks is how a gate earns a permanent || true.

Every check prints its population

── index completeness
   .debs in pool: 26   listed in an index: 26   orphans: 0 of 26
   architectures in pool: [amd64, arm64]   indexed: [amd64, arm64]

0 failures is also what a check that examined nothing looks like, so each section prints the population it measured and fails below a floor. This is not hypothetical for this repository: a check run over an empty pool/ otherwise reports exactly the same clean result as a healthy repository.

Why checks 4–8 were advisory first

The gate landed with real defects it could not demand be fixed:

  • 22 of 23 .debs were in no index. dpkg-scanpackages ran without --multiversion, so only the newest version of each package was indexed and apt-get install noetl=<older> could not resolve.
  • arm64 was in pool/ but indexed nowhere. The index path was hardcoded to binary-amd64.
  • Release carried only Date, and listed itself at 38 bytes against a real 1792 — apt-ftparchive release D > D/Release scans the directory it is writing into, so the redirect creates the file before the scan reads it.

The blocker was a product question rather than a technical one: the newest arm64 .deb was 2.8.7 while amd64 was at 5.0.1, so simply indexing what was there would have advertised arm64 as supported at a version years behind.

noetl/cli#93 resolved it by building a current arm64 .deb every release — build-deb is a two-entry matrix over (ubuntu-latest, amd64) and (ubuntu-24.04-arm, arm64), not a cross-compile. Measured on the live published repository afterwards:

.deb in no index        22 of 23  ->  0 of 26
architectures indexed   [amd64]   ->  [amd64, arm64]
Release header fields    1 (Date) ->  8
Release self-entry      present   ->  absent, both suites
newest arm64             2.8.7    ->  5.0.3

Nothing was removed. The older arm64 files remain installable as the genuinely older releases they are, which is what --multiversion exists for.

Check 6 is the one nobody had reported

Without --arch, dpkg-scanpackages writes every .deb it finds into whichever index it is told to produce — the Architecture: field is ignored. That was harmless while only amd64 shipped, and it would have started serving arm64 binaries to amd64 clients the moment arm64 existed. It was added as a check in the same change that promoted the others.

How the indexes are generated

In noetl/cli's release.yml, after the matrix .debs are downloaded into pool/main/:

ARCHES="$(find pool/main -maxdepth 1 -name '*.deb' \
  | sed -nE 's/.*_([a-z0-9]+)\.deb$/\1/p' | sort -u | tr '\n' ' ')"

for ARCH in ${ARCHES}; do
  DEST="dists/${DIST}/main/binary-${ARCH}"
  dpkg-scanpackages --multiversion --arch "${ARCH}" pool/main /dev/null > "${DEST}/Packages"
  gzip -9c "${DEST}/Packages" > "${DEST}/Packages.gz"
done

rm -f "dists/${DIST}/Release"
apt-ftparchive -o APT::FTPArchive::Release::Origin=NoETL ... \
  release "dists/${DIST}" > "${RUNNER_TEMP}/Release.${DIST}"
mv "${RUNNER_TEMP}/Release.${DIST}" "dists/${DIST}/Release"

Three details carry the fixes, and each is easy to undo by accident:

  • --multiversion keeps every version in the index, not just the newest.
  • --arch "${ARCH}" is what makes check 6 pass.
  • writing Release to $RUNNER_TEMP and mving it in is what keeps Release out of its own checksum table. A redirect straight into the directory reintroduces the self-entry.

sed -nE … p rather than sed -E s/// matters too: the substitution form passes non-matching filenames through unchanged, which turns a stray file in pool/ into an "architecture" named after its own path.

Proving a check can fail

Each check is injected one at a time against the real published tree before being trusted:

drop all but one stanza from an index   -> exit 1
put an arm64 stanza in the amd64 index  -> exit 1
remove Codename from Release            -> exit 1
re-introduce the Release self-entry     -> exit 1
all restored                            -> exit 0

A check that has never produced a failure is indistinguishable from a check that cannot produce one. Add the positive control in the same change as the check.

Related

Clone this wiki locally