Package supply chain

Package supply-chain

This asks where software came from. It does not ask whether an implant is still running. pkg-audit can tell you that a package you installed was in the June 2026 AUR supply-chain wave; compromise-check tells you whether the systemd unit that wave dropped is still on the machine. Run both — they answer the two halves of the same incident.

Four commands touch package provenance/safety and are easy to confuse. Rule of thumb: -P is the broad, all-ecosystem one; -A/-p are AUR-only and each answers a different question.

Command	What it answers	Use it when	Output
fettle -P / pkg-audit	Across all ecosystems (AUR/APT/Flatpak/Snap): where did my installed software come from, and has it been tampered with?	you want one whole-system supply-chain report	findings → ~/.fettle/reports/
fettle -A / aur-audit (arch)	AUR health census: age, votes, out-of-date, orphan, recently-changed, maintainer-change (re-adoption tell), reverse-dependents (NO-DEPENDENTS/NO-HARD-DEPS, LIB for unused libraries — nothing on the system needs it) + removal candidates	you want to vet how well-maintained your AUR pkgs are — and spot leftovers	table → ~/.fettle/reports/
fettle -p / aur-precheck (arch)	AUR pre-install / quick sweep: is this package (or every installed AUR pkg) risky right now — orphaned, out-of-date, stale, compromised name, malicious maintainer?	before building an AUR pkg (the yay hook), or a fast all-installed check	CRIT/WARN lines

pkg-audit runs each provider whose package manager is present and reports one normalized Finding format with one severity language:

• AUR (Arch): vanished from the AUR since the last run, orphan / out-of-date / stale / known-bad via AUR RPC + IOC feed.
• APT (Debian): third-party repos/PPAs, [trusted=yes], third-party-http, debsums file integrity.
• Flatpak: no longer offered by its remote, non-flathub origin, broad sandbox permissions (host/home filesystem, devices=all), http remotes.
• Snap: no longer in the Store, sideloaded / unverified publisher, classic/devmode confinement.
• VS Code / VSCodium extensions: no longer listed in the gallery the editor is actually wired to (read from its product.json, never inferred from the profile directory), registry install vs sideloaded .vsix.
• GitHub CLI extensions: origin repository no longer resolves (deleted, renamed, or made private — indistinguishable to an unauthenticated request), which repository each came from. "Withdrawn upstream" deserves its own note, because it is the strongest signal in this list and the least obvious. Removal is what a registry does to malware: it is what Arch did to firefox-patch-bin after the 2025 RAT, and to 1,579 packages in June

2026. So an app you still have installed that is no longer offered by the place it came from is worth a look — it may equally have been renamed or retired, which is why the finding says investigate rather than panic.

On the AUR this is a disappearance, not an absence. "Not in the AUR" is two very different situations wearing one label, and lumping them made the finding useless — on a real 79-package host it stood at 9 every run, mostly in-house packages that were never in the AUR at all, and a warning that is permanently on is one nobody reads. So fettle warns only when a package was there when it last looked and is gone now, using the same per-run snapshot that detects maintainer takeovers. The rest are listed quietly as not in the AUR, and never seen there — because "installed from somewhere else" and "deleted before fettle first ran" are genuinely indistinguishable, and it does not pretend otherwise. The vanished entry is kept for as long as the package is installed, so the finding does not evaporate on the next run.

And it must be asked of the right registry. VS Code and VSCodium can each be wired to either Microsoft's marketplace or Open VSX, and the profile directory does not tell you which — measured on a real Arch box, Code - OSS keeps its profile in .vscode-oss and is patched to use the marketplace. Inferring the registry from the directory name reported two extensions as withdrawn that were present in the gallery their editor actually uses; they had simply never been published to the other one. fettle reads extensionsGallery.serviceUrl from the editor's product.json, and when it cannot determine the gallery it skips the check and says so, because asking the wrong registry does not give a weaker answer — it gives a confident wrong one.

It is asked over the network, which is the trap: a store that is merely unreachable would otherwise make every app you own look withdrawn at once. A non-zero exit is therefore not enough — the tool has to say, in as many words, that it looked and the thing was not there. Anything else is reported as UNVERIFIABLE, once for the run: not checked, rather than checked and clean.

• DNF/YUM repos (RHEL family): gpgcheck=0 (signatures not verified), plain-http URLs, third-party repositories.
• Containers (docker/podman): images pulled by the mutable :latest tag, image age, registry provenance, dangling images.
• GNOME Shell extensions: no longer listed on extensions.gnome.org (asked only for hand-installed ones — a packaged extension may never have been on e.g.o at all), which extensions are attributable to a package vs dropped in by hand, and whether they're enabled.
• VS Code / VSCodium extensions: which came from the configured registry vs a sideloaded .vsix.
• GitHub CLI extensions: which GitHub repository each gh extension came from.

Except for the distro-native ones (AUR on Arch, APT on Debian), every provider runs on every distribution — flatpak, snap, containers and GNOME extensions install the same way anywhere. A provider whose tool isn't installed says so rather than staying silent, because "flatpak is clean" and "flatpak was never looked at" must not look identical. For ecosystems you knowingly don't use:

[supplychain]
skip_sources = ["snap", "flatpak"]      # never checked here, never mentioned

[supplychain.hosts.wopr]                # optional per-machine override
skip_sources = ["snap"]

(Host tables only matter for a config you sync between machines — fettle remote runs on the remote, which reads the remote's config.)

Each provider prints a coverage line so uneven depth is explicit — a real malware/IOC feed exists only for the AUR, and fettle never pretends otherwise.

On GNOME extensions: extension JavaScript runs inside the gnome-shell process itself, not a sandbox, so an enabled one can observe and drive your whole session. fettle answers the question it can answer well — attribution: an extension under /usr/share came from a package and is traceable; one in ~/.local/share/gnome-shell/extensions was hand-installed and nothing records its origin. Enabled-and-unattributed is the finding that matters. There is no IOC feed for extensions.gnome.org, so this says nothing about whether an extension's code is malicious.

Some software belongs to a person, not to the machine — and fettle asks as that person. A maintenance run is usually root, because most of it has to be. But three sources keep their data inside a user's home directory, and asking as root returns somebody else's answer:

source	where it really lives	as root you get
GNOME extensions	the running login session	an error (fixed v1.13.0)
rootless podman images	~/.local/share/containers/storage	root's store — usually empty, silently
--user flatpak apps	~/.local/share/flatpak	the system apps plus root's own

Since v1.15.0 each is asked as the invoking user. podman is asked twice — once as root, once as you — because running containers as root is perfectly normal on a server, so picking one store would simply move the blind spot to the other. Findings name their store (podman: vs podman(paulda):) and the examined line says where it looked.

docker is deliberately left alone. It is one system-wide daemon reached through a group-owned socket: root can always reach it, an ordinary user only if they are in that group. Dropping privileges for docker would break the audit on every host where that is not true.

In all three cases, a source that cannot be read reports could not look — never an empty list. That distinction is the entire point: podman's failure mode was returning nothing at all, which for weeks was indistinguishable from a machine with no containers.

Every provider says what it examined, whether or not it found anything. coverage above says what a provider can answer; each run also prints what it did:

[gnome]     24 extensions examined — all traceable to a package; 5 enabled
[vscode]    11 editor extensions examined — all installed from a marketplace
[snap]      nothing to examine — no snaps installed
[flatpak]   nothing to examine — no flatpak apps installed
[aur]       76 AUR packages examined — 30 finding(s) below

Four outcomes stay distinct, because collapsing any two of them is how an audit starts lying: not installed · installed but nothing to examine · examined N, all clean · could not look (an UNVERIFIABLE finding). Before v1.14.0 the middle two both rendered as silence, so a provider that cleared 24 extensions and one that never ran produced identical output. The counts are stored in the JSON report too — "no findings" is worth nothing unless the report also says what was looked at.

The enabled GNOME extensions are listed by name. Extension code runs inside the gnome-shell process with your full session privileges, so which ones are running is the fact worth glancing at — and on a machine where every extension is distro-packaged, that list was the one thing the audit never mentioned. It is body detail, not a finding: it does not count towards N supply-chain finding(s) and never puts a warn on a clean machine.

Extensions belong to a login session, not to the machine. gnome-extensions asks the GNOME session bus for its answer, so it can only be read as the logged-in user. That matters more than it sounds: run pkg-audit as part of fettle -a, and the process is already root — -a elevates once for the mutating actions, and every action after that runs as root, including the ones that never asked to. sudo's env_reset has by then discarded DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR, so the tool exits 2 having listed nothing.

Since v1.13.0 fettle drops back to the invoking user and restores their runtime directory for this one query (dropping privileges alone is not enough — sudo -u resets the environment a second time). If there is genuinely no session to ask — a headless host, a service account, nobody logged in — you get extensions were NOT audited and the reason, never a clean tick.

On editor extensions: these are unsandboxed Node running with your full user privileges — filesystem, shell, SSH keys — and they auto-update. fettle reads the editor's own extension index, the only local record of where each one came from, and flags the ones installed from a sideloaded .vsix: those bypassed the registry entirely, so no namespace or publisher check ever applied. VSCodium installs from Open VSX, whose namespace vetting is lighter than Microsoft's marketplace — worth knowing when an extension's publisher field names a major vendor. fettle does not try to verify that a publisher is who they claim: doing that reliably needs a curated known-good list per registry, which is a maintenance burden it won't take on.

On gh extensions: these install straight from an arbitrary GitHub repository with no registry, review or signing — and, the part usually missed, gh runs them with your authenticated session available, so an extension can act as you against everything your token reaches. fettle reports the origin repository of each one, reading the extension directory's own records (a binary extension's manifest.yml, or a source extension's git origin) rather than parsing gh extension list, whose output has no stable format. Extensions owned by cli/github are treated as first-party and not flagged.

On containers specifically: an image is pulled by name, and :latest is a mutable pointer — the bits behind it change without the name changing, so nothing records what actually ran. An image is also frozen at build time: unlike a distro package, no updater touches it, so every CVE published since its build date is still inside. That makes age the headline signal ([containers] max_age_days, default 90; ignore accepts name globs). fettle deliberately does not scan image contents for vulnerable packages — that is trivy's/grype's job. If the daemon can't be queried (it's stopped, or you're not in the docker group) that is reported as a finding rather than passing silently.

Refreshing images — fettle -C / container-update. The audit tells you an image is stale; this pulls it. It is opt-in (never in the default set) and needs no root. Nothing is pulled implicitly — each image is decided by:

[containers]
auto_update   = "ask"            # "ask" (default) | "always" | "never" — overrides both lists
never_update  = ["pyemba-*"]     # globs, matched against "repo:tag" and the bare repo
always_update = ["python"]

First match wins: auto_update → never_update → always_update → otherwise ask. An auto_update value that is none of the three is reported rather than ignored — auto_update = false reads as "never" but silently meant "ask". Under --yes (cron, fettle remote) the "ask" case is skipped, not auto-approved — an image you never explicitly opted into is never pulled without a human seeing the question. --dry-run prints the decision for every image and pulls nothing.

Two things are never offered:

• Images built here. A locally-built image has no registry to refresh from — docker pull cvetool:latest resolves to Docker Hub, which never served it. It is identified by having no RepoDigest (every pulled image has one) and reported as built here, not from a registry.
• Nothing, silently. If a runtime's daemon cannot be queried, the images behind it are not counted as considered, and the summary says which runtime went unread.

Both runtimes are used, not just the first. docker and podman keep separate image stores; when both are installed each image is labelled with the runtime it came from. The same is true of the audit half — a host with both used to have one of them audited while the report read as though it covered the machine.

fettle aur-precheck <pkg>… is the install-time helper: it prints machine-readable CRIT/WARN lines for the named packages and always exits 0. With no package named (fettle aur-precheck or fettle -p) it scans every installed AUR package instead — a quick safety sweep. Tunable via env vars (AUR_PRECHECK=false to disable, AUR_PRECHECK_MAX_AGE_DAYS, YAY_ALLOWLIST_FILE, …).

Exit status: 1 if anything was CRITICAL, else 0 — so fettle aur-precheck foo && yay -S foo means what it looks like it means. (The yay hook reads stdout and ignores the status, so this changes nothing for it.)

The allowlist is a trust boundary. An entry in ~/.config/yay/allowlist.txt suppresses a CRITICAL malware warning for the package it names, so fettle refuses a world-writable or foreign-owned allowlist — it says so and checks every package anyway, rather than honouring suppressions from a file anyone could have written.

If the IoC feeds cannot be read, it says so. An unreachable feed yields an empty malware list, which would otherwise mean a compromised package passes the gate in silence. That warning is emitted as a WARN line, so it reaches the yay hook at build time too.

The bundled yay hook (~/.config/yay/init.lua) calls it per package before a build — point its helper at fettle aur-precheck (it prefers fettle on PATH, falling back to the legacy aur-precheck.sh).

How it differs from the others: aur-precheck is the fast, self-contained, env-driven per-package gate (no config/TOML load, silent when clean — built for the hook); aur-audit is the detailed health report; pkg-audit is the cross-ecosystem umbrella that folds AUR health+IoC in alongside APT/Flatpak/Snap. (aur-ioc-scan was retired in v0.73.0 — pkg-audit runs everything it did.)

Pre-upgrade gate

Before yay -Sua builds anything, fettle -u / -a pre-checks the AUR packages it's about to upgrade against the IoC feeds — so a flagged package is caught before it's built/installed, not after. On any finding it shows it (a known-compromised name or malicious maintainer is loud; orphan/out-of-date/ stale are warnings) and prompts to continue or abort (default: abort). A clean set just prints a one-line "no indicators" and proceeds.

Because it runs in the update path, it applies to fettle remote <host> -u/-a too (the prompt comes over the ssh -t session). Under --yes a CRITICAL finding still aborts unattended — pass --force-aur to override; --no-aur-precheck (or aur_precheck_on_update = false in config) turns the gate off. It covers the yay -Qua upgrade set; --devel/-git rebuilds that don't bump a version stay covered by the yay hook and the post-update pkg-audit.

Package file integrity — -V / pkg-integrity

pkg-audit asks where your software came from. pkg-integrity asks a different question: has anything changed since it was installed? It re-reads every installed file and compares it against the manifest the package manager recorded at install time.

fettle -V                    # or --pkg-integrity, or `fettle pkg-integrity`

What it compares against, per distro. All three read a manifest that the package manager wrote into its own local database when the package was installed:

	source of truth	what is compared	fallback
Arch/Manjaro	pacman's MTREE, /var/lib/pacman/local/<pkg>/mtree	paccheck --sha256sum — full content hash	pacman -Qkk (file presence + properties only)
Debian/Ubuntu	the .md5sums dpkg installed, /var/lib/dpkg/info/<pkg>.md5sums	debsums — MD5 content hash	dpkg --verify
RHEL family	the file digests in the rpmdb	rpm -Va — size, mode, mtime, digest, owner, group, capabilities	(none needed — rpm is always present)

What that is worth, stated plainly. The manifest came from the same package, and anything able to rewrite a system file as root can usually rewrite the manifest too. So this is a tripwire, not a proof of authenticity — valuable because most intruders, and every botched upgrade, do not think to update the manifest. It is not a substitute for signature verification at install time, which is what pkg-audit covers.

Four outcomes, deliberately not summed together:

• Package Integrity — a packaged file whose contents no longer match, or that has gone missing. The finding.
• Permission drift (RHEL family) — mode, ownership or capabilities differ while the contents are untouched. True, and worth seeing — a world-writable binary matters — but it is not the same event as bytes changing, so it warns rather than alarms.
• Expected differences — files a tool rewrites after install, never a person: depmod's modules.dep/modules.alias index (once per installed kernel), plugin caches, ld.so.cache, mirror lists, anything under /run. These differ on every machine, so they carry no information. Counted, listed with -v. On RHEL this also covers rpm's own c/g/d markers — config files you edited, ghost files, documentation — and timestamp-only rows, since cp, rsync and every image builder rewrite an mtime without touching a byte.
• Not verified — files that could not be read, or (Debian) packages that ship no checksums at all. A gap in coverage, not a finding.

A verifier that failed is not a verifier that found nothing. This is the one thing to know about reading -V output, and it took a code review to close. None of these tools reports failure the way you would expect — measured, not read off a man page:

command	clean	found a discrepancy	could not run
paccheck	0	1	1
pacman -Qkk	0	1	1
debsums	0	2	255
dpkg --verify	0	0	0, printing nothing
rpm -Va	0	1	0, printing nothing

Two of the five cannot report failure at all, and paccheck uses one code for both outcomes. Until v1.11.0 fettle read none of them, so a verifier that died mid-run reported ✓ installed files match their packages and exited 0 — and on Arch, a verifier that printed an error message had that message counted as an altered file.

Now: debsums is judged by its exit status, paccheck and pacman -Qkk by whether a line actually names a file (a verdict quotes its path; error: failed to initialize alpm. does not), and dpkg --verify and rpm -Va are not trusted to be silent until their package database has been proven to list packages. If the verifier cannot be shown to have run, you get UNKNOWN — … packages were NOT verified and a non-zero exit, never a tick.

Why rpm -Va needs the extra split. It compares all nine attributes, where debsums and paccheck --sha256sum compare content alone. Grouping all nine into one count made every RHEL host red on first boot: measured across three freshly built cloud images, all 13 findings were an mtime or a directory mode and not one was a content change. An audit that is red on an untouched machine teaches you that red means nothing — and then the one digest mismatch that matters scrolls past with the rest.

Keeping them apart is the whole point. Measured on the author's workstation, the old combined output reported 82 "issues" — of which 65 were permission errors and 14 were depmod output. The three that remained were the ones worth looking at:

✗ Package Integrity: 3 file(s) differ from their package
    grub: '/etc/grub.d/30_os-prober' sha256sum mismatch
    networkmanager: '/usr/lib/NetworkManager/conf.d/20-connectivity.conf' …
    vscodium-bin: '/opt/vscodium-bin/resources/app/product.json' …
  Expected differences: 14 file(s) regenerated after install
! Not verified: 65 file(s) could not be read — re-run as root (`sudo fettle -V`)

It elevates itself. fettle -V prompts for sudo and re-runs, because unprivileged it cannot read a large share of the files it must hash — 65 of them above — and would otherwise print a confident answer about less of the system. The report still lands in your home, not root's. Under --dry-run it stays passwordless and tells you what it could not reach.

This makes it the mirror image of container-update, and the reason fettle tracks "read-only" and "needs no root" as two separate questions: reading can need privilege too. It is read-only — it changes nothing — and it still needs root.

Not in the default set: a full-content hash of every installed file takes ~35s on a desktop and longer on a server, and it is a check you run for a reason, not on a timer.

Before v0.72.0 this lived inside sys-audit as its packages category. It was a package question inside the firmware/boot scanner, and it made every -S run pay for the hashing pass.