fettle 1.19.0

[1.19.0] — hardening-audit learns about mandatory access control

Three versions since 1.16.0, and they are one piece of work. hardening-audit
answered six questions and said nothing about the mechanism that is supposed to contain a
compromised service. It now answers nine, including AppArmor on Debian and Ubuntu and
SELinux on the RHEL family.

version	what changed
1.17.0	-H elevates, because the policy is root-only. --user opts out.
1.18.0	AppArmor axis.
1.19.0	SELinux axis.

The two systems are not in the same state in the field

This is the part I did not expect, and it is why the two axes ask different questions.
Measured on five untuned installs, none of them tuned by hand:

host	mandatory access control	processes actually confined
Debian 13	AppArmor, 106 profiles loaded	0 of 19
Ubuntu 26.04	AppArmor, 173 profiles loaded	2 of 20
AlmaLinux 9.8	SELinux, enforcing	141 of 147
Rocky Linux 9.8	SELinux, enforcing	140 of 146
Fedora 44	SELinux, enforcing	150 of 154

A stock Debian 13 loads 106 AppArmor profiles and confines nothing that is running.
sshd, systemd-resolved, systemd-networkd, ModemManager and udisksd are all
unconfined. A stock AlmaLinux 9 confines everything except the login session.

So the AppArmor axis reports whether anything is confined, and the SELinux axis reports
what mode the machine is in and whether it agrees with itself.

AppArmor: enabled is not the same as confining

The three profile modes are never added together. enforce applies a policy, complain
logs and applies nothing, and unconfined applies nothing either. That last group is
between 74 and 79 profiles per host and is not a defect: distros ship them so applications
keep working after Ubuntu set kernel.apparmor_restrict_unprivileged_userns=1.
/etc/apparmor.d/brave says so itself, and Ubuntu's documentation calls such a profile a
trivial bypass rather than a restriction.

One finding, at Low: AppArmor is enabled and no running process is confined by it, naming
apparmor-profiles and apparmor-profiles-extra when they are absent. It fires on a
stock Debian 13, deliberately, because that is the real state and there is something
specific to do about it.

SELinux: what mode, and does the machine agree with itself

"Enabled or disabled" has five answers on RHEL 9. The one worth building for is that Red
Hat deprecated SELINUX=disabled in /etc/selinux/config: the system then starts with
SELinux enabled and no policy loaded, and selinux=0 on the kernel command line is
the documented way to actually disable it. An admin who edited that file believes SELinux
is off. It is on, with no policy, and both the file and getenforce agree with the
mistaken belief.

Four findings, each with a measured floor of zero, covering not-enforcing, runtime
disagreeing with the configured mode, that RHEL 9 config trap, and booleans that differ
from the policy default.

Four rules were dropped because the numbers said so

• AppArmor profile files on disk that are not loaded. Filenames and profile names are
  different namespaces. 151 files produce 168 profiles, 13 of them children like
  zgrep//sed, and only 106 filenames match a profile name. The naive difference reports
  45 non-problems.
• AppArmor complain mode. The floor swings from 2 to 23 across three ordinary hosts, 16
  of Debian's 23 come from sbuild alone, and the apparmor package owns every one of
  those files.
• A curated list of dangerous SELinux booleans. Four high-risk-sounding booleans are
  on by default on every host tested (selinuxuser_execstack, unconfined_login,
  httpd_enable_cgi, nfs_export_all_rw), and two more on Fedora.
• Flagging SELinux booleans that are off. httpd_can_network_connect and
  samba_export_all_rw are off everywhere and are exactly what you enable to run a
  reverse proxy or a Samba share, so flagging them reports what the machine needs in order
  to work.

What survives measurement is narrow and quiet. An AppArmor process running unconfined
while a profile exists for that exact executable measured 0 on all three hosts, and
SELinux booleans differing from the policy default measured 0 on all three.

-H needs root now

The AppArmor policy is root-only. Unprivileged, aa-status prints "You do not have enough
privilege to read the profile set" and exits 0, and
/sys/kernel/security/apparmor/profiles is mode 0444 and still returns EACCES because
securityfs enforces past the mode bits. So the unprivileged view is the reassuring half of
the answer.

--user keeps the old behaviour, applies to read-only audits only, and refuses when
combined with an action that changes the system. Verified on Debian 13: identical findings
either way, 28 s as root against 27 s with --user.

SELinux is the opposite and needs almost no privilege, so that axis gives a full answer
under --user apart from the boolean comparison, which needs semanage from
policycoreutils-python-utils, a package stock AlmaLinux 9 does not install.

Verified before release

Nine live runs across five guests, plus a full lab matrix of 13 actions across 6 guests
reverting between runs: 71 pass, 7 issue, 0 FAIL, 0 skip. Every issue was examined and
none is a defect. Five are aur-audit correctly declining on non-Arch targets, and two are
Ubuntu hitting the apt lists lock held by apt-daily, which fettle reported truthfully and
which pass on a re-run.

One path is not verified on a machine, and this says so rather than implying otherwise:
the RHEL 9 SELINUX=disabled state. Creating it needs a config edit and a reboot, and so
does recovering, so it rests on unit tests and Red Hat's documentation.

SELinux is in far better shape than AppArmor, so this axis asks a different question.
Measured on three untuned guests:

host	mode	confined processes
AlmaLinux 9.8	enforcing	141 of 147
Rocky Linux 9.8	enforcing	140 of 146
Fedora 44	enforcing	150 of 154

No daemon runs unconfined on a stock EL9 install. The unconfined processes are the login
session and nothing else. Coverage is not the question here; mode and configuration drift
are.

Enabled or disabled has five answers on RHEL 9

state	how it is told apart
enforcing	/sys/fs/selinux/enforce is 1
permissive	enforce is 0, policy loaded
disabled at boot	selinux=0 on the kernel command line
SELINUX=disabled on RHEL 9+	the config file, plus the release
runtime and config disagree	getenforce against SELINUX= in the config

The fourth is why this axis is worth having. Red Hat deprecated disabling SELinux through
SELINUX=disabled: on RHEL 9 the system then starts with SELinux enabled and no policy
loaded, and selinux=0 on the kernel command line is the documented way to actually
disable it. An admin who edited that file believes SELinux is off. It is on, with no
policy, and both the config file and getenforce agree with the mistaken belief.

Four findings, each with a measured floor of zero

selinux-not-enforcing at Medium covers permissive, disabled at boot and no-policy, with
the wording naming which. selinux-mode-mismatch at Low catches a setenforce that will
not survive a reboot. selinux-config-disabled-el9 at Low catches the state above.
selinux-boolean-changed at Low lists booleans that differ from the policy default, which
measured 0 differences on all three hosts.

The boolean rules that measurement rejected

Of 17 candidate high-risk booleans, four are on by default on every host tested
(selinuxuser_execstack, unconfined_login, httpd_enable_cgi, nfs_export_all_rw) and
two more on Fedora. A curated "dangerous booleans" list would fire four times on a stock
EL9 host.

The ones that are off are no better. httpd_can_network_connect and samba_export_all_rw
are legitimately switched on by anyone running a reverse proxy or a Samba share, and
flagging those reports what the machine needs in order to work. The kernel-sysctl axis
already refuses that on principle. Booleans are counted, never judged individually.

semanage boolean -l is the exception and it is not always available: it needs root and
policycoreutils-python-utils, which a stock AlmaLinux 9 does not install. Its absence is
reported as blindness, never as "no booleans were changed".

Verified on five hosts

host	result
AlmaLinux 9.8, permissive (temporarily)	1 Medium, 1 Low — permissive, and running mode disagreeing with the config
AlmaLinux 9.8, enforcing	nothing to report, plus semanage blindness
Rocky 9.8	nothing to report, semanage present so no blindness
Fedora 44	nothing to report, 150 of 154 confined, policy version 35
Debian 13	not applicable, covered by the apparmor axis

The permissive run was done by temporarily setting setenforce 0 on the AlmaLinux guest
and reverting in the same operation. The RHEL 9 config-file state was not created on a
machine, because recovering from it needs a config edit and a reboot; that finding is
covered by unit tests against synthetic inputs and by Red Hat's documentation.

The AppArmor axis's not-applicable wording now points at this axis instead of saying
SELinux "is not used here".

---

What to download

file	for
fettle_<version>_all.deb	Debian, Ubuntu, and derivatives
fettle-<version>-1.noarch.rpm	RHEL, Rocky, AlmaLinux, Fedora
fettle-<version>-1-any.pkg.tar.zst	Arch, Manjaro, EndeavourOS
fettle-<version>-zipapp.tar.gz / .zip	anything else — runs on any Linux with python 3.11+
fettle.pyz	the zipapp on its own, if you do not want the archive

fettle is pure standard library: it needs python 3.11 or newer and nothing else.
On RHEL/Rocky/Alma...

fettle 1.16.0

[1.16.0] — the audit stops looking in the wrong place, and stops hanging

Four fixes since 1.12.0, and three of them are the same bug wearing different clothes:
software that belongs to a person was being checked as though it belonged to the
machine. A maintenance run is usually root, because most of it has to be — and at that
moment the personal half of the host goes quiet.

Every one was found by reading real run logs on a real workstation, not by re-reading code.

version	what had been happening
1.13.0	GNOME extensions reported NOT audited for a week straight, on a desktop with 24 of them working. gnome-extensions asks the session bus, and sudo had discarded the address.
1.14.0	Four of six audit providers printed their coverage sentence and nothing else — "examined 24 and cleared them all" looked exactly like "never ran".
1.15.0	11 rootless podman images had never once been audited. Not an error: as root, podman answers from root's store, which was empty. No report fettle had ever written mentioned podman. --user flatpak apps were invisible the same way.
1.16.0	A stopped snapd made fettle hang forever — -c, -P and -a, --dry-run included.

The distinction that hid three of them

Being in fettle's no-root set does not mean an action executes unprivileged. It means
it does not elevate on its own. Run it inside -a and the process has already re-exec'd
under sudo for the mutating half, so everything after that is root — including
pkg-audit, which reads three things that live in your account.

Those three are now asked as the invoking user regardless of how the run started. podman is
asked twice, once per store, because running containers as root is ordinary on a server
and picking one identity would simply move the blind spot. docker is deliberately left
alone: it is one system daemon behind a group-owned socket that root can always reach and an
ordinary user often cannot, so dropping privileges there would break it.

The measure that matters: an elevated fettle -P and an unprivileged one now return the
same findings. The audit describes the machine, not the way it was launched.

What you will notice

Every provider now states what it looked at, so a clean result stops being silent:

[gnome]     24 extensions examined — all traceable to a package; 5 enabled
              enabled and running inside gnome-shell (5 of 24): …
[container] 22 container images examined — across docker, podman, podman(paulda)
[snap]      nothing to examine — no snaps installed

Four outcomes stay distinct, because collapsing any two is how an audit starts lying: not
installed · installed but nothing to examine · examined N, all clean · could not
look.

Upgrading from 1.12.0 is a drop-in replacement — no configuration changes, no report
format changes. You may see findings you have not seen before, on a machine that has not
changed: those were always there.

Honest gaps

• The flatpak fix is unverified end to end. The host it was written on has no flatpak
  apps to observe the difference against. The code path is identical to podman's, which
  is verified — but identical-looking is not verified.
• The snap probe's healthy-path latency is unmeasured; its timeouts are conservative
  choices, not fitted ones.

---

The snapd hang in detail

Every bug fixed this week was fettle giving a wrong answer. This one made fettle
stop responding entirely, and it was live on the reporting host.

Found by accident: the test suite stopped returning while verifying an unrelated fix.
A stack dump rather than a guess showed a real snap command blocked in
subprocess.communicate:

subprocess.run → command.run → _prune_disabled_snaps → clean_caches → _clean

Reproduced outside fettle entirely, on a host where snapd had been deliberately
disabled:

/usr/bin/snap	present, so fettle uses it
snapd.service / snapd.socket	inactive (dead), service disabled
snap list, snap list --all, snap version	never return

Not an exotic configuration. On Arch and Manjaro snapd ships preset: disabled, so
anyone who installs it and never runs systemctl enable --now snapd.socket has a
permanently hanging snap binary — and had a permanently hanging fettle. -c, -P and
-a all wedged, --dry-run included, because the snap inventory is a read-only query
and read-only queries deliberately bypass the dry-run gate.

The obvious probe is wrong, and was measured before being built on

/run/snapd.socket still exists while snapd is disabled — a stale file left behind
from the last time the service ran. "Does the socket exist?" therefore answers yes on
exactly the host where snap does not work. A plausible-sounding fix that does nothing.

Two parts

command.run gained an opt-in timeout=. There is no default and that is
deliberate: pacman -Syu legitimately runs for twenty minutes and rpm -Va for several,
so a blanket limit would kill the very commands fettle exists to run. A timeout returns
Proc(124) — the status GNU timeout uses, so callers match a convention rather than an
invented number — keeps whatever partial output the tool managed first, and never raises.

A cached util.snap_ready() probe asks snap version under a short clock, once per
run. Every snap call site gates on it:

action	before	after
clean	hung forever	skips the prune, warns why
pkg-audit	hung forever	UNVERIFIABLE — snaps were NOT audited
update	hung forever	skips the refresh, warns why

examined deliberately stays None for the audit: "could not look" is not "examined
zero", and recording it as an empty examination would put this host in the same bucket
as one with no snaps at all — the exact false clean this model exists to prevent.

Measured after

fettle -c --dry-run completes in 5.2 seconds where it previously never returned.
fettle -P reports snaps were NOT audited — distinct from the nothing to examine — no
snaps installed it correctly printed that same morning, while snapd was still alive.

Honest gap: the healthy-path latency is unmeasured, because no working snapd was
available to time against. 5s (probe) and 30s (inventory) are conservative choices, not
fitted ones. Erring short reports "could not look", which is the safe direction; erring
long is what the old behaviour did, forever.

A test-isolation lesson came with it

The probe caches per process — right in production, where a wedged host would otherwise
pay the timeout at three separate call sites. It also leaks between tests: one test that
resolved it to False left every later snap test quietly skipping its own subject,
passing in isolation and failing in the suite. Reset by an autouse fixture in
conftest.py. Twenty-two test files also had their command.run fakes widened to accept
the new keyword — kept explicit rather than **kwargs, because a fake that silently
swallows new arguments cannot catch a call site that forgot to pass one.

---

What to download

file	for
fettle_<version>_all.deb	Debian, Ubuntu, and derivatives
fettle-<version>-1.noarch.rpm	RHEL, Rocky, AlmaLinux, Fedora
fettle-<version>-1-any.pkg.tar.zst	Arch, Manjaro, EndeavourOS
fettle-<version>-zipapp.tar.gz / .zip	anything else — runs on any Linux with python 3.11+
fettle.pyz	the zipapp on its own, if you do not want the archive

fettle is pure standard library: it needs python 3.11 or newer and nothing else.
On RHEL/Rocky/Alma 9 and Ubuntu 22.04 the system python3 is older than that, so the
packages pull in a suitable interpreter and the launcher finds it for you.

Verifying what you downloaded

sha256sum -c SHA256SUMS --ignore-missing

fettle 1.12.0

[1.12.0] — first release since 1.0.0: a sixth feature family, and five false-assurance bugs closed

This is the first tagged release since 1.0.0 (2026-08-10) and carries twelve versions of
work. Two things happened in it.

compromise-check (-M) — the sixth feature family

A read-only host triage pass that asks "is something running here that nobody
installed?" — four groups, and it never emits a fix, only what to investigate:

• persistence — systemd units, timers, cron and at jobs (every user's, not just
  root's) that no package owns
• boot — bootloader config, initramfs and ESP tampering
• kernel — /etc/ld.so.preload, the eBPF surface, loaded modules reconciled against
  the taint flags
• processes — running executables with no package behind them, and hidden PIDs

It is opt-in, needs root, self-elevates, and runs last under --everything. Every
threshold in it was calibrated by measuring the false-positive floor on real machines
first: six rules that sounded right were rejected because they were not (one of them worth
6,272 hits on a healthy box). It renders as one ranked table, it reports what it could not
look at as loudly as what it found, and it is on the HTML dashboard and in the exit
status. Versions 1.1.0 → 1.7.4.

Five bugs from an external code review — all of them false assurance

A code review on 2026-08-12 reopened four features that the QA pass had already swept, and
found the same defect class in each: reading a tool's silence, or its exit status, as good
news. Individually:

version	what it stopped doing
1.8.0	deleting /var/lib/pacman/db.lck on every clean — including while a transaction held it, which is the documented way to corrupt a package database
1.9.0	running yay -Sua, flatpak and snap after the system upgrade had already failed
1.10.0	upgrading Debian from package lists it could not refresh, and reporting success
1.11.0	reporting installed files match their packages when the verifier had died, and inventing findings out of its error messages
1.12.0	keeping a finding you had already fixed on the dashboard, and giving a green OK to hosts where no audit had ever run

The measurements behind them are worth stating, because none of these tools reports failure
the way you would expect: apt-get update exits 0 with every repository unreachable,
dpkg --verify and rpm -Va exit 0 printing nothing when they cannot read their
database, and paccheck returns 1 for "found a mismatch" and "cannot open the database"
alike. Each fix is now driven by a measured exit code or a probe, never an assumption.

Every one was reproduced before it was fixed and re-verified in a container afterwards; the
dashboard fix was verified by rebuilding the author's real 17-host dashboard and diffing it
card by card. Full detail per version below.

The rest

--quiet reaches the table-rendering audits (1.7.0), the compromise-check QA sweep and
its shared table renderer (1.7.0), a Python 3.11–3.13 permission bug that never appeared on
the author's 3.14 box (1.3.1), and documentation split between the README and the wiki
(1.0.1–1.0.2, 1.7.2).

Upgrading from 1.0.0 is a drop-in replacement — no configuration changes, no report
format changes. compromise-check is opt-in, so nothing new runs unless you ask for it.

---

The dashboard fix in detail (H-12)

Fifth fix from the 2026-08-12 code review (H-12). Two bugs in _host_problems(), both of
which let the fleet page describe a machine as something it no longer was.

A resolved finding stayed on the card forever

The card shows the newest report per tool — but clean reports were filtered out before
that choice was made:

for e in host["reports"]:
    if _is_empty(e):
        continue                  # clean reports discarded here
    if e.get("timestamp", "") >= newest.get(t, {}).get("timestamp", ""):
        newest[t] = e             # newest chosen from what survived

So "newest" meant the newest report that found something. Fix a finding, run the audit
again, and the clean result was thrown away in favour of the old complaint.

Measured on the real corpus rather than a fixture. ec1 was showing 47 packages with a known CVE (17 High). Its advisory-check history: 47 findings on 24 July, then 0 on 30
July and 0 on 6 August. The CVEs had been cleared for thirteen days and the dashboard was
still reporting them as current.

Selection now happens first and emptiness is interpreted afterwards.

Run logs alone earned an OK security verdict

Host freshness was computed from reports + logs. A run-log is what fettle -u writes,
and -u audits nothing — so a host that only ever updated had a fresh timestamp, no
findings, and a green OK on a security dashboard for a machine where no check had ever
run. The fixture said it in one line: OK · no reports · latest: <today>.

Audit freshness now comes from reports alone, and a host with none says
"no audit has run on this host — only run logs, which check nothing". Empty reports
still count as coverage, because a clean audit is an audit.

The staleness chip is reworded from "has not reported in N days" to "no audit in N
days" — the card's latest: line still counts run-log activity, and the two must not
appear to contradict each other. On the real corpus this made four hosts' ages larger and
more honest: fettle-fedora and fettle-ubuntu went from 11 days to 16, because a fresh
run-log had been masking a stale audit.

Verification

Four tests, three proved to fail against the old code; the fourth is the guard that a
newer finding still wins over an older clean run, since order is what matters here, not
emptiness.

Then the whole real dashboard was rebuilt before and after and diffed chip by chip across
17 hosts. Exactly two cards changed — ec1 lost the resolved CVE chip, bifrost-lab
gained the no-audit line. Nothing else moved, which is the result that mattered: a fix that
silently dropped a current finding would be worse than the bug it replaced.

---

What to download

file	for
fettle_<version>_all.deb	Debian, Ubuntu, and derivatives
fettle-<version>-1.noarch.rpm	RHEL, Rocky, AlmaLinux, Fedora
fettle-<version>-1-any.pkg.tar.zst	Arch, Manjaro, EndeavourOS
fettle-<version>-zipapp.tar.gz / .zip	anything else — runs on any Linux with python 3.11+
fettle.pyz	the zipapp on its own, if you do not want the archive

fettle is pure standard library: it needs python 3.11 or newer and nothing else.
On RHEL/Rocky/Alma 9 and Ubuntu 22.04 the system python3 is older than that, so the
packages pull in a suitable interpreter and the launcher finds it for you.

Verifying what you downloaded

sha256sum -c SHA256SUMS --ignore-missing

fettle 1.0.0

[1.0.0] — the first official release

fettle keeps a Linux machine updated and clean, audits where its software came from and
whether it has been tampered with, and scans the firmware and boot chain — from one
command surface, on four distro families.

Supported: Arch and Manjaro, Debian and Ubuntu, RHEL / Rocky / AlmaLinux, and
Fedora. Pure standard library: it needs python 3.11 or newer and nothing else, which
is what lets it ship itself to a remote host as a single file and run there under
whatever interpreter it finds.

What 1.0.0 rests on. Every action was specified, run against seven live systems, and
fixed where it misbehaved or explained itself badly. That pass is written down, feature
by feature, in docs/qa/ — roughly ninety findings, including actions that
reported success while doing nothing at all, a preview that deleted run history, and an
unattended flag that would have purged a list fettle had guessed at.

One rule came out of it and now governs the output everywhere: a check that could not
look must never render identically to a clean result. "Not installed", "could not
read", "does not apply" and "nothing found" are four different answers, and fettle says
which one it means.

Installing. Attached to this release: .deb, .rpm, .pkg.tar.zst, a zipapp that
runs anywhere there is a python 3.11+, and a prebuilt x86_64 binary that needs no python
at all. Every package is built and then installed and run in a clean container of its
own distro before it is published. SHA256SUMS covers everything.

Still experimental: fettle web, the browser UI. It is the one feature the QA pass
did not reach — it both serves a page and runs privileged actions from a password typed
into a browser, and it has not been swept. Localhost-only by default; keep it that way.
fettle report, the static HTML dashboard, is not experimental and is included.

Known limit: the prebuilt binary needs glibc 2.38 or newer, so it does not run on
Ubuntu 22.04, Debian 12 or RHEL 9. Those have their own packages here, and the zipapp
works everywhere.

---

What to download

file	for
fettle_<version>_all.deb	Debian, Ubuntu, and derivatives
fettle-<version>-1.noarch.rpm	RHEL, Rocky, AlmaLinux, Fedora
fettle-<version>-1-any.pkg.tar.zst	Arch, Manjaro, EndeavourOS
fettle-<version>-zipapp.tar.gz / .zip	anything else — runs on any Linux with python 3.11+
fettle.pyz	the zipapp on its own, if you do not want the archive

fettle is pure standard library: it needs python 3.11 or newer and nothing else.
On RHEL/Rocky/Alma 9 and Ubuntu 22.04 the system python3 is older than that, so the
packages pull in a suitable interpreter and the launcher finds it for you.

Verifying what you downloaded

sha256sum -c SHA256SUMS --ignore-missing