3.3.0 ISO: systemd-fstab-generator writes no mount units at boot — root stays read-only, so-setup cannot run

[hassanjafari24]

Version

3.3.0

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

32

Storage for /

293

Storage for /nsm

1200

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Fresh 3.3.0 ISO install intended as a SENSOR node. Root filesystem is read-only on
every boot and so-setup cannot run.

The visible failure is systemd-remount-fs.service, but that unit is a victim.
systemd-fstab-generator produces no mount units at all during boot. Running the
same generator by hand after boot produces the complete, correct set and exits 0.

Posting here rather than a general Linux forum because the ISO performs the
partitioning and writes the fstab, and the only working control I have is another
node installed from this same ISO. If this is out of scope, I'd appreciate a
pointer to the right place.

Reproduced across two independent clean installs, on every boot.

ENVIRONMENT

Broken node: SENSOR, single 1536G disk, node-local LVM-thin
Control node: MANAGERSEARCH, same ISO, same hypervisor host, identical VM config
except a 600G disk. Installed and runs without fault.

Oracle Linux 9, kernel 6.12.0-206.104.3.3.el9uek.x86_64 (UEK)
Proxmox VE 9.2.10 - SeaBIOS, q35, virtio-scsi-single, iothread=1, cpu: host

SYMPTOM

./so-functions: line 28: /root/sosetup.log: Read-only file system

hostnamectl fails, the hostname never persists (prompt stays localhost), network
config never applies, and Setup exits with "The management IP could not be
determined."

From dmesg:

[ 6.647033] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 6.664480] systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
[ 6.665427] systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
[ 6.666328] systemd[1]: Failed to start Remount Root and Kernel File Systems.

Under 20 ms, with no diagnostic output anywhere.

THE ACTUAL CAUSE

/run/systemd/generator/ on the broken node contains only:

local-fs.target.wants/systemd-remount-fs.service -> /usr/lib/systemd/system/systemd-remount-fs.service

No -.mount, no boot.mount, no nsm.mount, no tmp.mount, no swap unit.

The same directory on the working manager:

-rw-r--r--. 1 root root 345 boot.mount
-rw-r--r--. 1 root root 226 'dev-mapper-system\x2dswap.swap'
drwxr-xr-x. 2 root root 120 local-fs.target.requires
drwxr-xr-x. 2 root root 60 local-fs.target.wants
-rw-r--r--. 1 root root 267 -.mount
-rw-r--r--. 1 root root 268 nsm.mount
drwxr-xr-x. 2 root root 60 swap.target.requires
-rw-r--r--. 1 root root 296 tmp.mount

So no fstab entry is realised at boot - not /, /boot, /nsm, /tmp or swap. Writes
to /tmp fail read-only because the tmp LV was never mounted either.
systemd-remount-fs then has no -.mount to act on and exits 1 silently, and dbus
never starts, so systemctl returns "Failed to connect to bus: Connection refused"
for the rest of the session.

Note the local-fs.target.wants symlink IS present on the broken node. That is
written after the mount units, so the generator appears to run to completion and
simply produce nothing.

THE GENERATOR WORKS WHEN RUN BY HAND

Same box, after boot:

sudo mkdir -p /run/g
sudo /usr/lib/systemd/system-generators/systemd-fstab-generator /run/g /run/g /run/g
exit=0

Output: -.mount 267, boot.mount 345, nsm.mount 268, tmp.mount 296, swap unit 226 -
the same file set at the same sizes as the working manager's.

/etc/fstab as installed:

/dev/mapper/system-root / xfs defaults 0 0
UUID=a60e1f05-7ae1-4b55-a565-83dc5c24ab13 /boot xfs defaults 0 0
/dev/mapper/system-nsm /nsm xfs defaults 0 0
/dev/mapper/system-tmp /tmp xfs nodev,nosuid,noexec 0 0
/dev/mapper/system-swap none swap defaults 0 0

Partitioning as installed: 293G root, 1.2T /nsm, 8G swap, 2G tmp.

RULED OUT

Malformed /etc/fstab - findmnt --verify --verbose returns "Success, no
errors or warnings"; the generator parses it
correctly on demand
Disk/filesystem corruption - "XFS (dm-0): Ending clean mount", no I/O errors,
lsblk shows RO 0 on all devices
Bad ISO - same ISO produced a healthy manager on the same host
Incomplete install - partitioning correct, as above
Stale ISO live environment - ide2: none,media=cdrom; reproduces after full
stop/start
Second NIC on a carrier-less
bridge - NIC removed, identical failure
Masked or empty units - find /etc/systemd/system -lname /dev/null -o -empty
returns nothing, on both nodes
SELinux difference - both Enforcing/targeted, no /.autorelabel on either
Kernel cmdline difference - /proc/cmdline identical on both nodes
Modified unit or binary - stock 779-byte unit, no drop-in directory, helper
-rwxr-xr-x root root 19744
Disk size (1536G vs 600G) - helper and generator both succeed by hand on the
1536G disk

NO DIAGNOSTICS AVAILABLE

journalctl -b -u systemd-remount-fs returns "-- No entries --". Booting with
systemd.default_standard_error=kmsg printk.devkmsg=on adds nothing beyond the four
systemd lines above. mount -o remount,rw / succeeds immediately and the system is
fully functional from that point - it is broken only at boot.

QUESTION

Has anyone seen systemd-fstab-generator produce no output during boot on the 3.3.0
ISO? Is there anything in the ISO's kickstart or dracut configuration that could
affect generator execution on first boot, and is there a supported way to capture
generator stderr that early?

Happy to run anything and report back - the node is idle and not in production.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Did you verify the ISO hash? What is your SHA256 hash for the ISO?

[hassanjafari24]

Yes.
SHA256 of the ISO is 0938c73b76ce30ec9e4394d312c79ea7cac721b6818541697279a6221f7d870d for securityonion-3.3.0-20260911.iso, matching the published value.
my manager node was installed from this same file, on the same Proxmox host, and is healthy. The sensor has also been installed twice from this ISO, with identical symptoms both times.

[cm-ops]

Have you tried to xfs_repair that partition to see if it repairs it?

[hassanjafari24]

Thanks for the suggestion. It turned out not to be filesystem corruption. The root cause was SELinux labelling.

At boot: systemd-fstab-generator: Failed to open /etc/fstab: Permission denied, then systemd-remount-fs.service failed, so root stayed read-only and nothing in fstab was mounted.

/etc/fstab on the sensor was labelled system_u:object_r:root_t:s0. On my working manager, installed from the same ISO, it's etc_t. restorecon -Rnv / showed 50 mislabelled files, mostly ones the installer writes during disk setup: /etc/fstab, /etc/crypttab, /etc/lvm/devices/system.devices, /etc/nvme/hostnqn and hostid, all root_t.

Fix, from the console:

sudo mount -o remount -o rw /
sudo restorecon -Rv /
sudo systemctl reboot

After the reboot, root is rw and all mounts are present. The same thing happened on both sensor installs and not on the manager. The sensor has a much larger disk (1.5 TB against 600 GB), in case that affects the installer's path. Happy to provide anything else useful.