additional files in `/boot/` are ignored

[3hhh]

Describe the bug

Additional files in /boot/ are ignored by heads.

To Reproduce
Steps to reproduce the behavior:

1. sudo touch /boot/testfile
2. sudo reboot

Expected behavior

heads complains about a /boot/ integrity difference (additional file).

Actual behaviour

OS boots / heads doesn't complain.

Side Note

Removing files correctly causes a red warning.

[tlaurion]

Duplicate of #1108

Please reopen if you disagree.

---

As far as Heads is concerned for preboot security (making sure that what is going to be booted is in the expected state) the goal is to make sure that nothing trusted would refer to those added files. For example, a modified grub entrey pointing to a new kernel/initrd/Xen.

[tlaurion]

@3hhh reopening to clarify your expectations and to document this a little bit more.

You would expect Heads to

• Prevent default boot if new file present even though nothing detached signed changed?

Actual behavior:

• existing but modified files are covered
• deleted files are covered
• new files ate not covered

#1108 would expend what is showed when files are changed, to also cover presence of new files. The problem with Heads is that there is no way, from within Heads itself alone, to troubleshoot the origin of the changes/addition/removal. The OS could inform better the user, but outside of that, I'm not sure of the best behavior to implement.

---

To me, I expect Heads to tell me what changed since digest was signed, where what is important is what would impact the behavior of booted OS, which is changed files possibiky referring to each other, but changing the OS behaviorm

The good habit here currently requires the user to check Qubes dom0 updates, where I posted on Qubes OS forum some validation tricks to make sure that what is under /boot is desired.

We can expect in the longer terms that fwupd will eventually drop firmware images under /boot for Heads to pick up and propose firmware upgrade. That will imply Heads validating detached signatures that will be added in the firmware, but is related to files being added under /boot.

Another better, but more complicated approach, would be to have helpers scripts under dom0, maybe pushed as contribution packages to be installed by Heads users, to have dom0 updater report on the changes, report on files not having been deployed by Qubes under /boot, and have that helper script show a report prior of gently reminding the user that those modified/deleted files will be asked to be signed on next default boot operation which will ask to set up a new boot default if grub changed, and or pushing the user for him to generate a new digest of /boot content and sign it if new files were discovered without others being modified/deleted.

Otherwise, the best approach would be for Qubes to sign kernel and Xen binaries, or detach sign them and have Heads verify those. And once more, have Qubes provide a static and bigger initramfs containing all the drivers needed, otherwise initrd is different for all Qubes users. But grub config will still be dynamic and other files landing into /boot, without modifying others already present, currently doesn't show under default boot entree tamper detection.

---

But to go back to this issue.

• booting the default boot could open a warning for found new files ( otherwise if files were changed/modified/deleted, default boot behavior is to show those changes but cannotnreport on new files)
  • it could then permit to boot as normal, since generation of new digest + detach signing is not required now, asking for user to investigate those new additional files
  • it could give option to generate new hashes + detach sign them.

Thoughts?

[3hhh]

On 11/24/22 19:17, tlaurion wrote: You would expect Heads to - Prevent default boot if new file present even though nothing detached signed changed?

Yes, that's exactly what I would expect. Additional files can be an equal security risk as missing or changed files, so they should be treated the same way. For example I noticed this heads behaviour while I was considering to extend Qubes OS to load some files from `/boot` under certain circumstances. Assuming a user had not configured that file, an attacker could create it to produce an unwanted insecure configuration and then stage other attacks. With your `fwupd` example, one could e.g. install some insecure firmware.

The habit here currently requires the user to check Qubes dom0 updates, where I posted on Qubes OS some validation tricks to make sure that what is under /boot is desired.

If one's really scared about "did something change since the last time I booted", I'd recommend the following Opsec practice: Shut down Qubes, boot it, update, then reboot - all while being present in person.

Another, but more complicated approach, would be to have helpers scripts under dom0, maybe pushed as contribution packages to be installed by Heads users, to have dom0 updater report on the changes, report on files not having been deployed by Qubes under /boot, and have that helper script prompt a report prior of gently reminding the user that those modified/deleted/added files will be asked to be signed on next default boot operation which will ask to set up a new boot default if grub changed, and or pushing the user for him to generate a new digest of /boot content and sign it.

I prefer the Opsec practice. It's more simple and doesn't rely on additional code. ;-)

Otherwise, the best approach would be for Qubes to sign kernel and Xen binaries, or detach sign them and have Heads verify those. And once more, have Qubes provide a static and bigger initramfs containing all the drivers needed, otherwise initrd is different for all Qubes users.

Qubes initrd may already contain user-specific configuration (e.g. usbguard configuration), i.e. there'll likely always be custom initramfs for some users. Alternatively Qubes OS would have to put more configuration into `/boot`, which would make this issue even more relevant.

----- But to go back to this issue. - booting the default boot could open a warning for found new files ( otherwise if files were changed/modified/deleted, default boot behavior is to show those changes but cannotnreport on new files) - it could then permit to boot as normal, since generation of new digest + detach signing is not required now, asking for user to investigate those new additional files - it could give option to generate new hashes + detach sign them.

Please prevent boot unless there's a resign or the insecure boot option is chosen.

[tlaurion]

For example I noticed this heads behaviour while I was considering to extend Qubes OS to load some files from /boot under certain circumstances. Assuming a user had not configured that file, an attacker could create it to produce an unwanted insecure configuration and then stage other attacks.

Heads is pre-boot security only. Its goal is to make sure that what the user accepted, through basic opsec (boot Qubes and if dom0 has updates, apply them and sign changes affecting /boot asap). The reason of my disagreement is that if nothing refers to the new files that were added under /boot, outside of files that changed, then the new files in question that were added are not referred by any files that will be considered when booting.

Let's go back at current protections as explained under https://forum.qubes-os.org/t/verifying-installation
1- High level detached signed digest (authenticity+quick integrity check) of all saved digests, signed by the user when a change is detected on /boot files (deletion/modification, but not addition of files) https://forum.qubes-os.org/t/verifying-installation/11739/4
sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig -
This takes shecksums of all /boot/kexec*.txt files and verify that they match integrity/authenticity of detached signature under /boot/kexec.sig
Those verify that, with a default boot configured and a TPM disk unlock key setuped:
/boot/kexec_default.1.txt /boot/kexec_default_hashes.txt /boot/kexec_hashes.txt /boot/kexec_key_devices.txt /boot/kexec_lukshdr_hash.txt /boot/kexec_rollback.txt
Are in expected state.

Let's detail

• /boot/kexec_default.1.txt contains the user accepted boot options to be passed when selecting configured boot option. This included any option that would be passed to either Xen or the kernel to boot into the system
• /boot/kexec_default_hashes.txt contains all /boot accepted hashes, and their checksums are verified prior of booting.
  One could replicate this from within booted OS through cd /boot && sudo sha256sum -c /boot/kexec_hashes.txt
• /boot/kexec_key_devices.txt and /boot/kexec_lukshdr_hash.txt are related to currently managed LUKS containers managed through Heads, where default boot will refuse to boot default boot option if the LUKS header changed
• /boot/kexec_rollback.txt is a file containing TPM counter value, which is an increment-only that can be used to prevent roll-back attacks on the signed kernel and initramfs configurations. (needs to be enforced)

stage other attacks

In current Heads state, Heads not warning about new files under /boot is not helping the OS to prevent OS chained attacks.

This post https://forum.qubes-os.org/t/verifying-installation/11739/11 contains what I would implement.
The idea here would not to "block" current trusted and accepted boot configuration to be booted if new files are present.
But would go in more depth prior of making the user accept new files to be part of detached, and trusted files under boot when a trusted /boot configuration changed. Thje difference between the two statements is that currently, when /boot files monitored are not changed, nothing says to the user that something was "Added there"

With your fwupd example, one could e.g. install some insecure firmware.

Including a firmware image that would be dropped in place, where there is no suppot, as of today, to propose to the user to flash such image if it was dropped under /boot anyway, unless the user calls flash.sh /boot/firmware.rom manually from the recovery shell. Dropping a firmware image to /boot is actually really convenient, and is the desired way for the future. But mixing concepts here should not be made. that firmware image should be accompanied by a detached signature as well. And that detached signature would be provided to LVFS as part of the firmware update package. And prior of being proposed to the user as trusted, this also means that Heads will need to have those public keys for supported FWUPD distributors, or otherwise, provide a big fat warning if a firmware is being dropped not being verified in any way. This is why this is not part of Heads as of now, the pieces to make this work correctly are not present as of now. Which requires users to verify those firmware images themselves prior of manually flashing them, without any automatic update mechanisms in place.

Qubes initrd may already contain user-specific configuration (e.g. usbguard configuration), i.e. there'll likely always be custom initramfs for some users. Alternatively Qubes OS would have to put more configuration into /boot, which would make this issue even more relevant.

Having a Quebes signed, static initrd by default would permit supported hardware to benefit from detached signed or signed kernel initrd and xen binaries and would permit Heads to verify those way more easily, if they were available. As you pointed out, a user created initrd upon dom0 pushed updates (as of today) could not be verified in any other way then current way from Heads. when dom0 changes /boot, this is because either grub, grub.cfg, Xen or kernel packages were deployed. When that happens, we know grub.cfg will be modified to point to new Xen kernel initrd binaries, and Heds currently and correctly (in my opinion) of the changes related to the changes. Ideally, I would love to see more details in what default boot configuration changed to quickly audit changes in enforced boot parameters. That could be subject to another issue, parsing currently detached signed /boot/kexec_default.1.txt and comparing to what would become the new boot default trusted boot parameters. But this is currently delegated to the user.

---

So to synthetise my proposition here based on https://forum.qubes-os.org/t/verifying-installation/11739/11?u=insurgo

Not much "new files" here, so example is not that much relevant.
But the idea would be, prior of signing upon modified files detected (again grub.cfg, past trusted xen kernel and initrd having vanished (deleted), to actually give a report in a more user friendly form then below, showing to the user that some files were added:

[user@dom0 boot]$ sudo find ./ -type f ! -name '*kexec*' -print0 | sudo xargs -0 sha256sum > /tmp/kexec_hashes.txt
[user@dom0 boot]$ diff /boot/kexec_hashes.txt /tmp/kexec_hashes.txt 
8,9c8,9
< 3e5f6ccc321ac18ac47d581db40f840b76507884d233fee0e0ad85bdc42d0345  ./efi/EFI/qubes/xen.efi
< 3e5f6ccc321ac18ac47d581db40f840b76507884d233fee0e0ad85bdc42d0345  ./efi/EFI/qubes/xen-4.14.5.efi
---
> 7df8ea17b75e0a38d0840769732e3192006611c10e5a516beb732e47560e35ae  ./efi/EFI/qubes/xen.efi
> 7df8ea17b75e0a38d0840769732e3192006611c10e5a516beb732e47560e35ae  ./efi/EFI/qubes/xen-4.14.5.efi
22c22
< 49de5d9fd4286e5ff6d61d21bd7ae5982f765efe71d19d01e2e81e7a5cfc4794  ./xen-4.14.5.gz
---
> 43467546e0a7787eeb93d49523b01c5d4e15deddbbbd1315df95048350139def  ./xen-4.14.5.gz

The above currently compares detached signed boot digest with runtime-generated one.
New files would show as > and could be processed to provide a "New files detected", not currently having an impact if no currently trusted files changed.

But then now what?

---

What I read from your proposition:
Refuse to default boot in presence of new files detected.
How would you want to handle that?

---

My proposition:

Propose to the user new files as separated from changed files when changed files are detected when attemtping to default boot.

This output could show to him the following under a "less" output
Changed files:
-changed file + date (existing)
-changed file (deleted)

New files

• file + creation date (existing)

And then when he exits the detailed information screen, propose to him to:
Unsafe boot + investigate / Generate and sign new digest

Thoughts welcome!

[3hhh]

The idea here would not to "block" current trusted and accepted boot configuration to be booted if new files are present.
But would go in more depth prior of making the user accept new files to be part of detached, and trusted files under boot when a trusted /boot configuration changed. Thje difference between the two statements is that currently, when /boot files monitored are not changed, nothing says to the user that something was "Added there"

No, please don't do that for the reasons I outlined: You don't really know whether the new files aren't referenced from any initrd code. You only knew for sure if you exclusively cared about Qubes OS and would review their code whenever they push an update and 100% trust your review skills.

Otherwise any new file may be used to break previously made boot integrity assumptions by the user.

With the idea you outlined above, the attacker would not need to fear any heads message informing the user about new files until the next time the user needs to sign again. That's why it's a bad idea IMHO.

If you don't want to block the boot process as I proposed, feel free to only warn the user before booting. The user needs to be able to make an informed decision about boot integrity every time s/he boots IMHO.

[tlaurion]

@3hhh I appreciate challenges in ideas, thank you for that.

Let's make this discussion really practical so you can quote snippets and tell me what you would love to see.

Content of the script I run personally under dom0 prior and after dom0 updates, which simulates what Heads does for detached signature validatation + output of files that changed:

[user@dom0 boot]$ cat /boot/verify_detached_signatures.sh 
sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig - && ( if [ -e /usr/bin/sudo ] ; then SUDO=sudo; fi; cd /boot && $SUDO sha256sum -c /boot/kexec_hashes.txt ) | grep -v OK

Additional context: I have done a sudo touch /boot/tampered prior of calling the script prior and after updates.

Prior of update:

[user@dom0 ~]$ /boot/verify_detached_signatures.sh
gpg: Signature made Thu 24 Nov 2022 09:42:41 AM EST
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93

The script return on the command line without giving output. The file /boot/tampered missed detection.

We update dom0.
The update widget gives us the following detailed output:

Updating dom0

local:
    ----------
    kernel:
        ----------
        new:
            1000:5.15.74-2.fc32.qubes,1000:5.15.76-1.fc32.qubes,1000:5.15.78-1.fc32.qubes
        old:
            1000:5.15.74-2.fc32.qubes,1000:5.15.74-1.fc32.qubes,1000:5.15.76-1.fc32.qubes
    kernel-qubes-vm:
        ----------
        new:
            1000:5.15.74-2.fc32.qubes,1000:5.15.76-1.fc32.qubes,1000:5.15.78-1.fc32.qubes
        old:
            1000:5.15.74-2.fc32.qubes,1000:5.15.74-1.fc32.qubes,1000:5.15.76-1.fc32.qubes
    python3-qubesadmin:
        ----------
        new:
            4.1.27-1.fc32
        old:
            4.1.26-1.fc32
    python3-xen:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32
    qubes-core-admin-client:
        ----------
        new:
            4.1.27-1.fc32
        old:
            4.1.26-1.fc32
    qubes-core-qrexec:
        ----------
        new:
            4.1.20-1.fc32
        old:
            4.1.18-1.fc32
    qubes-core-qrexec-dom0:
        ----------
        new:
            4.1.20-1.fc32
        old:
            4.1.18-1.fc32
    qubes-core-qrexec-libs:
        ----------
        new:
            4.1.20-1.fc32
        old:
            4.1.18-1.fc32
    qubes-gpg-split-dom0:
        ----------
        new:
            2.0.64-1.fc32
        old:
            2.0.63-1.fc32
    qubes-img-converter-dom0:
        ----------
        new:
            1.2.13-1.fc32
        old:
            1.2.12-1.fc32
    qubes-pdf-converter-dom0:
        ----------
        new:
            2.1.14-1.fc32
        old:
            2.1.13-1.fc32
    xen:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32
    xen-hypervisor:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32
    xen-libs:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32
    xen-licenses:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32
    xen-runtime:
        ----------
        new:
            2001:4.14.5-14.fc32
        old:
            2001:4.14.5-13.fc32

After update, running the script reproducing Heads current behavior leading to possible /boot tampering report:

[user@dom0 ~]$ /boot/verify_detached_signatures.sh
gpg: Signature made Thu 24 Nov 2022 09:42:41 AM EST
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
sha256sum: ./System.map-5.15.74-1.fc32.qubes.x86_64: No such file or directory
./System.map-5.15.74-1.fc32.qubes.x86_64: FAILED open or read
./efi/EFI/qubes/xen.efi: FAILED
./efi/EFI/qubes/xen-4.14.5.efi: FAILED
sha256sum: ./config-5.15.74-1.fc32.qubes.x86_64: No such file or directory
./config-5.15.74-1.fc32.qubes.x86_64: FAILED open or read
sha256sum: ./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.15.74-1.fc32.qubes.x86_64.conf: No such file or directory
./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.15.74-1.fc32.qubes.x86_64.conf: FAILED open or read
./xen-4.14.5.gz: FAILED
sha256sum: ./symvers-5.15.74-1.fc32.qubes.x86_64.gz: No such file or directory
./symvers-5.15.74-1.fc32.qubes.x86_64.gz: FAILED open or read
./grub2/grub.cfg: FAILED
sha256sum: ./initramfs-5.15.74-1.fc32.qubes.x86_64.img: No such file or directory
sha256sum: ./vmlinuz-5.15.74-1.fc32.qubes.x86_64: No such file or directory
sha256sum: WARNING: 6 listed files could not be read
sha256sum: WARNING: 4 computed checksums did NOT match
./initramfs-5.15.74-1.fc32.qubes.x86_64.img: FAILED open or read
./vmlinuz-5.15.74-1.fc32.qubes.x86_64: FAILED open or read

So files that were deleted report FAILED open or read, while files changed report FAILED for integrity measurements. which Heads all report as changes.

What can easily be done additionally with what we currently have, simulating what Heads does to generate a new on the fly digest to be compared, whenever default boot is chosen and that on the fly generated /boot digest doesn't match detached signed one (because of new files):

[user@dom0 ~]$ cd /boot
[user@dom0 boot]$ sudo find ./ -type f ! -name '*kexec*' -print0 | sudo xargs -0 sha256sum > /tmp/kexec_hashes.txt
[user@dom0 boot]$ diff /boot/kexec_hashes.txt /tmp/kexec_hashes.txt 
7d6
< 19addf5e788ada97d735c0627ed13a9f57b839e6e286809ec029845dce191edb  ./System.map-5.15.74-1.fc32.qubes.x86_64
9,10c8,9
< 93f0cc629b16fcb24353437c2de4be691c63ee5c2caba2bba075c46038e7bfcf  ./efi/EFI/qubes/xen.efi
< 93f0cc629b16fcb24353437c2de4be691c63ee5c2caba2bba075c46038e7bfcf  ./efi/EFI/qubes/xen-4.14.5.efi
---
> fc3ec2b0a0d2f295843c2ab62e5ecdf2ff461553b913f179603643323b432e0a  ./efi/EFI/qubes/xen.efi
> fc3ec2b0a0d2f295843c2ab62e5ecdf2ff461553b913f179603643323b432e0a  ./efi/EFI/qubes/xen-4.14.5.efi
15d13
< cc484cca2360808bfe1ac83021d8e9c3ee374d9673b9631a300b349b9f2f0bb7  ./config-5.15.74-1.fc32.qubes.x86_64
18c16
< c05f3d5dd7dac662df25c32a029f4e79b3bb87ce80b21f49d6030e92dcbcb0f9  ./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.15.74-1.fc32.qubes.x86_64.conf
---
> 4b0f7043580f429fda8f37c8093056bc4a28de83773a7d8d9373734cecae8090  ./symvers-5.15.78-1.fc32.qubes.x86_64.gz
19a18
> 8e1a40520271548c1a4d91ef4b8da3abcaa254607873ce1cceb42a06f607d3f6  ./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.15.78-1.fc32.qubes.x86_64.conf
21c20
< 3d89a6f5023e04ceb41bbd73de4e113321ecb8fa6cc73858241b681ffb425783  ./xen-4.14.5.gz
---
> 017a5f69ac2417917904246a5782d93c157420e9977d390f833e92da4f3c54b5  ./xen-4.14.5.gz
23a23,24
> b73db33e6be0edcb5baba560211b793629e8df40c5e4d0189f0b672b0542c1b7  ./initramfs-5.15.78-1.fc32.qubes.x86_64.img
> ccbfb4f53f5ec1aece474a9b87bc63e268f387f4ef8e6e3ddeb2b363021db5c4  ./config-5.15.78-1.fc32.qubes.x86_64
25d25
< b452196e0b495968b2f02d1629a10b33e9ff4ae48722194645025d17722171d8  ./symvers-5.15.74-1.fc32.qubes.x86_64.gz
28a29,30
> 1b0f189c3d9f47b1abcf47ebfaeec6a195e8fe5039c0aa49a66208455e0ef68a  ./System.map-5.15.78-1.fc32.qubes.x86_64
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  ./tampered
314c316
< 64eb08769297f37ddcb1b752ff78fb1683771cb1fa47b7ccfa4b8ecbba11c4cd  ./grub2/grub.cfg
---
> d588ac1f54a801d5f5395d4c2e29a65660e40a9d2d614172bb9c33ce973f417a  ./grub2/grub.cfg
324d325
< eaa4a9c5f4a230aaf4ef84105c725aee14e6250053cc0bab35a275bf07d8a2a5  ./initramfs-5.15.74-1.fc32.qubes.x86_64.img
326c327
< 6fd1f992f469695d5f23b34a7db6efd0dd2e8f7c81e8fb5acc39179d488fcb9b  ./vmlinuz-5.15.74-1.fc32.qubes.x86_64
---
> 09745168e95caf5e25ba9ba3adf9fae92c0d29ac2d4273295c389e9d39256798  ./vmlinuz-5.15.78-1.fc32.qubes.x86_64

On a default boot attempt, without having applied updates touching /boot, the following code snipped would have detected :
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ./tampered alone.

But in case of update, /boot/tampered will just be part of the list, equivalently pertinent to any new files that were not part of the previous digest:

[user@dom0 boot]$ diff /boot/kexec_hashes.txt /tmp/kexec_hashes.txt  | grep ">"
> fc3ec2b0a0d2f295843c2ab62e5ecdf2ff461553b913f179603643323b432e0a  ./efi/EFI/qubes/xen.efi
> fc3ec2b0a0d2f295843c2ab62e5ecdf2ff461553b913f179603643323b432e0a  ./efi/EFI/qubes/xen-4.14.5.efi
> 4b0f7043580f429fda8f37c8093056bc4a28de83773a7d8d9373734cecae8090  ./symvers-5.15.78-1.fc32.qubes.x86_64.gz
> 8e1a40520271548c1a4d91ef4b8da3abcaa254607873ce1cceb42a06f607d3f6  ./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.15.78-1.fc32.qubes.x86_64.conf
> 017a5f69ac2417917904246a5782d93c157420e9977d390f833e92da4f3c54b5  ./xen-4.14.5.gz
> b73db33e6be0edcb5baba560211b793629e8df40c5e4d0189f0b672b0542c1b7  ./initramfs-5.15.78-1.fc32.qubes.x86_64.img
> ccbfb4f53f5ec1aece474a9b87bc63e268f387f4ef8e6e3ddeb2b363021db5c4  ./config-5.15.78-1.fc32.qubes.x86_64
> 1b0f189c3d9f47b1abcf47ebfaeec6a195e8fe5039c0aa49a66208455e0ef68a  ./System.map-5.15.78-1.fc32.qubes.x86_64
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  ./tampered
> d588ac1f54a801d5f5395d4c2e29a65660e40a9d2d614172bb9c33ce973f417a  ./grub2/grub.cfg
> 09745168e95caf5e25ba9ba3adf9fae92c0d29ac2d4273295c389e9d39256798  ./vmlinuz-5.15.78-1.fc32.qubes.x86_64

Of course, we could simplify the output visually.
NEW_FILES=$(diff /boot/kexec_hashes.txt /tmp/kexec_hashes.txt | grep ">" | awk -F "./" '{print "/boot/"$NF}')
echo -e "New files present on /boot: \n\n$NEW_FILES\n\nPress 'q' keyboard's key to exit this screen" > /tmp/inspect

And then have Heads do a less /tmp/inspect in its unescapable code path to show through less:

New files present on /boot: 

/boot/xen.efi
/boot/xen-4.14.5.efi
/boot/symvers-5.15.78-1.fc32.qubes.x86_64.gz
/boot/ab22ef20d85a490f8e042b251de691a9-5.15.78-1.fc32.qubes.x86_64.conf
/boot/xen-4.14.5.gz
/boot/initramfs-5.15.78-1.fc32.qubes.x86_64.img
/boot/config-5.15.78-1.fc32.qubes.x86_64
/boot/System.map-5.15.78-1.fc32.qubes.x86_64
/boot/tampered
/boot/grub.cfg
/boot/vmlinuz-5.15.78-1.fc32.qubes.x86_64

Press 'q' keyboard's key to exit this screen

So that would fit your use case of when new files were deployed on the default boot code path, refusing to boot without giving a warning first. But is that sufficient? What would you enforce more then that?

---

The question here is how that information should be presented to not confuse users even more and help them investigate/sign the changes?

Add in /tmp/inspect output for the user to continue to boot into unsafe mode to inspect? To go through options and generate new digests and sign if he recognises the changes? With the scenario you are talking about, booting the system would compromise even more and scare the user even more.

So my question there is more about what do we expect the user to be enough knowleadgeable about.

Questions:

• If the detached digest are still valid (no integrity change on the files already signed and known) : should we propose to the user to go even farther and delete such new detected files?
• This is why I presented the idea that such scripts should be documented for advanced users to duplicate functionality and be able to inspect in the OS prior of launching an update that would impact /boot, and have the possibility to inspect the changes, even going further and use OS tools to see if the files are coming from the OS updates.
• Otherwise Heads can say to the user: your have the fillowing /boot/unknown_filenames.bin found under /boot. But I'm not sure how to make that information actionable for the general user.
• From my knowledge, someone malintentioned would most probably add stuff inside of the initrd. Not sure why something else would be added under /boot, but for the OS to maybe depend on it. As said previously, OS is responsible to rotate those kernel/Xen/initrd, which are not "signed" as of right now in a way Heads can do something with the binaries alone. So it seems that again, if grub is modified to point to a new initrd/kernel, and that the user cannot verify the origin of those initrd themselves (important knowledge required there) then I am again more in the direction of adding the above logic to extend discoveries of new files, combined to the current output of modified files (deleted/modified) that were in the detached signed digest, which integrity of the digest is attested and can be used to do the comparison which output can be given, more extended, to the user.

---

More simpler version of my questions:

• New files not part of detached signed digest while all other files matches the detached signed digest: what options would you give to the end user to act upon the new files detected?
• Files deleted/modifed/added as compared to detached signed digest: Show new files added (would include new kernel/initrd/xen as a seperate section of files changed and files deleted).

[3hhh]

Content of the script I run personally under dom0 prior and after dom0 updates, which simulates what Heads does for detached signature validatation + output of files that changed:

sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig - && ( if [ -e /usr/bin/sudo ] ; then SUDO=sudo; fi; cd /boot && $SUDO sha256sum -c /boot/kexec_hashes.txt ) | grep -v OK

Well yes, things would get more complicated than a single line.

Here's what I came up with:

#!/bin/bash

#known issues:
# - all options require root access rights as files may otherwise remain "hidden" due to access right issues
# - TBD

[ $EUID -eq 0 ] || { >&2 echo "ERROR: Not root." ; exit 1 ; }

cd /boot/

#GPG verification commented out for test reasons
#sha256sum "$(find /boot/kexec*.txt)" | gpg --verify /boot/kexec.sig - && ( if [ -e /usr/bin/sudo ] ; then SUDO=sudo; fi; cd /boot && $SUDO sha256sum -c /boot/kexec_hashes.txt ) | grep -v OK
sha256sum -c /boot/kexec_hashes.txt | grep -v OK

#option 1: sort & uniq
#problem: displays added _and_ removed files
(sed -rn 's/^[^\s]+\s+(.*)$/\1/p' kexec_hashes.txt ; find . -type f ! -path './kexec*') | sort | uniq -u | sed -rn 's/^(.*)$/\1 ADDED OR REMOVED/p'

#option 2: comm or diff with sort
comm -23 <(find . -type f ! -path './kexec*' | sort) <(sed -rn 's/^[^\s]+\s+(.*)$/\1/p' kexec_hashes.txt | sort) | sed -rn 's/^(.*)$/\1 ADDED/p'

#option 3: sh/bash/awk/whatever code
#...

exit 0

Such information should at least be displayed as warning to be confirmed by the user IMHO.

[3hhh]

Btw none of the above options identify added directories as heads currently doesn't save the directory structure at all.

That might be something to be improved upon as well.

[3hhh]

Btw none of the above options identify added directories as heads currently doesn't save the directory structure at all.

So IMHO heads should add something like sudo find /boot/ as e.g. /boot/kexec_tree.txt to its output, sign and verify that file as well and warn the user, if this directory structure is different on boot.