Add support for 6.6.x workstation kernels #45

legoktm · 2024-03-25T20:48:58Z

Description

Linux 6.6 changed mkdebian, so our previous copy and override would've needed significant work to follow upstream. I think it's easiest to just create our own Debian directory with all the metadata we want, and any future kernel upgrades will require synchronizing debian/rules with what upstream does. This is shown in the current file with two different stanzas for 5.15 and 6.6. The script has gotten sufficiently complex now so I've taken the opportunity to port it to Python and jinja2 templating.

For 6.6, I took the workstation 5.15 config, ran make olddefconfig and then copied over the Qubes kernel config's netfilter part. I also updated the build container to bookworm for these 6.6 builds.

Linux has a concept of "tinyconfig", which is the smallest possible kernel that can be built, which is a good way to test our build infrastructure without taking too much time. Add 5.15 and 6.6 tinyconfigs and use those in CI, cutting it down from about 10-11min to 5min.

Fixes script/mkdebian changes with 6.6 #41
Fixes bug: if grsecurity patch fails with "patch: **** out of memory" error, build continues instead of failing #40
Refs Use bullseye as base for docker container instead of buster #30
- only done for 6.6 kernels, 5.15 are still on buster.
Refs Add support for 6.6-series kernels with grsecurity patches for server and workstation #43
- only done for workstation, not server.

Test plan

Run through all the make targets:

And then:

CI passes
Visual review

legoktm

Mostly notes/questions for myself.

Dockerfile

Makefile

build-kernel.sh

legoktm · 2024-03-26T23:50:22Z

debian/control.workstation

+Package: securedrop-workstation-grsec
+Section: admin
+Architecture: ${DEBARCH}
+Pre-Depends: qubes-kernel-vm-support (>=4.0.31)


I assume we can drop this now? The 4.1 bookworm VM I have is on 4.1.19+deb12u1 already.

Per freedomofpress/securedrop-builder@de34206 this was also only needed for the dkms build, so we can drop it too.

This analysis didn't account for the switch to minimal templates, which are missing this package. So we do need a dependency on qubes-kernel-vm-support, though we can make it unversioned.

debian/control.workstation

grsecurity-urls.py

legoktm · 2024-03-27T21:05:50Z

For whatever reason, the 6.6.22 kernel doesn't have networking in Qubes with HVM. Here's the diff of kernel modules between a stock Debian bookworm kernel (with working networking) and ours: https://gist.github.com/legoktm/292e4a5dbd0c733d837cb6f0d72c0773

legoktm · 2024-03-27T22:21:57Z

The qubes-antispoof.service unit failed, because of network filter errors. Here's the beginning of the error:

Mar 27 18:17:33 d12-test systemd[1]: Starting qubes-antispoof.service - Qubes anti-spoofing firewall rules...
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:1-2: Error: Could not process rule: Operation not supported
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]: ^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: No such file or directory; did you mean chain ‘prerouting’ in table ip ‘qubes’?
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: No such file or directory; did you mean chain ‘antispoof’ in table ip ‘qubes’?
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: No such file or directory; did you mean set ‘downstream’ in table ip ‘qubes’?
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: No such file or directory; did you mean set ‘allowed’ in table ip ‘qubes’?
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: Could not process rule: No such file or directory
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: Could not process rule: No such file or directory
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: Could not process rule: No such file or directory
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:2:10-14: Error: Could not process rule: No such file or directory
Mar 27 18:17:33 d12-test nft[691]: table ip qubes {
Mar 27 18:17:33 d12-test nft[691]:          ^^^^^
Mar 27 18:17:33 d12-test nft[691]: /etc/qubes/qubes-antispoof.nft:23:1-2: Error: Could not process rule: Operation not supported
Mar 27 18:17:33 d12-test nft[691]: table ip6 qubes {
Mar 27 18:17:33 d12-test nft[691]: ^^

So the missing networkfilter modules from above are even more suspect:

-nf_conntrack
-nf_defrag_ipv4
-nf_defrag_ipv6
-nf_nat
 nfnetlink
-nf_reject_ipv4
-nf_reject_ipv6
 nf_tables
-nft_chain_nat
-nft_ct
-nft_masq
-nft_reject
-nft_reject_ipv4
-nft_reject_ipv6

legoktm · 2024-03-28T16:20:05Z

The newly built kernel (freedomofpress/securedrop-apt-test#225) now has networking on Qubes \o/

legoktm · 2024-03-28T21:04:24Z

OK, we now have seemingly functional kernels! The remaining steps here are to add some back-compat for the 5.15 series, since we're planning to keep building those for a while on the server.

thedeadliestcatch · 2024-04-01T04:24:42Z

A quick note: I'm not sure why you are pressing on HVM modes, you can actually use Xen's pvh grub.

thedeadliestcatch · 2024-04-01T04:35:48Z

The newly built kernel (freedomofpress/securedrop-apt-test#225) now has networking on Qubes \o/

Remember Qubes has transitioned away from iptables. nft is used now instead. I would strongly suggest considering building monolithic kernels with a minimal config. If you transition away from HVM to pvgrub, and even if you don't, it will be a good idea. Removing LKM support has several benefits in terms of reducing attack surface in the kernel for ROP and code injection scenarios (after all, LKM support comes with the implicit need for a dynamic linker in kernel space).

zenmonkeykstop · 2024-04-01T13:27:36Z

Yup, SDW AppVMs were previously HVM, but looks like we're moving to PVH for the next workstation release on Qubes 4.2, it's working pretty well on a test branch.

legoktm · 2024-04-01T15:14:27Z

I would strongly suggest considering building monolithic kernels with a minimal config. If you transition away from HVM to pvgrub, and even if you don't, it will be a good idea. Removing LKM support has several benefits in terms of reducing attack surface in the kernel for ROP and code injection scenarios (after all, LKM support comes with the implicit need for a dynamic linker in kernel space).

This is a good point and something I started wondering about mid-last week, whether there was any benefit to building individual modules. I'll look into doing a monolithic build.

thedeadliestcatch · 2024-04-01T16:48:49Z

Yup, SDW AppVMs were previously HVM, but looks like we're moving to PVH for the next workstation release on Qubes 4.2, it's working pretty well on a test branch.

Unrelated to the kernel-builder but remember pvgrub requires a working GRUB bootloader in the virtual disk and some issues might come up (if this is not done properly) when creating qubes from disposable templates with their kernel set to pvgrub. MIght save you time. Use a separate dispvm template from sys-usb and co or you might face a non functional system once dom0 does the USB controller handoff.

legoktm · 2024-04-01T22:28:23Z

OK, marking this as ready for review now. I've split the monolithic kernel suggestion to its own issue.

zenmonkeykstop · 2024-04-03T13:26:47Z

So far:

securedrop-workstation-6.6 built successfully
securedrop-core-5.15 does not have a complete config, starts prompting at Enable dynamic slab page orders (SLUB_DYNORDER) [N/y/?] (NEW)

others to follow.

zenmonkeykstop · 2024-04-03T14:09:58Z

6.6.23-1-grsec-workstation results

set up standaloneVM based on debian-12-xfce
copied and installed kernel directly (qvm-copy kernelname.deb & apt install ./path/to/deb
flipped standaloneVM to PVH with pvgrub2-pvh as kernel, and rebooted
booted successfully
uname -r reports 6.6.23-1-grsec-workstation
paxtest and meltdown.sh test results attached (only fail in the latter is srbds)

testlog.txt

legoktm · 2024-04-03T15:48:17Z

securedrop-core-5.15 does not have a complete config, starts prompting at Enable dynamic slab page orders (SLUB_DYNORDER) [N/y/?] (NEW)

The 6.6 build command runs make olddefconfig all, so the config file will be up to date, so I removed the explicit make olddefconfig from the script but didn't add it back in for 5.15. To quickly verify this, I generated a 5.15.16 tinyconfig that is missing settings (will commit), did a build and ran into the same config prompt. I then added make olddefconfig to the 5.15 d/rules part, and now the tiny-5.15 build works fine.

legoktm · 2024-04-03T18:08:41Z

Some reproducibility issue seems to have snuck in...debugging.

New configuration was generated by taking the old 5.15 config, taking a 6.6.22 grsec-patched kernel tree and running `make xen.config` and then `make olddefconfig` in a bookworm VM.

We are missing netfilter modules that the Qubes firewall/network needs, so copy their config directly: <https://github.com/QubesOS/qubes-linux-kernel/blob/b93840a3c8254c696bc3fedcb792bb3ed99bfa19/config-base>.

Reduce the diff between the two. We can't eliminate it fully, but there are no superificial differences anymore.

Build on bookworm so we can take advantage of newer gcc and the hardening features they bring.

Instead of dynamically generating the debian/ packaging information, just keep our own. This will hopefully make future upgrades easier since we'll just need to synchronize the d/rules file. Instead of continuing to use increasingly complex shell scripts, take the opportunity to port build-kernel.sh to Python and use jinja2 for templating the dynamic parts of the debian/ folder. Take the opportunity to start xz-compressing the orig tarballs. This brings us in line with stable kernel releases and allows us to reuse upstream tarballs when possible (implemented in a later commit).

* Allow varying the base distro in the Dockerfile * Add conditionals to d/rules to vary how we build the kernel packages * Add tinyconfig from 5.15.16, intentionally using an old version that has missing settings

To avoid repacking the kernel source on non-grsec builds, we can reuse the upstream tarball. The only thing we modify in the kernel source is `.config`, which we can place in the debian/ directory.

* Build tinyconfig kernels instead of "vanilla" ones for speed * Have one job build both times instead of duplicating into a second job * Use diffoscope on the .changes file but ignore the buildinfo file * Install xz-utils so the packages can be diffed instead of hexdumped

legoktm · 2024-04-09T14:51:44Z

I better documented the changes around xz and rebased and signed this commit stack.

libelf-dev is a leftover from the u2mn kernel module builds that we needed on buster/4.0 but haven't since. Switch qubes-kernel-vm-support to an unversioned `Depends` as it's only needed in the postinst and the necessary version is already in both bullseye and bookworm.

rocodes

Good sleuthing, and thank you!

new build needed

rocodes · 2024-04-19T15:55:04Z

Tacked on 2 wee commits:

restore pre-depends
pass BUILD_VERSION to the container (tested via Add workstation grsec 6.2.28-2 kernels securedrop-apt-test#232)

legoktm · 2024-04-19T16:30:39Z

Both of your commits LGTM, thanks. (I still don't understand the Pre-Depends vs Depends, I will take some time next week to better understand the dependency.)

rocodes · 2024-04-22T15:02:35Z

Per internal chat, kernels looking good, gonna merge -thanks @legoktm and team for all your work on this

legoktm mentioned this pull request Mar 25, 2024

script/mkdebian changes with 6.6 #41

Closed

legoktm force-pushed the kernel-6 branch 2 times, most recently from c2b75ff to ec1c58b Compare March 26, 2024 04:18

zenmonkeykstop assigned legoktm and zenmonkeykstop Mar 26, 2024

legoktm commented Mar 26, 2024

View reviewed changes

legoktm force-pushed the kernel-6 branch 2 times, most recently from 615c69f to 5d4b064 Compare March 28, 2024 20:55

legoktm force-pushed the kernel-6 branch 2 times, most recently from e466358 to 0d6f913 Compare April 1, 2024 20:54

legoktm mentioned this pull request Apr 1, 2024

bug: if grsecurity patch fails with "patch: **** out of memory" error, build continues instead of failing #40

Closed

legoktm changed the title ~~WIP: Add support for 6.6.x workstation kernels~~ Add support for 6.6.x workstation kernels Apr 1, 2024

legoktm mentioned this pull request Apr 1, 2024

Build a monolithic kernel #48

Open

legoktm force-pushed the kernel-6 branch from 7b41ec3 to 0dc5956 Compare April 3, 2024 15:51

cfm self-requested a review April 5, 2024 01:18

legoktm force-pushed the kernel-6 branch from 05c2ffa to f5f0615 Compare April 9, 2024 14:46

legoktm added 9 commits April 9, 2024 10:50

Start adding 6.6 support

4f7504a

New configuration was generated by taking the old 5.15 config, taking a 6.6.22 grsec-patched kernel tree and running `make xen.config` and then `make olddefconfig` in a bookworm VM.

Copy netfilter config from Qubes's config

ce3aeda

We are missing netfilter modules that the Qubes firewall/network needs, so copy their config directly: <https://github.com/QubesOS/qubes-linux-kernel/blob/b93840a3c8254c696bc3fedcb792bb3ed99bfa19/config-base>.

Harmonize postinsts

656798a

Reduce the diff between the two. We can't eliminate it fully, but there are no superificial differences anymore.

Pass -it to docker if we're a tty

9b9b900

Update Dockerfile to bookworm

c2135c4

Build on bookworm so we can take advantage of newer gcc and the hardening features they bring.

Retrofit 5.15 support

392f0d2

* Allow varying the base distro in the Dockerfile * Add conditionals to d/rules to vary how we build the kernel packages * Add tinyconfig from 5.15.16, intentionally using an old version that has missing settings

Reuse upstream tarball for non-grsec builds

0a289e6

To avoid repacking the kernel source on non-grsec builds, we can reuse the upstream tarball. The only thing we modify in the kernel source is `.config`, which we can place in the debian/ directory.

Improve CI

c57fd07

* Build tinyconfig kernels instead of "vanilla" ones for speed * Have one job build both times instead of duplicating into a second job * Use diffoscope on the .changes file but ignore the buildinfo file * Install xz-utils so the packages can be diffed instead of hexdumped

legoktm force-pushed the kernel-6 branch from f5f0615 to 586d25e Compare April 9, 2024 14:51

legoktm added 2 commits April 17, 2024 11:17

Update README

0594be8

legoktm force-pushed the kernel-6 branch from 586d25e to 0594be8 Compare April 17, 2024 15:20

rocodes previously approved these changes Apr 17, 2024

View reviewed changes

rocodes mentioned this pull request Apr 18, 2024

Add 6.6.28 workstastation grsec kernels freedomofpress/securedrop-apt-test#231

Merged

3 tasks

rocodes added 2 commits April 19, 2024 11:49

Include pre-depends on qubes-kernel-vm-support in grsec metapackage.

5a9b8ef

Pass BUILD_VERSION env var to kernel-building container

edb3a47

rocodes mentioned this pull request Apr 19, 2024

Add workstation grsec 6.2.28-2 kernels freedomofpress/securedrop-apt-test#232

Merged

3 tasks

rocodes merged commit 32dae88 into main Apr 22, 2024
12 checks passed

legoktm deleted the kernel-6 branch April 22, 2024 15:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for 6.6.x workstation kernels #45

Add support for 6.6.x workstation kernels #45

legoktm commented Mar 25, 2024 •

edited

legoktm left a comment

legoktm Mar 26, 2024

legoktm Mar 28, 2024

legoktm Apr 17, 2024

legoktm commented Mar 27, 2024

legoktm commented Mar 27, 2024

legoktm commented Mar 28, 2024

legoktm commented Mar 28, 2024

thedeadliestcatch commented Apr 1, 2024

thedeadliestcatch commented Apr 1, 2024

zenmonkeykstop commented Apr 1, 2024

legoktm commented Apr 1, 2024

thedeadliestcatch commented Apr 1, 2024

legoktm commented Apr 1, 2024

zenmonkeykstop commented Apr 3, 2024

zenmonkeykstop commented Apr 3, 2024

legoktm commented Apr 3, 2024

legoktm commented Apr 3, 2024

legoktm commented Apr 9, 2024

rocodes left a comment

rocodes commented Apr 19, 2024 •

edited

legoktm commented Apr 19, 2024

rocodes commented Apr 22, 2024

Add support for 6.6.x workstation kernels #45

Add support for 6.6.x workstation kernels #45

Conversation

legoktm commented Mar 25, 2024 • edited

Description

Test plan

legoktm left a comment

Choose a reason for hiding this comment

legoktm Mar 26, 2024

Choose a reason for hiding this comment

legoktm Mar 28, 2024

Choose a reason for hiding this comment

legoktm Apr 17, 2024

Choose a reason for hiding this comment

legoktm commented Mar 27, 2024

legoktm commented Mar 27, 2024

legoktm commented Mar 28, 2024

legoktm commented Mar 28, 2024

thedeadliestcatch commented Apr 1, 2024

thedeadliestcatch commented Apr 1, 2024

zenmonkeykstop commented Apr 1, 2024

legoktm commented Apr 1, 2024

thedeadliestcatch commented Apr 1, 2024

legoktm commented Apr 1, 2024

zenmonkeykstop commented Apr 3, 2024

zenmonkeykstop commented Apr 3, 2024

6.6.23-1-grsec-workstation results

legoktm commented Apr 3, 2024

legoktm commented Apr 3, 2024

legoktm commented Apr 9, 2024

rocodes left a comment

Choose a reason for hiding this comment

rocodes commented Apr 19, 2024 • edited

legoktm commented Apr 19, 2024

rocodes commented Apr 22, 2024

legoktm commented Mar 25, 2024 •

edited

rocodes commented Apr 19, 2024 •

edited