rpm-pkg kernel rpm build method doesn't work on CBL-Mariner

[dankamongmen]

I'm accustomed to using the rpm-pkg target of the kernel Makefile to build custom kernel RPMs on Fedora-like systems. I would have expected this to work on CBL-Mariner, but it fails with "Error reading spec file in tarball". I debugged this for a bit, and it looks like this comes down to missing RPM macros from /etc/rpm, installed as part of the fedpkg and fedora-packager RPMs (not part of CBL-Mariner).

I can of course build a kernel tarball and just install it myself, but it seems useful for CBL-Mariner to ensure the rpm-pkg kernel build process works, especially since it almost does.

If there's some other recommended method for building custom kernels which I've missed, my apologies!

[dankamongmen]

Here's the documentation for rpm-pkg: https://docs.fedoraproject.org/en-US/quick-docs/kernel/build-custom-kernel

[dankamongmen]

I am able to successfully build a kernel RPM if i explicitly run buildrpm -bs kernel.spec and then run rpmbuild --rebuild /usr/src/mariner/SRPMS/_kernelsrpm_, so this isn't a deep problem with tooling. I think we're just missing some expected contents of /etc/rpm.

[dankamongmen]

to save y'all some time, the error message is coming from rpmbuild, which applies tar xOvof to the generated kernelsrc+kernel.spec tarball in order to examine file contents without unpacking the tarball.

[dankamongmen]

People working on #3550 might be interested in this.

[dankamongmen]

I'm going to grab source material for fedpkg and fedora-packager out of Fedora's pagure.io, build them on Mariner, and see if that helps me out. in the meantime, i've confirmed that i can build bzImage and modules just fine manually.

[elsaco]

@dankamongmen there are changes in the CBL-Mariner kernel that are not part of the upstream Linux kernel. Microsoft already supplies kernel SRPMS at https://packages.microsoft.com/cbl-mariner/2.0/prod/base/srpms. It's not sound practice to build a vanilla kernel even though it might work inside CBL-Mariner too. Why not just use the SRPMS from CBL-Mariner repos, make the changes you want and rpmbuild it. The kernel SRPM installs in /usr/src/mariner.

[dankamongmen]

@dankamongmen there are changes in the CBL-Mariner kernel that are not part of the upstream Linux kernel. Microsoft already supplies kernel SRPMS at https://packages.microsoft.com/cbl-mariner/2.0/prod/base/srpms. It's not sound practice to build a vanilla kernel even though it might work inside CBL-Mariner too. Why not just use the SRPMS from CBL-Mariner repos, make the changes you want and rpmbuild it. The kernel SRPM installs in /usr/src/mariner.

thanks for your response, @elasco! this certainly works for me if this is the recommended way to build a custom kernel for Mariner.

i'm not sure i follow your comment about "not sound practice....inside CBL-Mariner." we're a team within Microsoft that is hoping to standardize on CBL-Mariner due to understanding it as Microsoft's "default Linux distro", which perhaps is a bad way of describing it? this won't necessarily be in any VM; we're building an appliance. i'd like to know what kind of problems you predict from building a vanilla kernel with a custom config and running it with the Mariner userspace, if any. feel free to mark this bug as closed, though (though it might be good to document this alternative method of kernel building somewhere, or indeed why you believe custom kernels to be a bad idea in conjunction with Mariner).

[mfrw]

/cc @christopherco

[christopherco]

@dankamongmen thanks for highlighting this! So far, we've published a baseline way for anyone to create custom kernel specs and get them built with the Mariner toolkit (link to reference). That being said, enabling a more streamlined kernel rpm build flow, especially one supported in the upstream kernel's makefile, is something I'd like us to invest in. We'll look into enabling the rpm-pkg target and see if we can add support for it in Mariner 3.0 (if not sooner)

[dankamongmen]

I was able to build a 6.1.1 kernel today with the Mariner kernel.spec; thanks for pointing me at it. btw, i'm surprised i'm able to sign all this stuff. cbl-mariner-ca-20211013.pem is only public material (as we would hope, given that it's in the SRPM), yet i'm somehow signing modules and the image...

[christopherco]

Good to hear you were able to get 6.1.1 kernel built!

Regarding signing, in-tree kernel modules will be signed by an ephemeral key generated during the kernel's build, and the kernel adds this ephemeral key to its trusted keyring. So those kernel modules should load successfully for that specific kernel.

I wouldn't expect out-of-tree kernel modules to successfully load unless you disable the kernel module signature enforcement in the kernel cmdline. I would be very curious to understand more if you were somehow able to load OOT signed modules

[dankamongmen]

Good to hear you were able to get 6.1.1 kernel built!

6.1.3 now =]. we are having a bit of a problem booting successfully, but i'm sure it's my fault, and expect to have it resolved shortly: https://msazure.visualstudio.com/One/_workitems/edit/16601952/?workitem=16722851

Regarding signing, in-tree kernel modules will be signed by an ephemeral key generated during the kernel's build, and the kernel adds this ephemeral key to its trusted keyring. So those kernel modules should load successfully for that specific kernel.

yep, we've verified this, and are forcing signed modules.

I wouldn't expect out-of-tree kernel modules to successfully load unless you disable the kernel module signature enforcement in the kernel cmdline. I would be very curious to understand more if you were somehow able to load OOT signed modules

do i look like a barbarian, sir? no such modules for us.

we are running custom XDP programs, but AFAIK there are no signing requirements there, because eBPF is somehow different from modules, perhaps because they're spelled differently.

should you be interested, we build our kernel here: https://msazure.visualstudio.com/One/_git/Azure-Orbital-Garuda?path=/adobuild

WORKDIR=$(mktemp -d)
cd "$WORKDIR"
KERNBALL=linux-$KERNVER.tar.xz
wget --no-verbose https://cdn.kernel.org/pub/linux/kernel/v6.x/$KERNBALL -O "$ORIGDIR"/rpm/SOURCES/kernel-$KERNVER.tar.xz
MARKERNVER=5.15.82.1-2
wget --no-verbose https://packages.microsoft.com/cbl-mariner/2.0/prod/base/srpms/kernel-$MARKERNVER.cm2.src.rpm
rpm2cpio kernel-$MARKERNVER.cm2.src.rpm | cpio -idmv -D "$ORIGDIR"/rpm/SOURCES
cp -v "$KCONFIG" "$ORIGDIR"/rpm/SOURCES/config || { echo "couldn't read $KCONFIG" >&2 ; usage >&2 ; exit 1 ; }
rpmbuild --define "_topdir $ORIGDIR/rpm" -ba "$ORIGDIR/src/kernel.spec"

with a kernel.spec and config kept in our source control. probably the crappiest element of this flow is that there's no automatic pickup of changes to your kernel.spec (you'll notice we're currently downloading the 5.15.82 SRPM, despite 5.15.86 being available). ideally (in my mind), your divergent kernel.spec will go away, and we will all use that constructed by a make rpm-pkg in upstream kernel source (this would also eliminate things like #4514).