"kernel-devel" and kernel module builds for TemplateVMs documentation missing

[mtalexan]

Qubes OS version:

R3.2 and R4.0

Affected component(s):

Documentation

---

Steps to reproduce the behavior:

Search documentation for details on building kernel modules for TemplateVMs

Expected behavior:

Documentation exists with at least basic directional instructions

Actual behavior:

No documentation exists. The closest only discusses how to use different kernels in VMs (https://www.qubes-os.org/doc/managing-vm-kernel/)

General notes:

The topic is clearly not self-explanatory as there are a number of open issues regarding the kernel-devel package for Fedora-* TemplateVMs, and there are many hits on Google Groups discussions about the topic, most without a clear resolution.
The consensus seems to be that the Qubes-provided kernel is unlikely to work for most use cases. If the drivers weren't provided as part of the Fedora kernel the Qubes-provided variant is based on, the module is frequently tightly coupled with specific kernel versions. This means the usual rpmfusion and the like, which are frequently the primary/sole sources of these modules for most users, won't contain a version that works with, or can be built against, the Qubes kernel. Instead they'll only support the most recent handful of kernel versions the stock Fedora provided, the oldest of which is usually newer than the newest Qubes dom0 kernel version.
Even for akmods and dkms drivers that try to build against the "current" kernel, this holds true. These types of modules are designed to be built from source automatically for the specific user's system under the assumption the kernel-devel package has been installed. They fail consistently though, even when the kernel-devel package for the Qubes-provided kernel is installed in the Fedora-XX TemplateVM that is set to use the Qubes-provided kernel. This is usually a result of the build-from-source configurations in the packages akmods/dkms are building. The source packages will frequently need/assume specific tools and/or settings are available for the "current" kernel. Then the equivalent tools and settings are needed within the dom0 Qubes-provided kernel, but are not available or accessible. Usually these tools would be provided thru the dnf repositories, but Qubes fails to provide many of these that match the kernel versions released for dom0. They aren't available thru the usual Fedora dnf repositories either, becuase the Fedora repository provided versions are built against the Fedora-provided kernels.
The recommendation, as pieced together from the various threads, is to instead create an "unsafe" TemplateVM clone that uses the Fedora/Debian-provided kernel instead of the tested and validated dom0 kernel as an HVM, and build/run the kernel modules against that. For things like a netVM, this is frequently the only option available to get WiFi chips working that require non-free binary drivers.

A documentation page discussing this issue, specifically this pieced together "resolution" should be created to consolidate what's known on this topic rather than having parts of it scattered all over the internet. A link to the page on how to change the kernel version in a VM, including some clarification about which parts of it are relevant, and the basic run-down given above would be sufficient.

---

Related issues:

#3835
#3742
#3158
#2908
#2844
#2692

[andrewdavidwong]

@mtalexan, would you be willing to submit a PR for this? Please see the Documentation Guidelines for details.

[mtalexan]

Sure. I'll create and submit the PR with a recommendation on how it could/should be hooked into the existing top few documentation pages that aren't part of qubes-doc

[cue-buh-zuh]

Thanks for taking the time to gather these different issues in one place. This has helped me find and understand this problem.

As of now, I am still unable to resolve this issue with Qubes 3.2. There has been a lot published in various threads, although much of it has not been fully explained.

My current goal is to compile the ZFS kernel modules to my system. This was working on previous kernel versions, but with recent updates it has broken. This seems to be related to the GCC plugins issue. Additional modules are not allowed, causing DKMS errors. I am not an expert in these systems and I am piecing together scripts in an attempt to reform my system.

This ZFS manual on the Qubes wiki site is now out of date. https://www.qubes-os.org/doc/zfs/

I have tried both compiling in dom0 and compiling in a Fedora 25 VM with the properly installed kernel-devel rpm for the headers. Porting objtool to my dom0 system does not seem to help.

Continued to see this in both systems:

> sudo dkms install -m spl -v 0.7.9
...
checking whether modules can be built... no
configure: error: *** Unable to build an empty module.
...
Error! Bad return status for module build on kernel: 4.14.57-1.pvops.qubes.x86_64 (x86_64)

Some standard documentation on how to do this using the custom kernel for HVM and copying the additional modules will be useful. This is beyond my experience.

Thank you in advance, I appreciate your help!

Sincerely,
Kevin

[Rudd-O]

This remains urgently necessary.

[andrewdavidwong]

It appears that @mtalexan never got around to this. If anyone can help, please speak up.

[mtalexan]

I'm sorry, but that's correct. I didn't get a chance to put together anything substantial. Please feel free to reassign it.

I did spend some time trying the approach I put together from various sources (described in the original issue), but hit problems when the solution was attempted soon after a new Fedora version came out. That's the time during which the largest difference exists between the kernel version used for dom0 and the stock Fedora one(s). Since kernel versions have compatibility tied to a range of gcc versions, and some features may require extra gcc features/plugins if selected, the customized Qubes kernel based on an older Fedora version pretty much requires the Qubes kernel developers to release their tools as well as the kernel source/headers since it's unlikely using the Fedora tools will work. It doesn't seem this is consistently done accurately, so for at least a few Qubes releases after I posted the issue it seemed the only functional solution was to completely rebuild the dom0 kernel from scratch without the options causing the "gcc extras" error (separate issue). Also switching completely to the Fedora kernel didn't seem to work either since it seemed to need the Qubes tools installed in it and I was never able to resolve that. Both problems were well beyond the scope of this documentation task, and that's the last I was able to look at it.

[marmarek]

FWIW, dom0-provided kernel does include headers by default, so separate kernel-devel package shouldn't be necessary. But some tools expect it to be installed anyway, and we need documentation how to deal with it.

[Rudd-O]

Nono, kernel-devel is for building kernel modules, kernel-headers is for userspace.

[marmarek]

Let me clarify: all necessary files for building kernel modules are included in dom0-provided kernel by default.