kernel-devel needs tools/objtool/objtool to compile out-of-tree kernel modules

[cregnec]

Qubes OS version:

Qubes release 4.0 (R4.0)

Affected component(s):

kernel-devel

---

Steps to reproduce the behavior:

dom0 kernel version is 4.14.18-1.pvops.qubes

Install kernel-devel
sudo qubes-dom0-udpate kernel-devel

Boot the fedora-26 template vm with the same kernel

Install qubes-kernel-vm-support
sudo dnf install qubes-kernel-vm-support

[user@fedora-26 ~]$ sudo dkms status
u2mfn, 4.0.17: added

Build the module
sudo dkms build -m u2mfn -v 4.0.17

Expected behavior:

Successful building of the module.

Actual behavior:

Kernel preparation unnecessary for this kernel. Skipping...

Building module:
cleaning build area...
make -j2 KERNELRELEASE=4.14.18-1.pvops.qubes.x86_64 -C /lib/modules/4.14.18-1.pvops.qubes.x86_64/build M=/var/lib/dkms/u2mfn/4.0.17/build...(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.14.18-1.pvops.qubes.x86_64 (x86_64)

DKMS make.log for u2mfn-4.0.17 for kernel 4.14.18-1.pvops.qubes.x86_64 (x86_64)
Fri Apr 20 13:21:39 CEST 2018
make: Entering directory '/usr/lib/modules/4.14.18-1.pvops.qubes.x86_64/build'
make[1]: *** No rule to make target 'tools/objtool/objtool', needed by '/var/lib/dkms/u2mfn/4.0.17/build/u2mfn.o'. Stop.
make[1]: *** Waiting for unfinished jobs....
AR /var/lib/dkms/u2mfn/4.0.17/build/built-in.o
make: *** [Makefile:1507: module/var/lib/dkms/u2mfn/4.0.17/build] Error 2
make: Leaving directory '/usr/lib/modules/4.14.18-1.pvops.qubes.x86_64/build'

General notes:

Similar bugs have been spotted on other distributions.
The reason is that "objtool needs to be included to sucessfully
build all out-of-tree kernel modules, when kernel code include metadata
are generated by this tool (for example when CONFIG_STACK_VALIDATION /
CONFIG_UNWINDER_ORC enabled - ORC is enabled by default in 4.14.9+)."

details can be found in the following link from Debian bug report.

---

Related issues:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833500

[marmarek]

First of all, kernel provided from dom0 already have u2mfn module, you don't need compile it. I know there is automatic dkms service trying to build this module at VM startup. But we've disabled it recently (QubesOS/qubes-core-agent-linux@76649d8), exactly for this reason - it's going to be part of the nearest update.
But I'm going to assume you use u2mfn just as an example, because in fact it apply to any out of tree module.
Unfortunately, adding objtool isn't enough to build out of tree kernel module, when using kernel provided from dom0. See #2844 for details. The easiest way to use out of tree modules is to use kernel installed in the VM.

@andrewdavidwong I think we should deprecate support of out of tree modules if using kernel from dom0. Kernel installed in VM works sufficiently ok for HVM (where such modules have the most use cases - drivers for PCI devices). We can't remove it completely yet, because of missing PVH support...
Where should we put such information? We have a page with supported versions, but not specific features.

[marmarek]

First of all, kernel provided from dom0 already have u2mfn module, you don't need compile it. I know there is automatic dkms service trying to build this module at VM startup. But we've disabled it recently (QubesOS/qubes-core-agent-linux@76649d8), exactly for this reason - it's going to be part of the nearest update.
But I'm going to assume you use u2mfn just as an example, because in fact it apply to any out of tree module.
Unfortunately, adding objtool isn't enough to build out of tree kernel module, when using kernel provided from dom0. See #2844 for details. The easiest way to use out of tree modules is to use kernel installed in the VM.

@andrewdavidwong I think we should deprecate support of out of tree modules if using kernel from dom0. Kernel installed in VM works sufficiently ok for HVM (where such modules have the most use cases - drivers for PCI devices). We can't remove it completely yet, because of missing PVH support...
Where should we put such information? We have a page with supported versions, but not specific features.

[andrewdavidwong]

How about in the Developer FAQ?

[andrewdavidwong]

How about in the Developer FAQ?

[HW42]

I think we should deprecate support of out of tree modules if using kernel from dom0. Kernel installed in VM works sufficiently ok for HVM (where such modules have the most use cases - drivers for PCI devices). We can't remove it completely yet, because of missing PVH support...

Not convinced if deprecating it at this point is a good idea. If you look at the last Linux versions we always had some patches which fixed important issues. So by using distro kernels people will have it much easier to build OOT modules but then need to track Xen related kernel fixes.

Of corse I try to always push fixes we track down to stable and 4.17 hopefully will look better (didn't tried the rc yet for new regressions) so this should get better, but not convinced that it's good enough yet.

I will have a look if I see an easy fix for the build problems.

[HW42]

I think we should deprecate support of out of tree modules if using kernel from dom0. Kernel installed in VM works sufficiently ok for HVM (where such modules have the most use cases - drivers for PCI devices). We can't remove it completely yet, because of missing PVH support...

Not convinced if deprecating it at this point is a good idea. If you look at the last Linux versions we always had some patches which fixed important issues. So by using distro kernels people will have it much easier to build OOT modules but then need to track Xen related kernel fixes.

Of corse I try to always push fixes we track down to stable and 4.17 hopefully will look better (didn't tried the rc yet for new regressions) so this should get better, but not convinced that it's good enough yet.

I will have a look if I see an easy fix for the build problems.

[vladimir-lu]

Would it not be sufficient to allow the kernel from dom0 to be installed in a TemplateVM so that you can build modules for it? That would require a separate package though.

It also doesn't help currently that dom0 is based on a different fedora release than the new templates. I understand that using HVM would have a higher memory requirement (perhaps this is incorrect) so for things like VPN (wireguard) I would like to avoid it.

Edit: To get objtool to build, check out the qubes-linux-kernel repository and run make prep. Then you can copy objtool into the module tree. I did this in a fedora-25 templatevm after having installed the gcc from dom0 packages.

[vladimir-lu]

Would it not be sufficient to allow the kernel from dom0 to be installed in a TemplateVM so that you can build modules for it? That would require a separate package though.

It also doesn't help currently that dom0 is based on a different fedora release than the new templates. I understand that using HVM would have a higher memory requirement (perhaps this is incorrect) so for things like VPN (wireguard) I would like to avoid it.

Edit: To get objtool to build, check out the qubes-linux-kernel repository and run make prep. Then you can copy objtool into the module tree. I did this in a fedora-25 templatevm after having installed the gcc from dom0 packages.

[homotopycolimit]

@vladimir-lu I'm stuck in a similar place. I was wondering if you could expand on your previous comment? How does this work?
I have been trying for several day to build a kernel module (v4l2loopback) and I'd like to use the qubes kernel instead of the fedora or debian distribution kernel.

[homotopycolimit]

@vladimir-lu I'm stuck in a similar place. I was wondering if you could expand on your previous comment? How does this work?
I have been trying for several day to build a kernel module (v4l2loopback) and I'd like to use the qubes kernel instead of the fedora or debian distribution kernel.