kernel-devel package have broken gcc plugin

[marmarek]

Qubes OS version (e.g., R3.2):

R3.2

Affected TemplateVMs (e.g., fedora-23, if applicable):

fedora-*

---

Since enabling gcc plugins, building kernel modules using kernel-devel package is broken:

make: Entering directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'
  CC [M]  /var/tmp/u2mfn-3.2.4/u2mfn.o
cc1: error: cannot load plugin ./scripts/gcc-plugins/latent_entropy_plugin.so
   ./scripts/gcc-plugins/latent_entropy_plugin.so: undefined symbol: _Z16gsi_insert_afterP20gimple_stmt_iteratorP21gimple_statement_base19gsi_iterator_update
scripts/Makefile.build:299: recipe for target '/var/tmp/u2mfn-3.2.4/u2mfn.o' failed
make[1]: *** [/var/tmp/u2mfn-3.2.4/u2mfn.o] Error 1
Makefile:1490: recipe for target '_module_/var/tmp/u2mfn-3.2.4' failed
make: *** [_module_/var/tmp/u2mfn-3.2.4] Error 2
make: Leaving directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'

This happen even if the kernel-devel package is built in exactly the same distribution, the same gcc version etc.

Kernel options responsible:

CONFIG_GCC_PLUGINS=y
# CONFIG_GCC_PLUGIN_CYC_COMPLEXITY is not set
CONFIG_GCC_PLUGIN_LATENT_ENTROPY=y

Those options were enabled in >=4.9.28-14, package now in current-testing:
QubesOS/updates-status#55

/cc @rtiangha

[rtiangha]

This is ringing a bell, but I think it was a while ago so I can't remember. It might be a missing dependency. When does this get triggered so I can try to replicate? Compiling u2mfn somewhere? I think I had to install elfutils-libelf-devel to fix it, and if that was the case, it'd need to be a Buildrequires for that kernel option but I might have forgotten to add it to kernel.spec. Can you try installing that package and try again?

[rtiangha]

This is ringing a bell, but I think it was a while ago so I can't remember. It might be a missing dependency. When does this get triggered so I can try to replicate? Compiling u2mfn somewhere? I think I had to install elfutils-libelf-devel to fix it, and if that was the case, it'd need to be a Buildrequires for that kernel option but I might have forgotten to add it to kernel.spec. Can you try installing that package and try again?

[rtiangha]

And on Debian, I think it's called something different. libelf-dev, I believe.

[rtiangha]

And on Debian, I think it's called something different. libelf-dev, I believe.

[marmarek]

Steps to reproduce:

1. Get Fedora-based VM
2. Install qubes-vm-kernel-support
3. Download and install https://yum.qubes-os.org/r3.2/current-testing/dom0/fc23/rpm/kernel-devel-4.9.29-17.pvops.qubes.x86_64.rpm
4. Copy /usr/src/u2mfn-3.2.4 somewhere (or switch to root)
5. Build that module with make -C /usr/src/kernels/4.9.29-17.pvops.qubes.x86_64 M=$PWD module

I've tried elfutils-libelf-devel but it doesn't help, neither gcc-plugins-devel.

[marmarek]

Steps to reproduce:

1. Get Fedora-based VM
2. Install qubes-vm-kernel-support
3. Download and install https://yum.qubes-os.org/r3.2/current-testing/dom0/fc23/rpm/kernel-devel-4.9.29-17.pvops.qubes.x86_64.rpm
4. Copy /usr/src/u2mfn-3.2.4 somewhere (or switch to root)
5. Build that module with make -C /usr/src/kernels/4.9.29-17.pvops.qubes.x86_64 M=$PWD module

I've tried elfutils-libelf-devel but it doesn't help, neither gcc-plugins-devel.

[rtiangha]

This is on my dev machine (Fedora 23-based template). Is this the expected output?

[user@KernelBuilder-fc23 tmp]$ cp -ar /usr/src/u2mfn-3.2.4 /tmp/
[user@KernelBuilder-fc23 tmp]$ make -C /usr/src/kernels/4.9.29-17.pvops.qubes.x86_64 M=$PWD u2mfn-3.2.4/
make: Entering directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'
  LD      /tmp/u2mfn-3.2.4/built-in.o
  CC [M]  /tmp/u2mfn-3.2.4/u2mfn.o
make: Leaving directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'
[user@KernelBuilder-fc23 tmp]$

If so, it probably is a dependency, but I'm not sure which one yet. I'll start a fresh template and try to find out.

My bash history cuts off so I'm not sure if I've caught everything I've installed in this template, but in the meantime, could it be one of these?

coreutils module-init-tools sparse qubes-kernel-vm-support dracut busybox bc openssl-devel cmake
debian-keyring debootstrap dpkg dpkg-dev dpkg-perl keyrings-filesystem perl-TimeDate
ubu-keyring xz-lzma-compat yaml python-devel systemd-devel qubes-libvchan-xen-devel
flex bison pycairo dracut rpm-sign sparse fedora-packager fedora-review perl-Math-BigInt

[rtiangha]

This is on my dev machine (Fedora 23-based template). Is this the expected output?

[user@KernelBuilder-fc23 tmp]$ cp -ar /usr/src/u2mfn-3.2.4 /tmp/
[user@KernelBuilder-fc23 tmp]$ make -C /usr/src/kernels/4.9.29-17.pvops.qubes.x86_64 M=$PWD u2mfn-3.2.4/
make: Entering directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'
  LD      /tmp/u2mfn-3.2.4/built-in.o
  CC [M]  /tmp/u2mfn-3.2.4/u2mfn.o
make: Leaving directory '/usr/src/kernels/4.9.29-17.pvops.qubes.x86_64'
[user@KernelBuilder-fc23 tmp]$

If so, it probably is a dependency, but I'm not sure which one yet. I'll start a fresh template and try to find out.

My bash history cuts off so I'm not sure if I've caught everything I've installed in this template, but in the meantime, could it be one of these?

coreutils module-init-tools sparse qubes-kernel-vm-support dracut busybox bc openssl-devel cmake
debian-keyring debootstrap dpkg dpkg-dev dpkg-perl keyrings-filesystem perl-TimeDate
ubu-keyring xz-lzma-compat yaml python-devel systemd-devel qubes-libvchan-xen-devel
flex bison pycairo dracut rpm-sign sparse fedora-packager fedora-review perl-Math-BigInt

[marmarek]

I've re-checked template version and it looks like this is the problem: gcc plugin built on fc24 do work on fc24 and fc25 VM. But the one built on fc23 do not work on fc24+.
So, with gcc-plugin we no longer have universal kernel-devel package. Does it even worth pursuing that? There is always alternative of using pvgrub2 and installing kernel inside the (template)VM: https://www.qubes-os.org/doc/managing-vm-kernel/#using-kernel-installed-in-the-vm

[marmarek]

I've re-checked template version and it looks like this is the problem: gcc plugin built on fc24 do work on fc24 and fc25 VM. But the one built on fc23 do not work on fc24+.
So, with gcc-plugin we no longer have universal kernel-devel package. Does it even worth pursuing that? There is always alternative of using pvgrub2 and installing kernel inside the (template)VM: https://www.qubes-os.org/doc/managing-vm-kernel/#using-kernel-installed-in-the-vm

[rtiangha]

Thanks; I just finished spinning up an FC25 template and was able to replicate your results.

Just throwing this out there: Would it work the opposite way? As in, compile all kernel related stuff in FC25 and then install that into FC23? As long as everything related to the kernel is compiled in FC25 and nothing gets recompiled in FC23, that should work as the kernel stuff should be able to work independently from the rest of the system. The only major issues would be if anyone had a need to compile their own kernel module (ex. custom driver) after the fact as there might be some issues with mismatched symbols due to the different versions of the compilers. But if no one needed to do that, maybe that could be an option?

Other options (maybe undesirable) are to update the FC23 gcc to match the one being used in FC25, although that may require other things to be recompiled too. Or another one could be to create a cross-compile toolchain in a Docker image that does only the kernel stuff and its output is what gets packaged as kernel updates for all the other templates. And if anyone needed to compile their own custom kernel module, they could look up how to build that image for themselves and do it there and then copy it over to where they need it. So everything regardless of distro would work off the same baseline.

Or another option that combines the two is ship a different compiler that only gets called for kernel work so whenever anything needs to be compiled against the running kernel, it uses that compiler rather than the system one.

[rtiangha]

Thanks; I just finished spinning up an FC25 template and was able to replicate your results.

Just throwing this out there: Would it work the opposite way? As in, compile all kernel related stuff in FC25 and then install that into FC23? As long as everything related to the kernel is compiled in FC25 and nothing gets recompiled in FC23, that should work as the kernel stuff should be able to work independently from the rest of the system. The only major issues would be if anyone had a need to compile their own kernel module (ex. custom driver) after the fact as there might be some issues with mismatched symbols due to the different versions of the compilers. But if no one needed to do that, maybe that could be an option?

Other options (maybe undesirable) are to update the FC23 gcc to match the one being used in FC25, although that may require other things to be recompiled too. Or another one could be to create a cross-compile toolchain in a Docker image that does only the kernel stuff and its output is what gets packaged as kernel updates for all the other templates. And if anyone needed to compile their own custom kernel module, they could look up how to build that image for themselves and do it there and then copy it over to where they need it. So everything regardless of distro would work off the same baseline.

Or another option that combines the two is ship a different compiler that only gets called for kernel work so whenever anything needs to be compiled against the running kernel, it uses that compiler rather than the system one.

[rtiangha]

And just thinking out loud some more, if the issue is u2mfn getting recompiled at install time with a different compiler than the rest of the kernel was compiled through dracut or whatnot, I think you might be able to get around that by creating a new kernel-qubes-modules package that has u2mfn and any other custom modules compiled with the same compiler and ensure those two packages get installed together (i.e. kernel and kernel-qubes-modules). That might be another option.

[rtiangha]

And just thinking out loud some more, if the issue is u2mfn getting recompiled at install time with a different compiler than the rest of the kernel was compiled through dracut or whatnot, I think you might be able to get around that by creating a new kernel-qubes-modules package that has u2mfn and any other custom modules compiled with the same compiler and ensure those two packages get installed together (i.e. kernel and kernel-qubes-modules). That might be another option.

[marmarek]

u2mfn was just a convenient example - this is about other modules - for example if someone need out-of-tree wifi driver.

Doing make gcc-plugins in kernel headers dir (/usr/src/kernels/.../) fixes the issues. Needs gcc-c++ and gcc-plugin-devel packages. But unfortunately it is incompatible with putting it to modules.img (rtiangha/qubes-linux-kernel@77f8e8e), as it is read-only in VM.

[marmarek]

u2mfn was just a convenient example - this is about other modules - for example if someone need out-of-tree wifi driver.

Doing make gcc-plugins in kernel headers dir (/usr/src/kernels/.../) fixes the issues. Needs gcc-c++ and gcc-plugin-devel packages. But unfortunately it is incompatible with putting it to modules.img (rtiangha/qubes-linux-kernel@77f8e8e), as it is read-only in VM.

[marmarek]

As for compiling the dom0 kernel under fc25 - I don't want to mix distributions/build environments that much. If anything, build it again just for kernel-devel package - maybe we could skip actual build in that case.

[marmarek]

As for compiling the dom0 kernel under fc25 - I don't want to mix distributions/build environments that much. If anything, build it again just for kernel-devel package - maybe we could skip actual build in that case.

[marmarek]

For now, the workflow for building kernel modules, with kernel headers included in modules.img is:

1. Copy /lib/modules/$(uname -r)/build somewhere:

   cp -r /lib/modules/$(uname -r)/build /tmp/build
   

2. Install required build dependencies: gcc-plugin-devel, gcc-c++ (and similar for Debian)

3. Re-build gcc plugin:

   make -C /tmp/build gcc-plugins
   

4. Use /tmp/build for actual modules build:

   make -C /tmp/build M=$PWD modules
   

This isn't exactly what I'd expect from provided kernel headers...

[marmarek]

For now, the workflow for building kernel modules, with kernel headers included in modules.img is:

1. Copy /lib/modules/$(uname -r)/build somewhere:

   cp -r /lib/modules/$(uname -r)/build /tmp/build
   

2. Install required build dependencies: gcc-plugin-devel, gcc-c++ (and similar for Debian)

3. Re-build gcc plugin:

   make -C /tmp/build gcc-plugins
   

4. Use /tmp/build for actual modules build:

   make -C /tmp/build M=$PWD modules
   

This isn't exactly what I'd expect from provided kernel headers...

[Rudd-O]

This bug is literally preventing me from doing work at my workplace — and it's remained like this for what, three months now? I'm trying to get a template's VMs to boot with an in-template kernel, and of course this doesn't work without the damn u2mfn module, which won't compile because of this exact bug.

There is always alternative of using pvgrub2 and installing kernel inside the (template)VM:
https://www.qubes-os.org/doc/managing-vm-kernel/#using-kernel-installed-in-the-vm

That is actually the very set of instructions which this fails upon.

Sigh.

[Rudd-O]

This bug is literally preventing me from doing work at my workplace — and it's remained like this for what, three months now? I'm trying to get a template's VMs to boot with an in-template kernel, and of course this doesn't work without the damn u2mfn module, which won't compile because of this exact bug.

There is always alternative of using pvgrub2 and installing kernel inside the (template)VM:
https://www.qubes-os.org/doc/managing-vm-kernel/#using-kernel-installed-in-the-vm

That is actually the very set of instructions which this fails upon.

Sigh.

[marmarek]

Be sure to compile it for that you compile u2mfn for that in-template kernel version - not for the one provided by dom0 - which fails because of this bug.

[marmarek]

Be sure to compile it for that you compile u2mfn for that in-template kernel version - not for the one provided by dom0 - which fails because of this bug.

[Rudd-O]

I gave up trying to compile the u2mfn module, because the module automagically appears in /lib/modules/$kernel of appvm based on template (after setting the appvm to prgrub2).

Now I have a different problem, when I try to start any GUI app in the booted appvm, the GUI app simply does not start.

Quite disappointed that these simple things do not work at all.

[Rudd-O]

I gave up trying to compile the u2mfn module, because the module automagically appears in /lib/modules/$kernel of appvm based on template (after setting the appvm to prgrub2).

Now I have a different problem, when I try to start any GUI app in the booted appvm, the GUI app simply does not start.

Quite disappointed that these simple things do not work at all.

[Rudd-O]

qubes-gui-agent.service dies with journal error Error while vchan send hdr, terminating. Absolutely nothing on Google about that. No matches on qubes-gui-* github repos either.

The dom0 side just hangs with "Waiting for qubes-session".

I am stuck.

[Rudd-O]

qubes-gui-agent.service dies with journal error Error while vchan send hdr, terminating. Absolutely nothing on Google about that. No matches on qubes-gui-* github repos either.

The dom0 side just hangs with "Waiting for qubes-session".

I am stuck.

[Rudd-O]

And now (after a VM reboot) it doesn't hang anymore, but no xterm window appears when I qvm-run the xterm on the app VM.

[Rudd-O]

And now (after a VM reboot) it doesn't hang anymore, but no xterm window appears when I qvm-run the xterm on the app VM.

[marmarek]

Those are exact symptoms of missing u2mfn module.

[marmarek]

Those are exact symptoms of missing u2mfn module.

[Rudd-O]

But module is loaded. It loads right after the login prompt in xl console of the VM, and lsmod confirms it is loaded.

[Rudd-O]

But module is loaded. It loads right after the login prompt in xl console of the VM, and lsmod confirms it is loaded.

[Rudd-O]

When are we abandoning the kernels in dom0 model in favor of a supported and well tested in-VM kernel support?

…

On August 28, 2017 4:31:51 PM GMT+02:00, Rudd-O ***@***.***> wrote: But module is loaded. It loads right after the login prompt in xl console of the VM, and lsmod confirms it is loaded. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2844 (comment)

-- Rudd-O http://rudd-o.com/

[Rudd-O]

When are we abandoning the kernels in dom0 model in favor of a supported and well tested in-VM kernel support?

…

On August 28, 2017 4:31:51 PM GMT+02:00, Rudd-O ***@***.***> wrote: But module is loaded. It loads right after the login prompt in xl console of the VM, and lsmod confirms it is loaded. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2844 (comment)

-- Rudd-O http://rudd-o.com/

[Skyr]

If it's of any use to someone: I wanted to install the wireguard module via dkms (thus the naming). The following script creates an rw overlay and is called via my /rw/config/rc.local:

#!/bin/bash
LOWER=/usr/lib/modules/$(uname -r)
UPPER=/rw/dkms-overlay/$(uname -r)
WORK=${UPPER}-work
rm -rf $WORK
mkdir -p $UPPER $WORK
mount -t overlay overlay -o lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK} ${LOWER}

For each new kernel, I do
make -C /usr/lib/modules/$(uname -r)/build gcc-plugins (fix gcc plugin)
dkms install wireguard/0.0.20171221 (compile module)
Let's see if this works out… ;-)

[Skyr]

If it's of any use to someone: I wanted to install the wireguard module via dkms (thus the naming). The following script creates an rw overlay and is called via my /rw/config/rc.local:

#!/bin/bash
LOWER=/usr/lib/modules/$(uname -r)
UPPER=/rw/dkms-overlay/$(uname -r)
WORK=${UPPER}-work
rm -rf $WORK
mkdir -p $UPPER $WORK
mount -t overlay overlay -o lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK} ${LOWER}

For each new kernel, I do
make -C /usr/lib/modules/$(uname -r)/build gcc-plugins (fix gcc plugin)
dkms install wireguard/0.0.20171221 (compile module)
Let's see if this works out… ;-)