Falco 0.11.0 based container fails to load kernel module

[mfdii]

Tested on Debian Jessie (k8s-1.4-debian-jessie-amd64-hvm-ebs-2016-10-21 ami from AWS, kernel 4.4.26-k8s) and Ubuntu 16.04 (GKE provide AMI, kernel 4.13.0-1008-gcp) with the Falco 0.11.0 docker container.

* Setting up /usr/src links from host
* Unloading falco-probe, if present
rmmod: ERROR: Module falco_probe is not currently loaded
* Running dkms install for falco

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j1 KERNELRELEASE=4.13.0-1008-gcp -C /lib/modules/4.13.0-1008-gcp/build M=/var/lib/dkms/falco/0.11.0/build....
cleaning build area...

DKMS: build completed.

falco-probe.ko:
Running module version sanity check.
Error! Module version 7640717FDE125E312760246 for falco-probe.ko
is not newer than what is already found in kernel 4.13.0-1008-gcp (7640717FDE125E312760246).
You may override by specifying --force.

depmod...

DKMS: install completed.
* Trying to load a dkms falco-probe, if present
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.13.0-1008-gcp
Found kernel config at /host/boot/config-4.13.0-1008-gcp
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.11.0-x86_64-4.13.0-1008-gcp-8f812ada63c6c5ca48357a632afbb4aa.ko
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the sysdig community
Wed Jul 25 14:28:57 2018: Falco initialized with configuration file /etc/falco/falco.yaml
Wed Jul 25 14:28:57 2018: Loading rules from file /etc/falco/falco_rules.yaml:
Wed Jul 25 14:28:57 2018: Loading rules from file /etc/falco/falco_rules.local.yaml:
Wed Jul 25 14:28:57 2018: Unable to load the driver. Exiting.
Wed Jul 25 14:28:57 2018: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

Error in dmesg is [78962.345914] module: falco_probe: Unknown rela relocation: 4.

Manually trying to modprobe the module built by the container gives the error.

$ sudo modprobe falco-probe
modprobe: ERROR: could not insert 'falco_probe': Exec format error

RPM based install works fine. Also, running falco-probe-loader from the RPM install with the kernel module compiled from the container image loads the kernel module.

[mstemm]

Btw, the latest version of sysdig has the same problem:

...
DKMS: install completed.
* Trying to load a dkms sysdig-probe, if present
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.13.0-26-generic
Found kernel config at /host/boot/config-4.13.0-26-generic
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.22.0-x86_64-4.13.0-26-generic-94a76b51a68ec35679d1401155803cb6.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
$ grep sysdig_probe /var/log/syslog | tail
Jul 25 14:37:00 ubuntu kernel: [ 2539.557399] module: sysdig_probe: Unknown rela relocation: 4
Jul 25 14:37:00 ubuntu kernel: [ 2539.580730] sysdig_probe: driver loading, sysdig-probe 0.21.0
Jul 25 14:37:00 ubuntu kernel: [ 2539.580805] sysdig_probe: failed to find page_fault_user tracepoint, disabling page-faults
Jul 25 14:37:00 ubuntu kernel: [ 2539.580806] sysdig_probe: failed to find page_fault_kernel tracepoint, disabling page-faults
Jul 25 14:37:12 ubuntu kernel: [ 2551.695765] sysdig_probe: driver unloading
Jul 25 14:37:59 ubuntu kernel: [ 2598.196968] module: sysdig_probe: Unknown rela relocation: 4

[mstemm]

The problem was that the falco image (as well as sysdig) had a newer version of binutils, which created kernel modules including a relocation R_X86_64_PLT32 that can't be read by some older kernels. This reddit page describes the problem in more detail. The kernel was only updated in 4.16 to handle this relocation, although it looks like the fix has also been backported to other kernels depending on your distro.

We'll fix this by downgrading binutils in the docker image so the built kernel module doesn't contain that relocation.

[mstemm]

And as a workaround, on debian-based distros you may be able to upgrade to a kernel that has the backport, which will allow the kernel module to be loaded.