OpenShift under vsphere: Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community

[saberkan]

I deployed falco on two different clusters using helm:

In cluster one OCP 4.6 under AWS : it works fine
In cluster two OCP 4.6 under vSphere : it doesn't work

Describe the bug
When falco is deployed on kuberentes under vsphere, falco pod doesn't succeed to download the driver.

How to reproduce it
$ helm install falco falcosecurity/falco

$ oc get pods
NAME                  READY   STATUS              RESTARTS   AGE
falco-2nvsd           0/1     CrashLoopBackOff    6          7m38s
falco-7lcvb           0/1     CrashLoopBackOff    6          7m38s
falco-ghpvx           0/1     CrashLoopBackOff    6          7m38s
falco-jns5f           0/1     ContainerCreating   0          7m38s
falco-rrh4n           0/1     CrashLoopBackOff    6          7m38s

$ oc logs falco-2nvsd
* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc)
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-6)
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2aa88dcf6243982697811df4c1b484bcbe9488a2/build/make.log (with GCC /usr/bin/gcc-5)
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 4.18.0-193.29.1.el8_2.x86_64, if present
* Trying to download prebuilt module from https://download.falco.org/driver/2aa88dcf6243982697811df4c1b484bcbe9488a2/falco_rhcos_4.18.0-193.29.1.el8_2.x86_64_1.ko
curl: (22) The requested URL returned error: 404 
Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community
Fri Dec 11 10:37:56 2020: Falco version 0.26.2 (driver version 2aa88dcf6243982697811df4c1b484bcbe9488a2)
Fri Dec 11 10:37:56 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Dec 11 10:37:56 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Dec 11 10:37:57 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Dec 11 10:37:57 2020: Unable to load the driver.
Fri Dec 11 10:37:57 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

Expected behaviour
Falco should be running under OCP 4.6 in vSphere cluster. As it works on OCP under AWS.

Environment

• Falco version:

$ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART         APP VERSION
falco   falco           1               2020-12-11 10:31:37.064145974 +0000 UTC deployed        falco-1.5.5   0.26.2   

• System info:
  Pod is not running, can't check.

• Cloud provider or hardware configuration:
  OpenShift 4.6 under vSphere.

• OS:

$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="46.82.202011061621-0"
VERSION_ID="4.6"
OPENSHIFT_VERSION="4.6"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202011061621-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.6"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.6"
OSTREE_VERSION='46.82.202011061621-0'

• Kernel:

uname -a
Linux ocp4-vsphere-9v44x-worker-2qdvw 4.18.0-193.29.1.el8_2.x86_64 #1 SMP Thu Oct 22 10:09:53 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:
  Helm

Additional context
Please notice that RHEL disto under AWS and vSphere may not be the exact same build. This may explain why the problem occur only under vSphere.

[rolfberkenbosch]

Found the solution for this problem. It looks like RedHat has removed the kernel-devel package from the 4.6 RHCOS installations. Just add the following to the machine config to let RHCOS install the kernel-devel pacakge again:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: worker-extensions
spec:
  extensions:
    - kernel-devel

And one for the master machine config:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: master-extensions
spec:
  extensions:
    - kernel-devel

[leodido]

Thanks for posting here the solution @rolfberkenbosch

Can we close this?

[saberkan]

Thank you @rolfberkenbosch for the solution, it woks. @leodido I'll close the case.

[kkujawa-sumo]

I see the same issue on OpenShift 4.6.4 on AWS,

Kernel Version:                         4.18.0-193.37.1.el8_2.x86_64
OS Image:                               Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)

falco chart version: 1.5.7
and solution with adding kernel-devel using MachineConfig works also for me but it takes lots of time to configure nodes. In tests with 8 nodes it takes from 20 to 30 minutes as nodes are rebooted when configuration needs to be changed through MachineConfig, please take a look on workflow for changing machine configuration.
In the time when machine is not configured and kernel-devel is not avialable falco pods are in CrashLoopBackOff.

May kernel-devel be added to falco docker image to enable building of missing modules?

[scorputty]

Found the solution for this problem. It looks like RedHat has removed the kernel-devel package from the 4.6 RHCOS installations. Just add the following to the machine config to let RHCOS install the kernel-devel pacakge again:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: worker-extensions
spec:
  extensions:
    - kernel-devel

And one for the master machine config:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: master-extensions
spec:
  extensions:
    - kernel-devel

Thanks for sharing, this solved my days long quest :-)