Discussion: Updating the default falco-driver-loader image

[LucaGuerra]

As we know we have three main avenues install the system instrumentation to get live syscall info into Falco:

• The kernel module, for most kernel versions starting from selected 2.6.xx
• The classic eBPF probe, for selected versions of kernels 4.14 and above
• The modern eBPF probe, for kernel versions 5.8 and above with BTF support (if you use this, you have none of the issues that are described below, but relatively few distros support that as of today)

The first two are the most compatible but require building the drivers or downloading a prebuilt for your kernel. While we have thousands of prebuilts it is sometimes necessary to build your own driver. If you're using a container-based install, this is what the falco-driver-loader image attempts to do.

That image has a noble purpose, to try and build the driver for the live system it is on provided that kernel headers are present, but it's not without issues. For historical reasons, a ton of old compilers are still installed in a relatively old image (Debian Buster) to retain compatibility with older kernels (3.x and below). However, that image is unable to build the driver for newer versions (kernel > 5.0 and perhaps some 4.xx as well). In addition, if you try to run the image through a SCA / vulnerability scanner tool it will alert about maybe hundreds of vulnerabilities which are present in very old compilers and accompanying libs that are necessary for older distributions; for all practical security purposes those are all false positives but they would show up flashing on a security status dashboard and would make compliance experts feel uneasy.

The sad fact is that it's not even possible to build an image that works for everyone. In fact, even driverkit has resorted to using several different images to to this job.

This led me to try and look into the compiler versions we actually use when creating our prebuilt and trying to understand if we want to upgrade the default driver loader image. To learn which compilers we may want to put there I ran a little experiment: took all the current driverkit configs we use and looked at all the image selections. The results are as follows (after removing duplicate kernel versions):

Count Name
----- ----
    1 falcosecurity/driverkit-builder-any-x86_64_gcc10.0.0_gcc9.0.0:latest
 1457 falcosecurity/driverkit-builder-any-x86_64_gcc12.0.0_gcc11.0.0:latest
  399 falcosecurity/driverkit-builder-any-x86_64_gcc8.0.0_gcc6.0.0_gcc5.0.0_gcc4.9.0_gcc4.8.0:latest  <<-- this is the current Falco image
    1 falcosecurity/driverkit-builder-centos-x86_64_gcc4.8.5:latest

Basically, we use the gcc 11-12 image most of the times. In fact, as a rule of thumb, we use the gcc11-12 image for 5.x and 6.x while we use the old image for 4.x and below. So, we're recently picking up many variations of kernel 5.x because it seems like it's the most widely supported at the moment (Ubuntu, Amazon Linux, some recent RHEL and CentOS...) while 4.x versions are not seeing many updates.

I believe it may be time to upgrade the base image to something like this: https://github.com/falcosecurity/driverkit/blob/master/docker/builders/builder-any-x86_64_gcc12.0.0_gcc11.0.0.Dockerfile since building on 5.x and 6.x seems to be becoming more important than the previous versions. If we could also add another gcc version that is compatible with some 4.x I believe it would work even better.

This issue acts as a place of discussion about what we want to do. There may be more ideas, and even more things we want to do for people that cannot use this base image, but I did want to open this discussion.

/kind design

[poiana]

There is not a label identifying the kind of this issue.
Please specify it either using /kind <group> or manually from the side menu.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[LucaGuerra]

I can also confirm that we use GCC 8 for all kernels 4.x, but the compiler version is not enough to be sure about whether a specific image will work with a specific kernel version.

[tspearconquest]

Double confirming the last comment. I tried to build and load the dkms driver from source with gcc8, 9, 10, and 11, and it would not work with any for me on Kernel 5.4 (AKS K8s 1.24 on Ubuntu 18.04), so I went back to the pre-built driver.

[LucaGuerra]

Kernel 5.4 (AKS K8s 1.24 on Ubuntu 18.04)

There was I, thinking that Ubuntu 18.04 was on kernel 4.x

Anyways, yes. Unfortunately it's practically impossible to create an image that is able to build the kmod or classic eBPF probe for every possible kernel version and distribution, and even if we could somehow do it it'd be extremely large in size making it impractical. And yet, we want to distribute a Falco image that is able to build it for some popular Linux distributions to make it easier to run Falco.

[FedeDP]

Well, building the eBPF probe should work with any clang version >=7, therefore we should always be able to build it.
The only real issue here is building the kmod, IMHO.
I agree with the idea of bumping the falco-driver-loader toolchains to eg: gcc 11/12 and a more recent clang, we should cover most of our users needs (that have shifted from old kernel {3,4}.x to newer {5,6}.x).

[Andreagit97]

I think we need to tackle this issue ASAP, in the meanwhile we have already collected at least 4 issue regarding this topic :/ :point_up:

[FedeDP]

👍 from me; open to give any help needed!

[tspearconquest]

Meanwhile, I've upgraded our clusters to 1.25 and 1.26. On AKS, this means we're now on Ubuntu 22.04 which has a kernel that can use the modern bpf probe. Happy to still test out fixes for the build issues with any of the three options though! Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Federico Di Pierro ***@***.***> Sent: Thursday, June 15, 2023 4:28:25 AM To: falcosecurity/falco ***@***.***> Cc: Thomas Spear ***@***.***>; Comment ***@***.***> Subject: Re: [falcosecurity/falco] Discussion: Updating the default falco-driver-loader image (Issue #2489) CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. 👍 from me; open to give any help needed! — Reply to this email directly, view it on GitHub<#2489 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATRTFZ2SWD34TYL4L4N3SXTXLLITTANCNFSM6AAAAAAW3XHZUY>. You are receiving this because you commented.Message ID: ***@***.***>