make the falco driver loader downloads first, then fallback to compilation #1599

leodido · 2021-04-02T15:30:48Z

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

ALL (since we don't have a scripts area).

What this PR does / why we need it:

This PR inverts the logic of the falco-driver-loader script.

The new logic (expand 👇 ) is:

kernel module case (default one)

Unloading the Falco module (falco.ko) if present
If the Falco module is present but the script was unable to remove it, exit (and try to use it, notice that the driver version is not checked)
Try to add a Falco module from the system modules directory - /lib/modules/$(uname -r) - exit in case operation succeeds (notice that the driver version is not checked)
Look for a Falco module - eg., falco_{distro}_{kernel_release}_{kernel_version}.ko - in ${HOME}/.falco/ - insert it if present (notice that the driver version is not checked)
Try to find a suitable prebuilt kernel module on https://download.falco.org
Try to compile the kernel module on the fly if not found a prebuilt one

eBPF probe case

Look for a Falco eBPF probe - eg., falco_{distro}_{kernel_release}_{kernel_version}.o - in ${HOME}/.falco/ - go to 4 (notice that the driver version is not checked)
Try to find a suitable prebuilt eBPF probe on https://download.falco.org
Try to compile the eBPF probe on the fly if not found a prebuilt one
Symlink ${HOME}/.falco/falco_{distro}_{kernel_release}_{kernel_version}.o to ${HOME/.falco/falco-bpf.o

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

The reason for this PR is to reduce the number of cases in which the Falco users need to wait for a Falco container to build a driver on the fly.

This will also speed up the boot of containerized Falco (assuming a prebuilt exists for the host machine, we now have ~4K prebuilt Falco drivers at https://download.falco.org).

The only cases in which https://download.falco.org will not be hit by a request will be:

a Falco eBPF probe already exists in ${HOME}/.falco/falco_{distro}_{kernel_release}_{kernel_version}.o
a Falco kernel module already exists in ${HOME}/.falco/falco_{distro}_{kernel_release}_{kernel_version}.ko
a Falco kernel module is already inserted (and has not been removed)
a Falco kernel module is already present in the modules directory

In this way, the Falco community could also have more real statistics about the Falco usage and usage patterns by looking at the logs of https://download.falco.org.

Notes that this PR also moves out from the load_bpf_probe function the target detection logic in its place (get_target_id) for minikube and COS.

Does this PR introduce a user-facing change?:

update: falco-driver-loader tries to download a Falco driver before then compiling it on the fly for the host
BREAKING CHANGE: SKIP_MODULE_LOAD env variable no more disables the driver loading (use SKIP_DRIVER_LOADER env variable introduced in Falco 0.24)

…IP_DRIVER_LOADER When we started to implemented 20200506-artifacts-scope-part-2 proposal , among a million other things, we renamed `SKIP_MODULE_LOAD` to `SKIP_DRIVER_LOADER`. We reatained compatibility with `SKIP_MODULE_LOAD` for a bunch of releases. Now, after 9 months have passed I think it's time to completely deprecate it. Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

…mpile it on-the-fly Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

…o compile one Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

leodido · 2021-04-02T15:31:13Z

Ready for review even if WIP

leodido · 2021-04-02T15:31:53Z

/milestone 0.29.0

leogr

Overall, it seems very good to me 👏

I have just found an issue that should easily be fixable (see the comment below about the need of mounting debugfs).

Other comments are just small thing, not really needed :)

docker/falco/docker-entrypoint.sh

leogr · 2021-04-06T16:42:26Z

scripts/falco-driver-loader

-	echo "* Mounting debugfs"
-
-	if [ ! -d /sys/kernel/debug/tracing ]; then
-		mount -t debugfs nodev /sys/kernel/debug
-	fi


Under some circumstances, mounting debugfs is required to run Falco with the eBPF probe, but this piece seems to be lost in the refactoring.

The driver-loader/integration test is failing for the missing mount (logs here).
Indeed, without that, Falco fails to load the probe and exits immediately:

$ FALCO_BPF_PROBE="" falco Tue Apr 6 16:33:51 2021: Falco version 0.27.0-86+787eebb (driver version 5c0b863ddade7a45568c0ac97d037422c9efb750) Tue Apr 6 16:33:51 2021: Falco initialized with configuration file /etc/falco/falco.yaml Tue Apr 6 16:33:51 2021: Loading rules from file /etc/falco/falco_rules.yaml: Tue Apr 6 16:33:51 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Tue Apr 6 16:33:52 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml: Tue Apr 6 16:33:52 2021: Unable to load the driver. Tue Apr 6 16:33:52 2021: Runtime error: failed to open event raw_syscalls/sys_enter. Exiting.

Instead, after running mount -t debugfs nodev /sys/kernel/debug then Falco works as expected.

So, my advice here is just to restore those lines.

This is needed in systems where raw tracepoints are not available.
Agreed with @leogr that this needs to be reverted.

See:
https://github.com/falcosecurity/libs/blob/master/userspace/libscap/scap_bpf.c#L474-L485

I remember we discussed this privately. Anyways, I'm gonna put it back ASAP.

Thanks Leo, I agree we need to do this but I would prefer to do it in a second moment if you agree

Ideally, the correct place for this mount would be in Falco binary when the FALCO_BPF_PROBE env variable is present.

I'll revert it back for now, anyways.

scripts/falco-driver-loader

fntlnz · 2021-04-07T12:57:53Z

scripts/falco-driver-loader

-	echo "* Mounting debugfs"
-
-	if [ ! -d /sys/kernel/debug/tracing ]; then
-		mount -t debugfs nodev /sys/kernel/debug
-	fi


This is needed in systems where raw tracepoints are not available.
Agreed with @leogr that this needs to be reverted.

See:
https://github.com/falcosecurity/libs/blob/master/userspace/libscap/scap_bpf.c#L474-L485

scripts/falco-driver-loader

…iver-loader Co-authored-by: Lorenzo Fontana <lo@linux.com> Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

This is needed in systems where raw tracepoints are not available. Anyways, since this is needed when the inspector open (and actually loads) the eBPF probe, ideally the mount should not be done by this script but rather from Falco, or from Falco libs. Otherwise, users building the eBPF probe theirseleves and not using this script (and having a kernel without raw tracepoints) may need to mount this fs theirselves. Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

leodido · 2021-04-08T15:35:38Z

/milestone 0.28.0

leogr

LGTM! 👍

poiana · 2021-04-08T16:02:10Z

LGTM label has been added.

Git tree hash: f6ffb1c68c7f69ec073524101d2807b1b341aba3

poiana · 2021-04-08T16:20:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fntlnz, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [fntlnz,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

leodido added 4 commits March 29, 2021 10:50

update(scripts): look for a prebuilt Falco module before trying to co…

8e64631

…mpile it on-the-fly Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

update(scripts): look for a prebuilt Falco eBPF probe before trying t…

fb7a7e0

…o compile one Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

chore(scripts): remove banner about BPF JIT kernel config option

787eebb

Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

poiana added kind/feature do-not-merge/work-in-progress release-note dco-signoff: yes labels Apr 2, 2021

poiana requested review from fntlnz and leogr April 2, 2021 15:30

poiana added the size/L label Apr 2, 2021

poiana added this to the 0.29.0 milestone Apr 2, 2021

leogr reviewed Apr 6, 2021

View reviewed changes

fntlnz suggested changes Apr 7, 2021

View reviewed changes

poiana assigned fntlnz Apr 7, 2021

leodido and others added 2 commits April 8, 2021 15:07

chore(scripts): better default values in the help message of falco-dr…

85e2785

…iver-loader Co-authored-by: Lorenzo Fontana <lo@linux.com> Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>

leodido changed the title ~~wip: make the falco driver loader downloads first, then fallback to compilation~~ make the falco driver loader downloads first, then fallback to compilation Apr 8, 2021

poiana removed the do-not-merge/work-in-progress label Apr 8, 2021

poiana modified the milestones: 0.29.0, 0.28.0 Apr 8, 2021

leogr approved these changes Apr 8, 2021

View reviewed changes

poiana assigned leogr Apr 8, 2021

poiana added the lgtm label Apr 8, 2021

poiana added the approved label Apr 8, 2021

fntlnz approved these changes Apr 8, 2021

View reviewed changes

poiana merged commit 04110b0 into master Apr 8, 2021

poiana deleted the update/falco-driver-loader-downloads-first branch April 8, 2021 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make the falco driver loader downloads first, then fallback to compilation #1599

make the falco driver loader downloads first, then fallback to compilation #1599

leodido commented Apr 2, 2021 •

edited

Loading

leodido commented Apr 2, 2021

leodido commented Apr 2, 2021

leogr left a comment

leogr Apr 6, 2021

fntlnz Apr 7, 2021

leodido Apr 7, 2021 •

edited

Loading

fntlnz Apr 8, 2021

leodido Apr 8, 2021

fntlnz Apr 7, 2021

leodido commented Apr 8, 2021

leogr left a comment

poiana commented Apr 8, 2021

poiana commented Apr 8, 2021

make the falco driver loader downloads first, then fallback to compilation #1599

make the falco driver loader downloads first, then fallback to compilation #1599

Conversation

leodido commented Apr 2, 2021 • edited Loading

leodido commented Apr 2, 2021

leodido commented Apr 2, 2021

leogr left a comment

Choose a reason for hiding this comment

leogr Apr 6, 2021

Choose a reason for hiding this comment

fntlnz Apr 7, 2021

Choose a reason for hiding this comment

leodido Apr 7, 2021 • edited Loading

Choose a reason for hiding this comment

fntlnz Apr 8, 2021

Choose a reason for hiding this comment

leodido Apr 8, 2021

Choose a reason for hiding this comment

fntlnz Apr 7, 2021

Choose a reason for hiding this comment

leodido commented Apr 8, 2021

leogr left a comment

Choose a reason for hiding this comment

poiana commented Apr 8, 2021

poiana commented Apr 8, 2021

leodido commented Apr 2, 2021 •

edited

Loading

leodido Apr 7, 2021 •

edited

Loading