Output fields not filled with info showing N/A

[mazzy89]

Describe the bug

cmdline is empty and other fields too. It is hard to know from where this is generated.

How to reproduce it

Run Falco 0.20.0 in k8s 1.15.9 with OS Flatcar stable 2303.4.0

Expected behaviour

I would imagine having all the fields filled.

Screenshots

Environment

• Falco version:

Falco version: 0.20.0+d77080a

• System info:

machine":"x86_64","
nodename":"falco-dchpq","release":"4.19.95-flatcar","sysname":"Linux","version":"#1 SMP Sat Feb 8 07:25:12 -00 2020

• Cloud provider or hardware configuration: AWS
• OS:

• Kernel:

• Installation method:

official helm chart

    ebpf:
      ## NOTE: for the moment let's disable eBPF due to a bug in 0.20.0
      ## The bug fix will be wired in the 0.20.1 and the upstream Helm chart
      ## must be fixed too
      # Enable eBPF support for Falco
      enabled: false
      settings:
        # Needed to correctly detect the kernel version for the eBPF program
        # Set to false if not running on Google COS
        mountEtcVolume: false
    falco:
      # Whether to output events in json or text
      jsonOutput: true
      # Every second, Falco reads system call event counts that are populated by the kernel module/eBPF program.
      # The reading includes the number of system calls processed, and most importantly,
      # the number of times the kernel tried to write system call information to the shared buffer
      # between the kernel and user space, but found the buffer was full.
      # These failed write attempts are considered dropped system call events.
      # Ignore syscall event drops because of an issue in Falco
      #
      # These drops spam DD and cause high cost of logs
      syscallEventDrops:
        actions: []

Additional context

[fntlnz]

I think this is due to volume mounts in the helm chart but I will need to double check.

Also I think that this helm release is suffering this bug that was fixed here: #981

@nestorsalceda @leogr can you please take a look, you're way more familiar than me with the helm chart!

[fntlnz]

/assign @leogr
/assign @nestorsalceda

[mazzy89]

Posting here the bootstrap logs after having a talk with @leogr

falco-vzcpj falco * Setting up /usr/src links from host
falco-vzcpj falco ln: failed to create symbolic link '/usr/src//host/usr/src/*': No such file or directory
falco-vzcpj falco * Unloading falco-probe, if present
falco-vzcpj falco * Running dkms install for falco
falco-vzcpj falco
falco-vzcpj falco Kernel preparation unnecessary for this kernel.  Skipping...
falco-vzcpj falco
falco-vzcpj falco Building module:
falco-vzcpj falco cleaning build area....
falco-vzcpj falco make -j4 KERNELRELEASE=4.19.95-flatcar -C /lib/modules/4.19.95-flatcar/build M=/var/lib/dkms/falco/0.20.0+d77080a/build...........
falco-vzcpj falco cleaning build area....
falco-vzcpj falco Kernel cleanup unnecessary for this kernel.  Skipping...
falco-vzcpj falco
falco-vzcpj falco DKMS: build completed.
falco-vzcpj falco
falco-vzcpj falco falco-probe.ko:
falco-vzcpj falco Running module version sanity check.
falco-vzcpj falco  - Original module
falco-vzcpj falco    - No original module exists within this kernel
falco-vzcpj falco  - Installation
falco-vzcpj falco    - Installing to /lib/modules/4.19.95-flatcar/kernel/extra/
falco-vzcpj falco mkdir: cannot create directory '/lib/modules/4.19.95-flatcar/kernel/extra': Read-only file system
falco-vzcpj falco cp: cannot create regular file '/lib/modules/4.19.95-flatcar/kernel/extra/falco-probe.ko': No such file or directory
falco-vzcpj falco
falco-vzcpj falco depmod...
falco-vzcpj falco
falco-vzcpj falco DKMS: install completed.
falco-vzcpj falco * Trying to load a dkms falco-probe, if present
falco-vzcpj falco falco-probe found and loaded in dkms

[leogr]

If I am not wrong, it looks like that the startup script is not installing the falco-probe.ko (because the OS is read-only), but it seems to be already present:

falco-vzcpj falco    - Installing to /lib/modules/4.19.95-flatcar/kernel/extra/
falco-vzcpj falco mkdir: cannot create directory '/lib/modules/4.19.95-flatcar/kernel/extra': Read-only file system
falco-vzcpj falco cp: cannot create regular file '/lib/modules/4.19.95-flatcar/kernel/extra/falco-probe.ko': No such file or directory
falco-vzcpj falco
falco-vzcpj falco depmod...
falco-vzcpj falco
falco-vzcpj falco DKMS: install completed.
falco-vzcpj falco * Trying to load a dkms falco-probe, if present
falco-vzcpj falco falco-probe found and loaded in dkms

@mazzy89 If so, how has it been installed?

PS @fntlnz I guess this issue is not related to the helm chart.

[mazzy89]

@leogr maybe it has been installed initially at the first run of Falco in that node.

[mazzy89]

After spent some time to check with @leogr (thanks a lot for your time btw), we have found out that the issue is very likely connected to Flatcar and the fact that falco-probe.ko is missing and hence the instrumentation is not complete at the level of the OS.

In Flatcar the dir is read-only so the kernel module can't be installed under there.

I've tried also to enable eBPF but it did not help neither.

[mazzy89]

I'm going to try this https://docs.flatcar-linux.org/os/kernel-modules/ to empower the container to install custom module from the container itself into the OS

[mazzy89]

Probably this is going to require a change in the helm chart if it works modyfing /lib/modules to something else writable. I'll post the results here.

[mazzy89]

I booted the OS having in place the following unit

[Unit]
Description=Custom Kernel Modules
Before=local-fs.target
ConditionPathExists=/opt/modules

[Mount]
Type=overlay
What=overlay
Where=/usr/lib64/modules
Options=lowerdir=/usr/lib64/modules,upperdir=/opt/modules,workdir=/opt/modules.wd

[Install]
WantedBy=local-fs.target

Despite I can write now under /lib/modules from the OS, the container still can't access to the fs and I still get the error:

   - Installing to /lib/modules/4.19.95-flatcar/kernel/extra/
mkdir: cannot create directory '/lib/modules/4.19.95-flatcar/kernel/extra': Read-only file system
cp: cannot create regular file '/lib/modules/4.19.95-flatcar/kernel/extra/falco-probe.ko': No such file or directory

I'm not sure why this. Definitely I'm missing something.

Also, I was thinking to build the Kernel module from the OS. Indeed the OS supports this use case.

[leogr]

@mazzy89 I have realized right now that /lib/modules is mounted in read-only by the helm chart:

Mounts:
      /dev/shm from dshm (rw)
      /etc/falco from config-volume (rw)
      /etc/falco/certs from certs-volume (ro)
      /host/boot from boot-fs (ro)
      /host/dev from dev-fs (ro)
      /host/lib/modules from lib-modules (ro)
      /host/proc from proc-fs (ro)
      /host/run/containerd/containerd.sock from containerd-socket (rw)
      /host/usr from usr-fs (ro)
      /host/var/run/docker.sock from docker-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from falco-token-xj8wv (ro)

[mazzy89]

Thanks @leogr. Once I changed the Daemonset the falco module was able to be installed locally.

[mazzy89]

Although the kernel module is now loaded correctly still some logs are reported in this way:

so we have validated at least that the root cause is not the presence of the module

[mazzy89]

I've wired a PR in the upstream Helm chart. Also, I would ask here if you guys can point me out to a place where I can mention how to treat the case of installing the Falco Kernel module in OS like CoreOS/Flatcar. Happy to contribute in the documentation but just not sure where to add this information.

[leogr]

Hey @mazzy89 we suspect this issue might be a dup of #925 that has been fixed by #1099
A new version with the fix has not been released yet, but - if you wanna try - you can use the master branch.

[mazzy89]

Cool. Thank you @leogr. I'll give a try.

[leodido]

Hey @mazzy89 to grab the release containing that fix:

• https://bintray.com/falcosecurity/bin-dev/falco/0.21.0-7%2B7b44aaf
• https://bintray.com/falcosecurity/rpm-dev/falco/0.21.0-7%2B7b44aaf
• https://bintray.com/falcosecurity/deb-dev/falco/0.21.0-7%2B7b44aaf

And remember to use the new --disable-cri-async flag.

Hope to have helped!

[leodido]

We've been working on PR #1133 lately.

It (should definitely solve) solves the issue you're reporting as you can see from the following outputs (eg., k8s.pod, k8s.ns, etc. populated).

Notice Setuid or setgid bit is set via chmod (fd=<NA> filename=/tmp/ciao mode=S_IROTH|S_IRGRP|S_IWUSR|S_IRU
SR|S_ISGID|S_ISUID user=root process=chmod command=chmod +s ciao container_id=9321c4213111 container_name=k8s_falco_falco-daemo
nset-xlsmb_default_b115a244-ee5e-4cf6-a7e7-d26502f79147_0 image=test-renzo:latest) k8s.ns=default k8s.pod=falco-daemonset-xlsmb
 container=9321c4213111 k8s.ns=default k8s.pod=falco-daemonset-xlsmb container=9321c4213111 k8s.ns=default k8s.pod=falco-daemon
set-xlsmb container=9321c4213111

Error File below /etc opened for writing (user=root command=touch ciao parent=bash pcmdline=bash
file=/etc/ciao program=touch gparent=sh ggparent=<NA> gggparent=<NA> container_id=9321c4213111 image=test-renzo)
k8s.ns=default k8s.pod=falco-daemonset-xlsmb container=9321c4213111 k8s.ns=default k8s.pod=falco-daemonset-xlsmb
container=9321c4213111 k8s.ns=default k8s.pod=falco-daemonset-xlsmb container=9321c4213111

Also, the issue is solved when running Falco as a docker container (and not in Kubernetes). As you can see from the following outputs, container_name and image are no more NA.

Notice A shell was spawned in a container with an attached terminal (user=root eloquent_hodgkin (id=fc9254e8c43b) shell=bash parent=<NA> cmdline=bash terminal=34816 container_id=fc9254e8c43b image=ubuntu)

Notice Setuid or setgid bit is set via chmod (fd=<NA> filename=/tmp/test-file mode=S_IROTH|S_IRGRP|S_IWUSR|S_IRUSR|S_ISGID|S_ISUID user=root process=chmod command=chmod +s test-file container_id=fc9254e8c43b container_name=eloquent_hodgkin image=ubuntu:18.04)

Would be cool if you give it a shoot using one of the temp artifacts available here.

Otherwise, you can try it as soon as we merge PR 1133 using the falcosecurity/falco:master docker image.

[mazzy89]

Hi @leodido thanks a lot to get back here. You are great guys.

Definitely I will give a try after holiday Easter. I just paused at the moment the work on Falco due to other priorities but I'll get back to it soon. I still have to wire the target for Flatcar 😇