Why are all k8s.*.* fields are null?

[tomklino]

I've added to the output of all rules the fields:

• k8s.pod.name
• k8s.pod.id
• k8s.pod.labels

But every time an event is logged these values come out as null

for example:

{"output":"12:20:10.995144237: Notice A shell was spawned in a container with an attached terminal (user=root k8s_falco_falco-b7fr6_security_e59d29ac-8208-11e9-bd23-1229902c6cc4_0 (id=1231e98ac26a) shell=bash parent=<NA> cmdline=bash terminal=34816 podid=<NA> podlabels=<NA>)","priority":"Notice","rule":"Terminal shell in container","time":"2019-05-29T12:20:10.995144237Z", "output_fields": {"container.id":"1231e98ac26a","container.name":"k8s_falco_falco-b7fr6_security_e59d29ac-8208-11e9-bd23-1229902c6cc4_0","evt.time":1559132410995144237,"k8s.pod.id":null,"k8s.pod.labels":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":null,"proc.tty":34816,"user.name":"root"}}

I'm have looked through the docs to see if there is some configuration I needed to set to get this to work, but havn't found anything

[tomklino]

so turns out I left those arguments out by mistake:

args: [ "/usr/bin/falco", "--cri", "/host/run/containerd/containerd.sock", "-K", "/var/run/secrets/kubernetes.io/serviceaccount/token", "-k", "https://$(KUBERNETES_SERVICE_HOST)", "-pk"]

[tomklino]

I have to re-open that issue.
Even now with the arguments for Kubernetes enabled, for some of the events, all k8s.* fields are null. They are critical in my case for filtering events like privileged containers, and I need to understand if this is a bug or if there is something additional in the configurations/setup I need to add.

Here is an example:

11:07:47.478740297: Warning Sensitive file opened for reading by non-trusted program (user=root program=executor file=/etc/shadow parent=executor gparent=sh ggparent=<NA> gggparent=<NA> podid=<NA> podlabels=<NA> containerimage=gcr.io/kaniko-project/executor:debug) k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238 k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238 k8s.ns=<NA> k8s.pod=<NA> container=055736f1f238

And sometimes, for the same event, k8s.* fields are as expected:

10:43:42.051510527: Warning Sensitive file opened for reading by non-trusted program (user=root program=executor file=/etc/pam.d/other parent=executor gparent=sh ggparent=<NA> gggparent=<NA> podid=c2b94f06-82c7-11e9-bd23-1229902c6cc4 podlabels= containerimage=gcr.io/kaniko-project/executor:debug) k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2 k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2 k8s.ns=default k8s.pod=runner-tgajjktl-project-277-concurrent-0g6g7m container=deb108bd49d2

[fntlnz]

@tomklino can you give us more info about your environment?

Environment:

• Falco version (use falco --version):
• System info
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools (e.g. in kubernetes, rpm, deb, from source):
• Others:

[fntlnz]

/triage support

[tomklino]

I would love to:

falco version 0.15.0

# falco --support | jq .system_info
{
  "machine": "x86_64",
  "nodename": "falco-wvxqm",
  "release": "4.15.0-1039-aws",
  "sysname": "Linux",
  "version": "#41-Ubuntu SMP Wed May 8 10:43:54 UTC 2019"
}

We are running kubernetes on aws - falco is running as a daemonset in a privileged container and tolerations to allow it to run on master nodes
Using the falco image as provided (falcosecurity/falco:0.15.0) for the container image

# uname -a
Linux falco-wvxqm 4.15.0-1039-aws #41-Ubuntu SMP Wed May 8 10:43:54 UTC 2019 x86_64 GNU/Linux

# cat /host/etc/os-release
Defaulting container name to falco.
Use 'kubectl describe pod/falco-wvxqm -n security' to see all of the containers in this pod.
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[tcotav]

FWIW -- I'm also seeing this on AWS kubernetes/EKS. Run with daemonset.

# falco --version
falco version 0.15.3

# falco --support | jq .system_info
Fri Jun 14 22:20:12 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Jun 14 22:20:12 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Jun 14 22:20:13 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Jun 14 22:20:13 2019: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
{
  "machine": "x86_64",
  "nodename": "falco-daemonset-wqtdh",
  "release": "4.14.106-97.85.amzn2.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Fri Mar 15 17:07:54 UTC 2019"
}

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

# uname -a
Linux falco-daemonset-wqtdh 4.14.106-97.85.amzn2.x86_64 #1 SMP Fri Mar 15 17:07:54 UTC 2019 x86_64 GNU/Linux

[rastut]

Here the same, running on GKE 1.12.6

falco version 0.15.3

{
  "machine": "x86_64",
  "nodename": "gke-standard-cluster-2-default-pool-3e4415e6-gpnc",
  "release": "4.14.119+",
  "sysname": "Linux",
  "version": "#1 SMP Tue May 14 21:04:23 PDT 2019"
}

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

uname -a
Linux gke-standard-cluster-2-default-pool-3e4415e6-gpnc 4.14.119+ #1 SMP Tue May 14 21:04:23 PDT 2019 x86_64 GNU/Linux

Some times the events come with the data populated but sometimes doesn't contain the k8s data:

{"output":"12:17:41.118240629: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=default k8s.pod=nginx-falco container=ac0705e08932 shell=bash parent=docker-runc cmdline=bash terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2019-06-18T12:17:41.118240629Z", "output_fields": {"container.id":"ac0705e08932","evt.time":1560860261118240629,"k8s.ns.name":"default","k8s.pod.name":"nginx-falco","proc.cmdline":"bash","proc.name":"bash","proc.pname":"docker-runc","proc.tty":34816,"user.name":"root"}}

{"output":"12:17:49.061394964: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=<NA> k8s.pod=<NA> container=db69b394c785 shell=bash parent=docker-runc cmdline=bash terminal=34816)","priority":"Notice","rule":"Terminal shell in container","time":"2019-06-18T12:17:49.061394964Z", "output_fields": {"container.id":"db69b394c785","evt.time":1560860269061394964,"k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":"docker-runc","proc.tty":34816,"user.name":"root"}}

[tcotav]

fwiw, switching my ds deploy yaml to use :dev and I got the info

 18:11:58.187352558: Notice A shell was spawned in a container with an attached terminal (user=root k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde shell=bash parent=docker-runc cmdline=bash terminal=34816) k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde

18:12:07.147066927: Error File below /etc opened for writing (user=root command=touch normal.txt parent=bash pcmdline=bash file=/etc/normal.txt program=touch gparent=<NA> ggparent=<NA> gggparent=<NA>) k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde k8s.ns=default k8s.pod=falco-daemonset-ws254 container=6a5c0fa2afde

[fntlnz]

@tomklino as @tcotav is reporting this was solved in #731 would you be able to confirm it fixes for your case too?

[tomklino]

I will update Falco on our environment to 0.17.0 later this week and will get back to you

And thank you for checking in with me :-)

[fntlnz]

/close

I'm closing this but @tomklino please continue the discussion if you find anything new.

[poiana]

@fntlnz: Closing this issue.

In response to this:

/close

I'm closing this but @tomklino please continue the discussion if you find anything new.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tomklino]

Hi guys, so I updated to 0.17.0 a few days ago and let it run for a while.

I'm sorry to be the bearer of bad news, but that is still happening in 0.17.0:

14:42:35.118505676: Notice Privileged container started (user=<NA> k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5 image=nuvo/orca:0.9.3 podid=<NA> podlabels=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5 k8s.ns=<NA> k8s.pod=<NA> container=745ebd332ed5

If there's any info you need from me that can help I'll be happy to get it for you

[leodido]

Ok, reopening it and adding to milestone 0.18.0.

/kind bug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomklino]

Hi, just updated to 0.18.0 - unfortunately still experiencing the error

Is there any progress on this? Can I help in some way?

[jjo]

FYI we're facing the same issue with 0.18.0 as daemonset, kubernetes-1.15 on kops AMIs
/cc @dbarranco @jbianquetti-nami

[leodido]

Linking to #925 since that contains more info.
Closing this. Please go into #925 for further info.