Falco using tokens past their lifetime in Kubernetes 1.21

[plnordquist]

Describe the bug

In Kubernetes 1.21, the BoundServiceAccountTokenVolume feature is enabled by default. This feature replaces the mounted token secret from the past with a projected volume that includes an auto rotated token. With a default configuration, the Kubernetes API server monitors when tokens are used past their lifetime and annotates requests in the audit log with a stale-token annotation. Based on my testing, Falco does not load in rotated tokens and uses the first token mounted in the pod forever.

How to reproduce it

1. Deploy falco to Kubernetes with the Helm Chart
2. Wait 1 hour for the token to expire
3. Observe the audit log annotating requests from Falco with the stale-token annotation

Expected behaviour

Falco should load the rotated token instead of caching the initial token.

Environment

• Falco version: 0.30.0

• System info:

{
  "machine": "x86_64",
  "nodename": "falco-4wlp5",
  "release": "3.10.0-1160.42.2.el7.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Sep 7 14:49:57 UTC 2021"
}

• Cloud provider or hardware configuration:
• OS: CentOS 7

• Kernel: 3.10.0-1160.42.2.el7.x86_64

• Installation method: Kubernetes Helm Chart

Additional context

Example Audit log line:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "a6a52418-b225-43a2-b48b-56e489b9fbab",
  "stage": "RequestReceived",
  "requestURI": "/api/v1/watch/nodes?pretty=false",
  "verb": "watch",
  "user": {
    "username": "system:serviceaccount:services:falco",
    "uid": "146556d9-3753-4f47-922b-cd68f8597d1d",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:services",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/pod-name": ["falco-gg6rl"],
      "authentication.kubernetes.io/pod-uid": [
        "760c5c22-73c4-4da0-a21a-ca6e6b57513c"
      ]
    }
  },
  "sourceIPs": ["10.17.2.2"],
  "userAgent": "falcosecurity-libs",
  "objectRef": { "resource": "nodes", "apiVersion": "v1" },
  "requestReceivedTimestamp": "2021-11-10T23:26:08.128787Z",
  "stageTimestamp": "2021-11-10T23:26:08.128787Z",
  "annotations": {
    "authentication.k8s.io/stale-token": "subject: system:serviceaccount:services:falco, seconds after warning threshold: 1388501"
  }
}

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[kfox1111]

/remove-lifecycle stale

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[kfox1111]

/remove-lifecycle stale

[rdubya16]

Any update on this? We are seeing this as well.

[goroi]

We also experience the same issue. Is this a bug or falco has not been designed to reload the token ?

[ianmuge]

Any update on this?

[alacuku]

The current implementation of k8s client in falco does not reload the tokens. It has been implemented long before the token rotation was introduced in k8s. We are thinking to move the k8s client in a plugin using the official k8s client-go. But before that we still need to do some work on Falco and plugins ecosystem. For more info se the following tracking issue #2074. Once we have the multi-source support in Falco, we are good to move the k8s client in a plugin.

[jasondellaluce]

@alacuku, actually developing the k8s client as a plugin is not related to #2074, but to falcosecurity/libs#410 instead. Once we are able to enable plugins to extract fields from syscall events, then we can consider porting the k8s client to a plugin.

[alacuku]

Thanks @jasondellaluce!

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[ianmuge]

/remove-lifecycle stale

[ashuec90]

Hello @jasondellaluce Any update on this? We are using eks and we have to upgrade eks 1.20 to 1.21 by end of this month, and in eks 1.21 we need this otherwise we have to restart the falco pods everytime before the token is marked as stale.

Thanks

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[mike-stewart]

/remove-lifecycle stale

[gouthz]

Hi, Do we have any updates here? This is a blocker for us to upgrading our EKS cluster version to 1.23.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[kfox1111]

/remove-lifecycle stale

[Speeddymon]

Related issue: #2485

If you manually configure the deployment to use the projected volume (attempted via Kustomize post-render hook), falco crashes once the token is rotated. At least it restarts and will use the new token, but I'd really rather it not crash once an hour.

I could make it rotate once a day, which would at least reduce the crashing to once a day, but I'd also rather not see it crash then either!

[Andreagit97]

The new k8smeta plugin should solve this issue, give it a try with our dedicated helm chart and Falco 0.36.2 #2973 (comment)

[Andreagit97]

This should be solved by Falco 0.37.0! Feel free to reopen if this is still an issue