k8s-audit: Add custom fields to audit events

[sboschman]

Motivation

Instead of running a falco + falcosidekick instance for each of our self-hosted k8s clusters to add extra metadata like the k8s cluster name to the events, I would prefer to run a central falco cluster capable of ingesting k8s audit events from all of our clusters. Maintaining a per cluster falco instance becomes cumbersome.

Feature

Some way to add additional/extra/custom fields to each k8s audit event, just like falcosidekick can do, based on metadata sent with the request to the k8saudit plugin. As we are limited by what the k8s webhook audit backend can do, I am thinking of using http headers to supply additional metadata. This setup allows running a 'centralised' falco (cluster) ingesting k8s audit events from multiple k8s clusters without loosing for example the origin k8s cluster name.

Hopefully the custom fields added by the k8saudit plugin can be picked up by the solution for falco #2127

Alternatives

Currently running a falco instance for each k8s cluster, with a falcosidekick instance next to each falco instance. Falco ingests the k8s audit events, and pushes them to the falcosidekick instance. This falcosidekick instance adds the additional cluster specific labels before forwarding events to another falcosidekick instance used for falcosidekick ui and forwarding to another system. This setup becomes quite complex to maintain.

Additional context

As described in falco #2289:

My goal is to run a central falco (cluster) ingesting k8saudit events (or other cloud based events supported by falco plugins). A k8saudit event though misses important metadata to be able to trace the event back to the cluster (falcosecurity/falco#1704), like a cluster name/id.

The only way I see to do this, is by using the client authentication to provide additional metadata. A (reverse)proxy/ingress controller is responsible for handling authentication and pass additional metadata based on the authentication to falco by setting additional http headers. This is usually a pretty standard feature of a proxy (e.g. X-Forward-For headers). For example we can use mtls with k8s cluster specific client certs to authenticate. The proxy can forward the certificate subject field to the k8saudit plugin, and we can encode the cluster name in the subject for example.

I am looking into extending the k8saudit plugin with support to grab additional fields from http headers. Passing these values along the plugin event processing pipeline gets a bit ugly though. Only option seems to be altering the event from the http body with the header metadata. This unfortunately seems to mean the field names are fixed, as required by the extractor part.
Last challenge is to actually get the extra fields in the output, apart from editing all rules, the -p options seems atm the only way (related falcosecurity/falco#1704 (comment)). See falcosecurity/falco#2127 as well for additional output fields.
Downside of this approach is that each plugin requires a dedicated falco cluster, as the -p flag (with %field markers) is specific to each plugin. Falco validates the -p argument very early on, even before excluding rules with the -T/-t flags. E.g. adding a '%ka.auditid' with -p fails even if you only enable k8s rules (-t k8s).

Another downside is that the extra metadata is added to the output field. A better alternative is to only add these fields to the 'output_fields', not the 'output'. Just like falcosidekick allows to set 'custom fields', which are only added to the 'output_fields'.
Furthermore it would be ideal to be able to add 'custom' named fields to the output_fields of the alert. The additional fields a user wants to set, based on authentication/headers, is completely up to the user. Especially if you are forwarding the alerts with falcosidekick to other tools, the extra output_fields basically become 'labels' standardised across the organisation. E.g. alerts are forwarded as log lines to a log aggregation system. Each log line has a number of fixed labels so its origin can easily be found. The kubernetes cluster of origin is just one possibility, but one could add labels like division, project or team.

[jasondellaluce]

Sorry for being late. I'm gonna try to summarize all your proposals just to make sure we're in the same page:

• Including some/all HTTP headers of the api server webhooks alongside their body payload inside each event generated by the k8s_audit source
• Extracting the HTTP headers info as Falco rule fields either through the json plugin or with k8saudit-native fields
• Adding custom fields to the output_fields entry of each Falco rule on top of the ones used in the rule's output
• Adding custom "label-like" fields in each Falco rule (generic marker strings that are not strictly Falco fields)

[sboschman]

Ah yeah perhaps I should have included a summary of my ramblings.

I think the summary would come down to the following two proposals:

• optionally include the value of HTTP headers as user definable (custom) fields to each event generated by the k8s_audit plugin
• add extra fields to the output_fields entry of each Falco rule on top of the ones used in the rule's output

The second proposal is already requested in falco #2127. Of course it depends on the actual solution chosen for 2127, but I am hoping the first proposal would work nicely together with proposal two.

The focus of this issue is the HTTP headers for the k8s_audit plugin. But the implementation also allows other plugins, like the k8saudit-eks plugin to enhance each event with additional fields. The k8s_audit plugin uses HTTP headers as input, but I can imagine the k8saudit-eks plugin using the aws api to retrieve additional custom fields a user might want to add to each event (the additional fields are not part of the k8s audit event itself as defined by kubernetes, but knowing the audit event source the aws api might be able to supply more context of the event, for example the cluster name, or the aws region/zone the cluster is deployed in).

Trying to express the flow of things:

k8s part:
k8s api server -> kubeconfig with auth -> k8s audit event -> (a compatible audit webhook endpoint)

proxy (up to the user to pick one and configure it, so not a k8s_audit plugin problem to solve):
k8s audit event -> reverse proxy -> add custom headers, for example based on the auth supplied from kubeconfig -> forward headers and k8s audit event -> k8s_audit plugin embedded webserver

k8s_audit plugin:
receive headers + k8s audit event -> transform headers to extra fields -> parse k8s audit event, just as it works now -> merge extra fields and parsed fields -> output 'falco event' (SDK PushEvent object) for rule evaluation

falco engine:
PushEvent from plugin sdk/framework -> evaluate rules -> create falco alerts/events -> enhance output_fields of alert/event with extra variables (second proposal) -> emit alert/event

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.