FR: streaming audit endpoint #53455

tallclair · 2017-10-04T19:02:12Z

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

Description:

Provide a pull-based stream for audit logging. This would look similar to a watch request to the API server, and provide a stream of audit events. I would not expect the stream to return any PREVIOUS events (i.e. no associated storage), it would only stream events that occurred after the stream was opened. This would be a new audit backend plugin.

The should also have some options to filter events. Minimally, there should be a namespaced version that only includes audit events tied to that namespace, so that RBAC can be used to grant subdivided access. Secondarily, it might be nice to be able request only events tied to a given user or resource.

Motivation:

The current audit backends (log & webhook) require very high privileges (file access to the master) or prior configuration (webhook). There are a number of use cases for dynamic and unprivileged access to the audit logs. Examples include:

audit2rbac - generate RBAC rules from audit logs. It would be nice to simply point this tool at a running cluster. With the namespace subdivision, an admin could grant developers working in a namespace access to their audit logs to use this tool, without granting the full cluster audit logs.
Debugging - related to the above point, API audit logs can be very useful for debugging, and this would provide an option to surface a subset of the logs to developers.
3rd party audit plugins - This isn't a strong use case as the stream would be less reliable than the push-based webhook, but there might be other use cases for plugging in logging tools to the stream.

Challenges:

Audit logs can be very noisy, so there is a risk that providing this stream (and potentially multiple connections) could have a performance impact. Subdividing the audit logs (namespaces, resources, etc.) would help. We could also potentially restrict this to metadata-level, though that would negatively impact the debugging use case (maybe a request parameter?). Also, the stream should not have the reliability guarantees as the webhook & log backends, if the stream is congested events will be dropped.

/cc @crassirostris @liggitt @destijl @sttts @kubernetes/sig-auth-feature-requests

mbohlool · 2017-10-05T20:56:54Z

cc @caesarxuchao

crassirostris · 2017-12-04T11:16:45Z

Another important use case (xref #56683):

In current e2e test audit events are read from the log file. However, there are a couple of problems:

In load tests, a lot of audit events are produced. Network problem when transferring GBs of data will cause the tests to fail
Log file can be rotated during the tests, which also triggers failure
Current implementation cannot be used in any environment, it's tightly coupled with the configuration in k8s.io/cluster/gce

Remove this tag once functionality from feature request kubernetes#53455 is implemented.

Automatic merge from submit-queue (batch tested with PRs 55360, 56444, 56687, 56791, 56802). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add DisabledForLargeClusters tag to audit tests. Remove this tag once functionality from feature request #53455 is implemented. Fixes #56683. ```release-note NONE ```

fejta-bot · 2018-03-04T12:15:14Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

shyamjvs · 2018-03-28T09:53:35Z

@crassirostris @loburm Can we turn back on the audit tests on large-scale now?

crassirostris · 2018-03-28T10:51:37Z

@shyamjvs Nope, this feature is a pre-requisite for enabling audit tests on large scale

/remove-lifecycle stale

CaoShuFeng · 2018-04-17T02:16:03Z

/assign

CaoShuFeng · 2018-04-17T07:14:37Z

Hi, @tallclair
I have some questions about the implemention detail:

Provide a pull-based stream for audit logging. This would look similar to a watch request to the API server, and provide a stream of audit events.

This is a plugin work as a server which needs to bind a port?

The should also have some options to filter events. Minimally, there should be a namespaced version that only includes audit events tied to that namespace, so that RBAC can be used to grant subdivided access.

This requires a authentication progress.
But I think authentication should not be as complex as the apiserver, we chose one authentication method in the first?

it might be nice to be able request only events tied to a given user or resource.

Totally agree!!!

CaoShuFeng · 2018-04-17T07:16:25Z

Hi, I'd be happy to work on this if no one has startted it.

CaoShuFeng · 2018-05-08T04:16:07Z

it might be nice to be able request only events tied to a given user or resource.

Since audit events are non-nanmespaced objects, how to use RBAC to implement access control for this stream?

CaoShuFeng · 2018-05-30T06:45:28Z

Hi.
#64494

This tries to fix: kubernetes/kubernetes#53455

CaoShuFeng · 2018-06-11T11:50:25Z

@tallclair @loburm
I made a KEP for this feature request. Please take a look.
Thanks.
kubernetes/community#2241

tallclair · 2018-06-11T18:23:38Z

awesome, thanks!

fejta-bot · 2018-09-09T18:59:37Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2018-10-09T19:23:01Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2018-11-08T20:10:34Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2018-11-08T20:10:42Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tallclair · 2019-01-03T21:07:17Z

/remove-lifecycle rotten

liggitt · 2019-04-03T15:07:55Z

given the HA implications, the inverse of this (dynamic audit webhook registration) seems preferable

tallclair · 2019-04-04T21:29:34Z

Agreed. What I would actually like to see is an out-of-tree implementation of this. An application that is registered as a dynamic audit webhook and serves the audit stream (maybe with an in-memory history cache). It could either serve a custom API, or register itself as an extension API server.

liggitt · 2019-04-19T00:35:41Z

Sounds good. Closing this here then.

/close

k8s-ci-robot · 2019-04-19T00:35:42Z

@liggitt: Closing this issue.

In response to this:

Sounds good. Closing this here then.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tallclair added area/apiserver area/logging sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/auth Categorizes an issue or PR as relevant to SIG Auth. labels Oct 4, 2017

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 4, 2017

ericchiang added the area/audit label Oct 9, 2017

tallclair mentioned this issue Nov 17, 2017

Advanced Auditing 1.9 umbrella bug #54551

Closed

14 tasks

crassirostris mentioned this issue Dec 4, 2017

Add retries to advanced audit e2e tests. #56705

Closed

loburm added a commit to loburm/kubernetes that referenced this issue Dec 4, 2017

Add DisabledForLargeClusters tag to audit tests.

7a15983

Remove this tag once functionality from feature request kubernetes#53455 is implemented.

loburm mentioned this issue Dec 4, 2017

Add DisabledForLargeClusters tag to audit tests. #56791

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 4, 2018

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 28, 2018

tallclair mentioned this issue Mar 28, 2018

isolated advanced audit for multi tenants in kubernetes cluster #61766

Closed

k8s-ci-robot assigned CaoShuFeng Apr 17, 2018

tallclair mentioned this issue May 24, 2018

KEP: Dynamic Audit Configuration kubernetes/community#2188

Merged

CaoShuFeng mentioned this issue May 30, 2018

[WIP]pull based audit event stream #64494

Closed

CaoShuFeng added a commit to CaoShuFeng/community that referenced this issue Jun 11, 2018

pull based audit event stream

5134b0a

This tries to fix: kubernetes/kubernetes#53455

CaoShuFeng added a commit to CaoShuFeng/community that referenced this issue Jun 11, 2018

pull based audit event stream

4805970

This tries to fix: kubernetes/kubernetes#53455

CaoShuFeng added a commit to CaoShuFeng/community that referenced this issue Jun 11, 2018

pull based audit event stream

13cbc1c

This tries to fix: kubernetes/kubernetes#53455

CaoShuFeng mentioned this issue Jun 11, 2018

KEP: pull based audit event stream kubernetes/community#2241

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 9, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 9, 2018

k8s-ci-robot closed this as completed Nov 8, 2018

x13n mentioned this issue Jan 2, 2019

Split audit test cases into separate tests #71489

Merged

tallclair reopened this Jan 3, 2019

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jan 3, 2019

tallclair added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Mar 6, 2019

k8s-ci-robot closed this as completed Apr 19, 2019

tanjunchen mentioned this issue Jan 1, 2020

/test: resolve pending TODOs #86756

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FR: streaming audit endpoint #53455

FR: streaming audit endpoint #53455

tallclair commented Oct 4, 2017

mbohlool commented Oct 5, 2017

crassirostris commented Dec 4, 2017 •

edited

fejta-bot commented Mar 4, 2018

shyamjvs commented Mar 28, 2018

crassirostris commented Mar 28, 2018

CaoShuFeng commented Apr 17, 2018

CaoShuFeng commented Apr 17, 2018 •

edited

CaoShuFeng commented Apr 17, 2018

CaoShuFeng commented May 8, 2018

CaoShuFeng commented May 30, 2018

CaoShuFeng commented Jun 11, 2018 •

edited

tallclair commented Jun 11, 2018

fejta-bot commented Sep 9, 2018

fejta-bot commented Oct 9, 2018

fejta-bot commented Nov 8, 2018

k8s-ci-robot commented Nov 8, 2018

tallclair commented Jan 3, 2019

liggitt commented Apr 3, 2019

tallclair commented Apr 4, 2019

liggitt commented Apr 19, 2019

k8s-ci-robot commented Apr 19, 2019

FR: streaming audit endpoint #53455

FR: streaming audit endpoint #53455

Comments

tallclair commented Oct 4, 2017

mbohlool commented Oct 5, 2017

crassirostris commented Dec 4, 2017 • edited

fejta-bot commented Mar 4, 2018

shyamjvs commented Mar 28, 2018

crassirostris commented Mar 28, 2018

CaoShuFeng commented Apr 17, 2018

CaoShuFeng commented Apr 17, 2018 • edited

CaoShuFeng commented Apr 17, 2018

CaoShuFeng commented May 8, 2018

CaoShuFeng commented May 30, 2018

CaoShuFeng commented Jun 11, 2018 • edited

tallclair commented Jun 11, 2018

fejta-bot commented Sep 9, 2018

fejta-bot commented Oct 9, 2018

fejta-bot commented Nov 8, 2018

k8s-ci-robot commented Nov 8, 2018

tallclair commented Jan 3, 2019

liggitt commented Apr 3, 2019

tallclair commented Apr 4, 2019

liggitt commented Apr 19, 2019

k8s-ci-robot commented Apr 19, 2019

crassirostris commented Dec 4, 2017 •

edited

CaoShuFeng commented Apr 17, 2018 •

edited

CaoShuFeng commented Jun 11, 2018 •

edited