New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce buffered audit backend #60076

Merged
merged 1 commit into from Feb 26, 2018

Conversation

@crassirostris
Member

crassirostris commented Feb 20, 2018

This is a copy from #53006 with comments addressed, since @hzxuzhonghu cannot work on this right now and the code freeze is close. Thanks @hzxuzhonghu for the contribution!

This is the first PR to fix #53006 and then #53020

This PR introduces a new audit backend, that batches incoming audit events and sends these batches to the underlying audit backend. This new backend is designed to replace existing batching webhook backend and allow to use batching with log backend, to solve the performance issues.

@sttts @ericchiang @CaoShuFeng @tallclair would really appreciate your review

Buffered audit backend is introduced, to be used with other audit backends.
@crassirostris

This comment has been minimized.

Member

crassirostris commented Feb 21, 2018

/retest

var _ audit.Backend = &bufferedBackend{}
// NewBackend returns a buffered audit backend wraps delegate backend.

This comment has been minimized.

@sttts

sttts Feb 22, 2018

Contributor

nit: that wraps the delegate backend.

var _ audit.Backend = &bufferedBackend{}
// NewBackend returns a buffered audit backend wraps delegate backend.
func NewBackend(backend audit.Backend, config BatchConfig) audit.Backend {

This comment has been minimized.

@sttts

sttts Feb 22, 2018

Contributor

s/backend/delegate/

@sttts

This comment has been minimized.

Contributor

sttts commented Feb 22, 2018

Looks good so far. Missing a commit to integrate it, don't I?

@crassirostris

This comment has been minimized.

Member

crassirostris commented Feb 22, 2018

@sttts

Thanks!

Missing a commit to integrate it, don't I?

I'll do it in a separate PR, to make it easier to review

@sttts

This comment has been minimized.

Contributor

sttts commented Feb 22, 2018

I'll do it in a separate PR, to make it easier to review

Would help to see the removal of the old buffering code here. I guess all the logic is just copied and refactored to be generic?

@crassirostris

This comment has been minimized.

Member

crassirostris commented Feb 22, 2018

@sttts Yes, but removing the old code requires changing the code related to parsing command-line parameters, that's why I wanted to move it to a separate PR

@sttts

This comment has been minimized.

Contributor

sttts commented Feb 22, 2018

@crassirostris can you open that second PR?

@crassirostris

This comment has been minimized.

Member

crassirostris commented Feb 22, 2018

@sttts OK, I found a couple of nits related to the integration process, agree it's easier to make it all in one PR

Will update it soon

@sttts

This comment has been minimized.

Contributor

sttts commented Feb 22, 2018

agree it's easier to make it all in one PR

two PRs are fine. But seeing the WIP second PR helps :)

Introduce buffered audit backend
Signed-off-by: Mik Vyatskov <vmik@google.com>
@crassirostris

This comment has been minimized.

Member

crassirostris commented Feb 22, 2018

@sttts PTAL, I created a PR, based on this one: #60237

@k8s-ci-robot k8s-ci-robot added the lgtm label Feb 22, 2018

const (
// Default configuration values for ModeBatch.
defaultBatchBufferSize = 10000 // Buffer up to 10000 events before starting discarding.
defaultBatchMaxSize = 400 // Only send up to 400 events at a time.

This comment has been minimized.

@tallclair

tallclair Feb 22, 2018

Member

Something that came up recently was whether we need to cap the max request size too. I.e. in addition to this, have a byte limit.

Shouldn't block this PR as it can be added later, just wanted to mention it here.

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Feb 26, 2018

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Feb 26, 2018

Automatic merge from submit-queue (batch tested with PRs 57326, 60076, 60293, 59756, 60370). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit 2a373ac into kubernetes:master Feb 26, 2018

4 of 13 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-bazel-test
Details
pull-kubernetes-bazel-test Job triggered.
Details
pull-kubernetes-e2e-gce Job triggered.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job triggered.
Details
pull-kubernetes-e2e-kops-aws Job triggered.
Details
pull-kubernetes-kubemark-e2e-gce Job triggered.
Details
pull-kubernetes-node-e2e Job triggered.
Details
pull-kubernetes-unit Job triggered.
Details
pull-kubernetes-verify Job triggered.
Details
cla/linuxfoundation crassirostris authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gke Skipped
@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Feb 26, 2018

@crassirostris: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-verify 3f0e49a link /test pull-kubernetes-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-merge-robot added a commit that referenced this pull request Mar 1, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after #60076

This PR fixes #53020, to address #53006 later

In this PR buffered backend, introduced in #60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

sttts pushed a commit to sttts/apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/kube-aggregator that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/sample-apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/apiextensions-apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/kube-aggregator that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/sample-apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/apiextensions-apiserver that referenced this pull request Mar 2, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/apiserver that referenced this pull request Mar 7, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/kube-aggregator that referenced this pull request Mar 7, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/sample-apiserver that referenced this pull request Mar 7, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

k8s-publishing-bot added a commit to kubernetes/apiextensions-apiserver that referenced this pull request Mar 7, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/apiserver that referenced this pull request Mar 16, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/kube-aggregator that referenced this pull request Mar 16, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/sample-apiserver that referenced this pull request Mar 16, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28

sttts pushed a commit to sttts/apiextensions-apiserver that referenced this pull request Mar 16, 2018

Merge pull request #60237 from crassirostris/audit-use-buffered-backend
Automatic merge from submit-queue (batch tested with PRs 60542, 60237). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Audit use buffered backend

This is the next step after kubernetes/kubernetes#60076

This PR fixes kubernetes/kubernetes#53020, to address kubernetes/kubernetes#53006 later

In this PR buffered backend, introduced in kubernetes/kubernetes#60076, is used to replace ad-hoc solution for webhook and add an ability to enable buffering for the log audit backend.

```release-note
Log audit backend can now be configured to perform batching before writing events to disk.
```

/cc @sttts @tallclair @ericchiang @CaoShuFeng

Kubernetes-commit: 209cdd90489a5af11f2f124e592c57df82a3da28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment