Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

increase auth cache size #83643

Merged
merged 1 commit into from
Oct 9, 2019
Merged

increase auth cache size #83643

merged 1 commit into from
Oct 9, 2019
+8 −1

Conversation

lavalamp
Copy link
Member

@lavalamp lavalamp commented Oct 8, 2019 •
edited

The benchmark in #83519 demonstrates that cache performance gets drastically worse when the number of tokens exceeds the size of the cache.

We advertise 5k nodes and 10k namespaces, it's reasonable to assume each node has a token and each namespace has a service account, which has a token.

This PR makes the cache accept 32k tokens. (It was 4k.)

Authentication token cache size is increased (from 4k to 32k) to support clusters with many nodes or many namespaces with active service accounts.

/kind bug
/priority important-soon

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 8, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/auth Categorizes an issue or PR as relevant to SIG Auth. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 8, 2019
@lavalamp
Copy link
Member Author

lavalamp commented Oct 8, 2019

/test pull-kubernetes-kubemark-e2e-gce-big

@liggitt
Copy link
Member

liggitt commented Oct 8, 2019

The spelling bot has you flagged, lgtm otherwise

enj
enj approved these changes Oct 8, 2019
View reviewed changes
Copy link
Member

@enj enj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification on the comment may be useful.

staging/src/k8s.io/apiserver/pkg/authentication/token/cache/cached_token_authenticator.go
// Cache performance degrades noticably when the number of
// tokens in operation exceeds the size of the cache. It is
// cheap to make the cache big in the second dimension below,
// the memory is only consumed when that many tokens are being
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we consider the case of short lived SA tokens from the token request API, the cache will grow to the max size before dropping old tokens since there is no go routine or similar to purge the cache. I do not think that is immediately apparent from this comment.

That being said, even at the worse case I do not think this is an issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How short-lived are such tokens? How many of them do we make?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And under what conditions do we make them?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They live 10 minutes by default, and are provisioned per-pod.

@lavalamp
Copy link
Member Author

lavalamp commented Oct 8, 2019

The spelling bot has you flagged

on my permanent record, no doubt

@liggitt
Copy link
Member

liggitt commented Oct 9, 2019

/lgtm
/retest

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 9, 2019
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 6992d13 into kubernetes:master Oct 9, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Oct 9, 2019
@enj enj mentioned this pull request Oct 11, 2019
Merged
ohsewon pushed a commit to ohsewon/kubernetes that referenced this pull request Oct 16, 2019
@k8s-ci-robot @ohsewon
Merge pull request kubernetes#83643 from lavalamp/bigger-auth-cache
7fcd176 
increase auth cache size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liggitt liggitt liggitt left review comments

@enj enj enj approved these changes

@jianhuiz jianhuiz Awaiting requested review from jianhuiz

Assignees

@liggitt liggitt

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/auth Categorizes an issue or PR as relevant to SIG Auth. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.17
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@lavalamp @k8s-ci-robot @liggitt @fejta-bot @enj