Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache serializations across watchers #81914

Merged

Conversation

@wojtek-t
Copy link
Member

wojtek-t commented Aug 25, 2019

Reuse serialization result across watchers - before this change, we were serializing objects independently for every single watcher. With this PR, when dispatching an event in watchcache:

  • wrap the object into one that is caching its encodings
  • reuse them if multiple watchers expect the same serialization format

This is significant performance improvement:

  • in 5k tests, we save ~15% of memory allocations
  • in 5k tests, we save ~5% of CPU
    This is also addressing #75294

Ref kubernetes/enhancements#1152

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Aug 25, 2019

@wojtek-t: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Aug 25, 2019

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@wojtek-t wojtek-t force-pushed the wojtek-t:cache_serializations_across_watchers branch from 899779b to 3f20ed6 Aug 26, 2019
@wojtek-t

This comment has been minimized.

Copy link
Member Author

wojtek-t commented Aug 26, 2019

/retest

Copy link
Contributor

jpbetz left a comment

First pass, I'l try to get another pass out shortly

@wojtek-t wojtek-t force-pushed the wojtek-t:cache_serializations_across_watchers branch from 3f20ed6 to 7eb32f6 Aug 26, 2019
Copy link
Member Author

wojtek-t left a comment

The updates to first 5 commits were done in the parent PR: #81585

@wojtek-t wojtek-t force-pushed the wojtek-t:cache_serializations_across_watchers branch from 7eb32f6 to e876bce Aug 27, 2019
@wojtek-t wojtek-t force-pushed the wojtek-t:cache_serializations_across_watchers branch from e876bce to 3ad2feb Sep 11, 2019
// The following functions implement metav1.Object interface:
// - getters simply delegate for underlying object
// - SetSelfLink stores the value passed in the first call
// - all other setters ignore the passed value.

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

drop these comments, since they no longer apply

This comment has been minimized.

Copy link
@wojtek-t

wojtek-t Sep 30, 2019

Author Member

done

if cache, ok := o.serializations.Load().(serializationsCache); ok && len(cache) == 0 {
return
}
klog.Warningf("Unexpected cache invalidation for %#v", o.object)

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

logging a stack here would help track down which setter and what call site caused the invalidation. maybe think about a way to avoid logging O(millions) of these if we discover a path that causes this in production (package-level counter that logs first hundred or something hacky could work)

This comment has been minimized.

Copy link
@wojtek-t

wojtek-t Sep 30, 2019

Author Member

Did something slightly more expensive - which is logging once per second, though it shouldn't matter here. And I think it would better facilitate debugging.

func (o *cachingObject) SetLabels(labels map[string]string) {
o.lock.Lock()
defer o.lock.Unlock()
if equality.Semantic.DeepEqual(o.object.GetLabels(), labels) {

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

check if nil vs {} produces different serializations. I think we might want to make this stronger (reflect.DeepEqual) so that any difference invalidates

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

reflect.DeepEqual is safer, we just risk unnecessary cache invalidations (and we're not actually calling SetLabels in any known paths). same comment applies to all DeepEqual comparisons

This comment has been minimized.

Copy link
@wojtek-t

wojtek-t Sep 30, 2019

Author Member

done

return o.object.GetSelfLink()
}
func (o *cachingObject) SetSelfLink(selfLink string) {
o.lock.Lock()

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

consider doing an RLock first to check if we need to invalidate and actually set. maybe benchmark to see if the added code complexity is worth it

This comment has been minimized.

Copy link
@liggitt

liggitt Sep 27, 2019

Member

might consider refactoring these to call a helper to manage the RLock/Unlock/invalidateCacheLocked calls

o.conditionalSet(
  func()bool{ return o.object.GetSelfLink() == selfLink},
  func(){ o.object.SetSelfLink(selfLink)},
)

This comment has been minimized.

Copy link
@wojtek-t

wojtek-t Sep 30, 2019

Author Member

done

@wojtek-t wojtek-t force-pushed the wojtek-t:cache_serializations_across_watchers branch from f4d16a5 to 25a728a Sep 30, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Sep 30, 2019

@wojtek-t: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wojtek-t

This comment has been minimized.

Copy link
Member Author

wojtek-t commented Sep 30, 2019

/retest

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Oct 1, 2019

/lgtm
/approve

thanks for all the work on this

@k8s-ci-robot k8s-ci-robot added the lgtm label Oct 1, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Oct 1, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 7878160 into kubernetes:master Oct 1, 2019
13 of 14 checks passed
13 of 14 checks passed
pull-kubernetes-e2e-gce-100-performance Job triggered.
Details
cla/linuxfoundation wojtek-t authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Oct 1, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Oct 1, 2019

@wojtek-t: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce-100-performance 25a728a link /test pull-kubernetes-e2e-gce-100-performance

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.