Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add configuration point to SharedInformer to transform objects before storing #107507

Merged
merged 1 commit into from
Jan 24, 2022
Merged

Add configuration point to SharedInformer to transform objects before storing #107507

merged 1 commit into from
Jan 24, 2022

Conversation

alexzielenski
Copy link
Contributor

@alexzielenski alexzielenski commented Jan 12, 2022 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

We have teams who use client-go's SharedIndexInformer through controller-runtime extensively. Some applications create caches for object metadata. These caches can have tens of thousands of objects. We have found that some our controllers are being OOM-killed more frequently recently since the addition of managedFields.

This PR adds a configuration point to SharedInformer to arbitrarily transform the object before it is stored in the cache. We intend to use it to null out any large unused fields to save ram cost; since our controller cares only about a subset of the object's fields. It may also have other applications, as well.

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Adds SetTransform to SharedInformer to allow users to transform objects before they are stored.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jan 12, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @alexzielenski. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@alexzielenski
Copy link
Contributor Author

/hold

I still need to add tests. Posting this PR in the meantime for comments.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 12, 2022
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 12, 2022
@wojtek-t wojtek-t self-assigned this Jan 13, 2022
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 13, 2022
alexzielenski
alexzielenski commented Jan 13, 2022
View reviewed changes
Copy link
Contributor Author

@alexzielenski alexzielenski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've modified the PR to use refactor SharedIndexInformer to now share code with newInformer. SharedIndexInformer has a few quirks which required some minor changes but it did fit fairly nicely.

There is one hairy area around sending the notification on update events. The ResourceEventHandler interface does not have an argument for the event type, so the original logic which silenced Sync notifications for non-sync listeners may be different and the new implementation needs to be looked at.

staging/src/k8s.io/client-go/tools/cache/shared_informer.go Show resolved Hide resolved
staging/src/k8s.io/client-go/tools/cache/shared_informer.go Outdated Show resolved Hide resolved
@fedebongio
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 13, 2022
@alexzielenski alexzielenski force-pushed the client-preprocess-handler branch 2 times, most recently from fc19d82 to d992c5c Compare January 19, 2022 21:34
@alexzielenski
Copy link
Contributor Author

/hold cancel

test added

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 19, 2022
@wojtek-t
Copy link
Member

/ok-to-test

/lgtm
/approve

@wojtek-t
Copy link
Member

/lgtm cancel
/approve cancel

Test failure is relevant to this PR - please fix.

@k8s-ci-robot k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 20, 2022
@alexzielenski
modify SharedIndexInformer to use newInformer constructor which suppo…
754bf3b 
…rts transformers

avoids code duplication, allows transformer to be used with SharedIndexInformer
@alexzielenski
Copy link
Contributor Author

Test failure was a bug in the test. Since the transformer is run even for relists/updates it is important for it copy before mutating, if it is going to change the object. Otherwise, there is a potential data race. I've corrected the test and added a note to SetTransformer to watch out for this.

@alexzielenski
Copy link
Contributor Author

@wojtek-t Tests passing again. Please re-lgtm

@wojtek-t
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 24, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexzielenski, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 24, 2022
@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@alexzielenski
Copy link
Contributor Author

currently blocked by #107708

@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit f4046f9 into kubernetes:master Jan 24, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Jan 24, 2022
@csviri csviri mentioned this pull request Feb 1, 2022
Closed
@howardjohn howardjohn mentioned this pull request Feb 1, 2022
Closed
howardjohn
howardjohn reviewed Feb 1, 2022
View reviewed changes
staging/src/k8s.io/client-go/tools/cache/shared_informer.go
//
// Must be set before starting the informer.
//
// Note: Since the object given to the handler may be already shared with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this happen? Seems like a sharedIndexInformer can only run once, and ProcessDeltas is called in controller.processLoop which is not concurrent.

DeepCopy may be expensive so given one of the primary use cases for this is performance it seems fairly important to not do extra work when not needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a relist of existing data occurs it is possible for handledeltas to be called multiple times for the same object. The first time the transformation is applied it the object instance will not have been shared. Assuming you have a way of marking an already- transformed object in order to skip repeated transformation applications then a copy should not be necessary.

howardjohn
howardjohn reviewed Feb 1, 2022
View reviewed changes
staging/src/k8s.io/client-go/tools/cache/shared_informer.go
// other goroutines, it is advisable to copy the object being
// transform before mutating it at all and returning the copy to prevent
// data races.
SetTransform(handler TransformFunc) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit concerned that if an error occurs there is, as far as I can tell, no logging at all. This bubbles up to processLoop, which discards the error

howardjohn
howardjohn reviewed Feb 1, 2022
View reviewed changes
staging/src/k8s.io/client-go/tools/cache/shared_informer.go

// The TransformFunc is called for each object which is about to be stored.
//
// This function is intended for you to take the opportunity to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to list the limitations of this. Does the returned type have to be the same type as the input?

From what I can tell it does not but must implement runtime.Object and metav1.ObjectMetaAccessor or equivalent interfaces

@alexzielenski alexzielenski mentioned this pull request Feb 15, 2022
Merged
@dddddai dddddai mentioned this pull request Feb 22, 2022
Closed
@alexzielenski alexzielenski mentioned this pull request Mar 29, 2022
Closed
9 tasks
@dddddai dddddai mentioned this pull request May 23, 2022
Closed
@RainbowMango RainbowMango mentioned this pull request May 23, 2022
Closed
1 task
@rmohr rmohr mentioned this pull request Oct 17, 2022
Merged
@alexzielenski alexzielenski deleted the client-preprocess-handler branch November 16, 2022 18:49
@vincepri vincepri mentioned this pull request Dec 8, 2022
Closed
@d3c3mber
Copy link

d3c3mber commented May 23, 2024 •
edited

Just curious, why we are using the transform function in sharedIndexInformer.HandleDeltas(), can we move the transformation function up to the reflector? @alexzielenski

We have a client program that uses reflector directly, and we'd like to use the object transform functionality. Is it ok we also add transform function to reflector.go as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@howardjohn howardjohn howardjohn left review comments

@wojtek-t wojtek-t wojtek-t left review comments

@ncdc ncdc Awaiting requested review from ncdc

@resouer resouer Awaiting requested review from resouer

Assignees

@wojtek-t wojtek-t

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.24
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@alexzielenski @k8s-ci-robot @fedebongio @wojtek-t @k8s-triage-robot @d3c3mber @howardjohn