Informer Lifecycle Improvements

[kevindelgado]

What type of PR is this?

/kind feature

What this PR does / why we need it:
Implementation of Informer Lifecycle Managemt design doc
Discussed at the Nov 4 sig-api-machinery meeting:video

It does a few things that are broken up by commit:

1. Adds the OnError method to the ResourceEventHandler interface so that the handler is notified of ListAndWatch errors.
2. Adds the ability to define stop conditions when running individual informers via the new RunWithStopOptions() method on the SharedInformer interface. It exposes running informers with stop options on the various informer factories (currently only supports global stop options that apply to all informers of an informer factory)
3. Modifies the metadata informer factory and dynamic informer factory to start informers via RunWithStopOptions() and provides a way to retrieve the stop channel for given informer to determine when an individual informer is stopped.
4. Refactors some CRD specific integration testing helpers out of the GC integration test into a shared util package so that RQ integration test can also use them without duplicating code.
5. Modifies GC controller to recognize when a CRD informer has stopped and removes the monitor for it.
6. Modifies RQ controller to recognize when a CRD informer has stopped and removes the monitor for it.
7. Modifies controller-manager to run the GC and RQ controllers with RunWithStopOptions (meaning it always stops an informer upon reflector ListAndWatch error).

This is tested manually by modifying controller-runtime to use the new interfaces. Also by running the resource quota controller and GC controller with the new interfaces to verify that the informer shuts down as expected.

Unit testing has been added to the relevant pieces of tools/cache and metadatainformer/dynamicinformer. Integration tests for both the garbage collector and quota controller test that the install CRD/uninstall CRD/reinstall CRD flow continues to work while the informer gets stopped and restarted as expected.

Which issue(s) this PR fixes:

Fixes #79610
Unblocks work relating to kubernetes-sigs/controller-runtime#1192

1. Adds the OnError method to the ResourceEventHandler interface so that the handler is notified of ListAndWatch errors. 
2. Adds the ability to define stop conditions when running individual informers via the new `RunWithStopOptions()` method on the `SharedInformer` interface. It exposes running informers with stop options on the various informer factories (currently only supports global stop options that apply to all informers of an informer factory)
3. Modifies the metadata informer factory and dynamic informer factory to start informers via `RunWithStopOptions()` and provides a way to retrieve the stop channel for given informer to determine when an individual informer is stopped. 
4. Refactors some CRD specific integration testing helpers out of the GC integration test into a shared util package so that RQ integration test can also use them without duplicating code.
5. Modifies GC controller to recognize when a CRD informer has stopped and removes the monitor for it.
6. Modifies RQ controller to recognize when a CRD informer has stopped and removes the monitor for it.
7. Modifies controller-manager to run the GC and RQ controllers with RunWithStopOptions (meaning it always stops an informer upon reflector ListAndWatch error).

based on #98657

[k8s-ci-robot]

Hi @kevindelgado. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kevindelgado]

/assign @caesarxuchao
/assign @DirectXMan12

[fedebongio]

/triage accepted
/ok-to-test

[fedebongio]

/assign @yliaog

[kevindelgado]

/retest

[kevindelgado]

/retest

[kevindelgado]

/retest

[caesarxuchao]

/unassign

[kevindelgado]

friendly ping @yliaog @liggitt

[yliaog]

the interface has changed, please sync with the latest

[kevindelgado]

hmm ok, it looks like it will be changing back to returning an error, I will wait until #98657 is updated to sync with latest

[k8s-ci-robot]

@kevindelgado: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[atoato88]

FYI
I confirmed that this PR clears off controller manager logs said in #79610 on my local env.
Thank you to create this PR.
I'll wait for merging to master.

[kevindelgado]

FYI
I confirmed that this PR clears off controller manager logs said in #79610 on my local env.
Thank you to create this PR.
I'll wait for merging to master.

Yea sorry for how slow this is moving. This PR is blocked on #98657 which is still in review and I have been trying to push it forward without much luck. Feel free to ping that PR like I have been doing to try to see if progress can be made there.

[249043822]

RemoveEventHandler may access handler obj by fmt.Errorf， so this line should be wrapped by listener.lock. or may casuse DATA RACE

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[atoato88]

/remove-lifecycle stale

[dims]

Is this PR still needed, please rebase if so (or we can close it?)

[atoato88]

I think this PR is still needed because #79610 error occurs on v1.23.0 cluster created by kind.

@kevindelgado
Any comments and could you rebase?

[kevindelgado]

This PR is blocked on #98657

The issues from #79610 are certainly still present and will need to be addressed, but this PR has been deprioritized given the current state of #98657

[dims]

This PR is older than 4 weeks and needs as rebase.

Please rebase the PR against latest master and reopen if still needed.

/close

[k8s-ci-robot]

@dims: Closed this PR.

In response to this:

This PR is older than 4 weeks and needs as rebase.

Please rebase the PR against latest master and reopen if still needed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Pingan2017]

@kevindelgado
we encountered a bug when we merge this PR .
when delete a deployment, the gc controller set the deletionTimestamp for the rs and pod, then pod been deleted from etcd by kubelet, but the gc controller not delete rs and deploy form etcd.
When we analyzed the problem, we found apiserver restarted when delete deployment and the pod still exist in gc controller cache even pod deleted from etcd. the gc controller miss the pod deleted event .
By reading the code, we found that this bug is related to this PR.
Let's imagine a scenario:

1. gc controller start informer and add event handle
2. deploy a deployment , gc controller will received deployment, rs , pod event , and add them to cache.
3. delete this deployment, gc controler set the deletionTimestamp for rs , pod
4. apiserver restart
5. gc controller sync the deletableResources , but get some resource(eg. pod) failed,because the apiserver restarted
   

   kubernetes/pkg/controller/garbagecollector/garbagecollector.go

   Lines 173 to 178 in 5b1e538

   	func (gc *GarbageCollector) Sync(discoveryClient discovery.ServerResourcesInterface, period time.Duration, stopCh <-chan struct{}) {
   	oldResources := make(map[schema.GroupVersionResource]struct{})
   	wait.Until(func() {
   	// Get the current resource list from discovery.
   	newResources := GetDeletableResources(discoveryClient)

6. gc controller will remove the pod evnet handler from informer
7. kubelet delete pod from etcd
8. informer received the pod delete evnet ,but no event handler for gc controller, so gc controller not delete pod form cache, then , gc controller won't delete rs and deployment forever
9. gc controller get the pod resource again, but too late, pod has been deleted, gc won't received the pod deleted event

[oomichi]

This PR is blocked on #98657

The issues from #79610 are certainly still present and will need to be addressed, but this PR has been deprioritized given the current state of #98657

Hi @kevindelgado

#111122 has been merged instead of #98657
Can we move forward for this again?