Performance improve for karmada-controller-manager

[Poor12]

Signed-off-by: Poor12 shentiecheng@huawei.com

What type of PR is this?
/kind feature

What this PR does / why we need it:
As is talked in #1858，when member cluster is large(3k nodes, 10w pods), karmada-controller-manager is easy to be OOMkilled and retry to sync nodes and pods again and again.

Based on #2008, I tested the two clusters with 100 nodes and almost 300 pods. We can easily see that in cluster status controllers, list nodes and pods periodicly cost a lot when the cluster is large(a lot of unmarshal works) according to the frame graph.

So I modified the mechanism from full amount list to watch based on event to make the memory increase more stable.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:
None

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign rainbowmango after the PR has been reviewed.
You can assign the PR to them by writing /assign @rainbowmango in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Poor12]

cc @RainbowMango

[ikaven1024]

Is it safe to call read from map? Will it panic without clusterName key?

[Poor12]

Thanks for pointing out.

[ikaven1024]

NotFound error shall be returned?

[Poor12]

miss return statement

[ikaven1024]

How do you think about comparing with resource version?

[Poor12]

I think it's ok to comparing with obj, because we don't know the type of obj. If comparing with resource version, may add the cost of converting type.

[ikaven1024]

I test it on my side with go 1.17.5. It tells type assert is more faster than reflect.DeepEqual

var (
	obj1 interface{}
	obj2 interface{}
)

func init() {
        pod1Yaml := `<skipped>`
        pod2Yaml := `<skipped>`

	unstructuredObj1 := &unstructured.Unstructured{}
	unstructuredObj2 := &unstructured.Unstructured{}
	unstructured.UnstructuredJSONScheme.Decode([]byte(pod1Yaml), nil, unstructuredObj1)
	unstructured.UnstructuredJSONScheme.Decode([]byte(pod2Yaml), nil, unstructuredObj2)

        obj1, obj2 = unstructuredObj1, unstructuredObj2
}

func BenchmarkDeepEqual(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_ = reflect.DeepEqual(obj1, obj2)
	}
}

func BenchmarkAssert(b *testing.B) {
	for i := 0; i < b.N; i++ {
		o1, _ := meta.Accessor(obj1)
		o2, _ := meta.Accessor(obj2)
		_ = o1.GetResourceVersion() == o2.GetResourceVersion()
	}
}

goos: darwin
goarch: arm64
pkg: github.com/ikaven1024/gists/k8s/benchmark
BenchmarkDeepEqual
BenchmarkDeepEqual-8   	 7717909	       145.2 ns/op
BenchmarkAssert
BenchmarkAssert-8        	33085802	        36.39 ns/op
PASS

[RainbowMango]

Informer synced for gvr doesn't mean the eventHandler exit.
Since all controllers share the MultiClusterInformerManager, the informer might be created(and synced from another place).

[RainbowMango]

Looks weird that .metadata.ResourceVersion is not changed in an update event.
I doubt it.