karmada-search may crash due to concurrent map read and map write

[tedli]

What happened:

search pod crashed, fatal error: concurrent map read and map write.

What you expected to happen:

If errors occurred, log detail, or if caused by concurrent map read and map write, add lock to avoid it.

How to reproduce it (as minimally and precisely as possible):

No idea. 🤣

Anything else we need to know?:

search.log

The log didn't point out exactly where caused this, it makes sense, because when we talk about concurrent map read and map write, it must be at least 2 place, reading or writing, so it's not caused by a particular line of code, it should be caused by some code happened at a same time.

karmada/pkg/registry/search/storage/cache.go

Lines 142 to 159 in 5f16640

	func addAnnotationWithClusterName(resourceObjects []runtime.Object, clusterName string) []runtime.Object {
	resources := make([]runtime.Object, 0)
	for index := range resourceObjects {
	resource := resourceObjects[index].(*unstructured.Unstructured)
	annotations := resource.GetAnnotations()
	if annotations == nil {
	annotations = make(map[string]string)
	}
	annotations[clusterv1alpha1.CacheSourceAnnotationKey] = clusterName
	resource.SetAnnotations(annotations)
	resources = append(resources, resource)
	}
	return resources
	}

My point of view is that, it may caused by line 152, adding entry to annotations map, meanwhile other logic reading it, because runtime.Object is a reference, not a value.

Environment:

• Karmada version: v1.6.1
• kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
• Others:

[RainbowMango]

@tedli You are using v1.6.1, right?

[tedli]

Yes.

[zishen]

From the log you provided, it seems that goroutine 291924 [runnable] and goroutine 291894 [running] are operating on the annotations map at the same time.

But I don't know where goroutine 291924 comes from.

You did two searches at the same time?

[tedli]

Hi @zishen ,

You did two searches at the same time?

Did you mean two replicas of search deployment?

If so, no, only one replica.

[zishen]

I mean the two searches request.
httpserver will enable multiple coroutines when processing requests.

[tedli]

Hi @zishen ,

I mean the two searches request.

Maybe. This is my production setup, with front ui, so there may be multi request happens at same time.

By the way.

But I don't know where goroutine 291924 comes from.

It seemed all goroutines were dumped, not only goroutines that caused this panic.
And I just guess it may caused by annotations map, but I'm not sure.

[ikaven1024]

/assign