Add cluster failure threshold

[dddddai]

What type of PR is this?
/kind feature

What this PR does / why we need it:
Currently cluster status will be set to NotReady immediately if the check failed
To make it more robust, this pr adds cluster failure threshold(default to 30s), which is the duration of failure for the cluster to be considered unhealthy.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
Should the flag cluster-failure-threshold be the number of failures(default to 3) or the time(default to 30s)? Which is better?

Does this PR introduce a user-facing change?:

`karmada-controller-manager/karmada-agent`: Introduced `--cluster-failure-threshold` flag to specify cluster failure threshold.

[RainbowMango]

Should the flag cluster-failure-threshold be the number of failures(default to 3) or the time(default to 30s)? Which is better?

What's your option?

[dddddai]

Either is fine with me, time is probably more intuitive but users need to keep cluster-failure-threshold a multiple of cluster-status-update-frequency

Kubefed uses the number: https://github.com/kubernetes-sigs/kubefed/blob/v0.9.2/charts/kubefed/charts/controllermanager/templates/kubefedconfig.yaml#L19

[dddddai]

Thank you, I've addressed your comments

[RainbowMango]

I really want something like ClusterData which caches each cluster's last prob time and status. How do you think?

[RainbowMango]

By the way, we should clean up the data when deleting the cluster.

karmada/pkg/controllers/status/cluster_status_controller.go

Lines 98 to 102 in 92a134b

	// The resource may no longer exist, in which case we stop the informer.
	if apierrors.IsNotFound(err) {
	c.InformerManager.Stop(req.NamespacedName.Name)
	return controllerruntime.Result{}, nil
	}

[dddddai]

I really want something like ClusterData which caches each cluster's last prob time and status. How do you think?

For "last prob time", kubefed has its own Condition but we are using metav1.Condition where we don't have "last prob time"

For "status", kubefed stores it because it will restore the old status, we just return and retry in the next health check, I think they take the same effect

So I don't see where we are gonna use it? Please correct me if I'm wrong

[RainbowMango]

What I mean something like ClusterData, learn the idea, not the whole struct.
And last prob time means last success or last failure time the time will be changed only when the condition change.

[dddddai]

OK, added

[RainbowMango]

I can't tell but seems something doesn't feel right here.
Let me think about it.

[dddddai]

Have modified as you suggested

[RainbowMango]

Seems these tests interact with each other?

[dddddai]

Yes they are observed statuses ordered by time, because thresholdReadyCondition depends on previous statuses

[RainbowMango]

In this test, we are testing thresholdAdjustedReadyCondition, we should set up cache, current condition, and observed condition for each case. They should not depend on each other.

[dddddai]

That would be kind of tricky..How to set up the cache?(clusterDataMap and thresholdStartTime)

[dddddai]

We are testing clusterConditionStore.thresholdAdjustedReadyCondition where clusterConditionStore is stateful

[RainbowMango]

func TestThresholdAdjustedReadyCondition2(t *testing.T) {
	tests := []struct {
		name              string
		clusterData       *clusterData
		currentCondition  *metav1.Condition
		observedCondition *metav1.Condition
		expectedCondition *metav1.Condition
	}{
		{
			name:              "cluster just joined in ready state",
			clusterData:       nil, // no cache yet
			currentCondition:  nil, // no condition was set on cluster object yet
			observedCondition: &metav1.Condition{Status: metav1.ConditionTrue},
			expectedCondition: &metav1.Condition{Status: metav1.ConditionTrue},
		},
		{
			name:              "cluster just joined in not-ready state",
			clusterData:       nil, // no cache yet
			currentCondition:  nil, // no condition was set on cluster object yet
			observedCondition: &metav1.Condition{Status: metav1.ConditionFalse},
			expectedCondition: &metav1.Condition{Status: metav1.ConditionFalse},
		},
		{
			name: "cluster becomes not ready but still not reach threshold",
			clusterData: &clusterData{
				readyCondition:     metav1.ConditionFalse,
				thresholdStartTime: time.Now(),
			},
			currentCondition:  &metav1.Condition{Status: metav1.ConditionTrue},
			observedCondition: &metav1.Condition{Status: metav1.ConditionFalse},
			expectedCondition: &metav1.Condition{Status: metav1.ConditionTrue},
		},
	}

	for _, test := range tests {
		t.Run(test.name, func(t *testing.T) {
			cache := clusterConditionStore{
				clusterDataMap:   sync.Map{},
				failureThreshold: 10,
			}

			if test.clusterData != nil {
				cache.update("foo", test.clusterData)
			}

			cluster := &clusterv1alpha1.Cluster{}
			cluster.Name = "foo"
			if test.currentCondition != nil {
				meta.SetStatusCondition(&cluster.Status.Conditions, *test.currentCondition)
			}

			thresholdReadyCondition := cache.thresholdAdjustedReadyCondition(cluster, test.observedCondition)

			if test.expectedCondition.Status != thresholdReadyCondition.Status {
				t.Fatalf("expected: %s, but got: %s", test.expectedCondition.Status, thresholdReadyCondition.Status)
			}
		})
	}
}

[RainbowMango]

Just tried add some cases(not finished).

[RainbowMango]

I tested it on my side, I think we are ready to move further. Thanks @dddddai

[RainbowMango]

/lgtm
/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment