Flapping state of module in kyma-CR

[k15r]

Description

If a module goes into warning state during deletion the kyma-CR will report it alternating Is Deleting and Warning

Steps to reproduce

eg:
install nats module
create resources on nats:

• port forward nats and use the nats cli

nats stream create foo

delete the nats module

it will go into warning state as the manager will not remove the reconciler (resources exist on the nats cluster)
the result can be seen in the attachment

Environment Type

Managed

Environment Info

Kubernetes Version: x.y.z
Docker Version: x.y.z
Kyma CLI Version: x.y.z
If used - k3d/k3s/minikube versions

Attachments

Screen.Recording.2023-11-23.at.11.54.07.mov

[lindnerby]

We agreed on investigating the root cause of this behaviour estimate it to 1. Follow-up issue for the fix will be linked here as well.

[jeremyharisch]

After investigating into this Issue I have the following conclusion:

• The reason for this flickering is
  1. In the declarative pkg the status of the Manifest CR will be set to Deleting if the DeletionTimestamp has been set and the state is unequal to Deleting, then it triggers a new reconiliation
  2. Then if the DeletionTimestamp is set and the state is equal to Deleting it continues with the normal workflow of synchronising the resources, thus if the Module CR has the state Warning (even with set DeletionTimestamp), the state will be propagated to the Manifest CR -> Manifest has now Warning state
  3. Then it starts again with point 1.
• The scenario was "covered" in the E2E test, but due to a wrong implementation of the code, the test was green -> Test needs to be adapted + We need a new feature in the Template-Operator to simulate this scenario

Acceptance Criterias:

• Add new flag to Template-Operator which allows to set the State when Module is in Deleting; default Deleting -> Can be used to simulate more complex scenarios like the one describes in this ticket; I.e. Final-Deleting-State
• Fix flapping state bug
  • In declarative/ 2/reconciler.go in the syncResoruces function it always sets the status of the manifest CR to Deleting if the DeletionTimestamp is unequal to zero, this needs to be changed to avoid the status flapping between Deleting and Warning
  • Could look like the following

                if obj.GetDeletionTimestamp().IsZero() {
			obj.SetStatus(status.WithState(shared.StateProcessing).WithOperation(ErrWarningResourceSyncStateDiff.Error()))
		} else if status.State != shared.StateWarning {
			obj.SetStatus(status.WithState(shared.StateDeleting).WithOperation("manifest should be deleted"))
		}

• Adapt E2E Test in KLM
  • In warning_status_propagation_test.go:
    • Change "When Kyma Module is disabled with existing finalizer; Then there is no Module in KCP Kyma CR spec; And KCP Kyma CR is in "Warning" State; And Module Manifest CR is in a "Warning" State" to check CONSISTENTLY if Kyma and Manifest CR are in Warning state (First check with eventually condition and then with consistently; since this test should have been covered this scenaroi from the bug, but since it it flapping between the states the Eventually results to true
    • Maybe think of introducing a new util function to cover situations lwhere a condition first needs to be met (eventually) and then if it mets it should be consistent
    • I.e.

By("And Module Manifest CR is consistently in a \"Warning\" State")
			Eventually(CheckManifestIsInState).
				WithContext(ctx).
				WithArguments(kyma.GetName(), kyma.GetNamespace(), module.Name, controlPlaneClient,
					shared.StateWarning).
				Should(Succeed())
			Consistently(CheckManifestIsInState).
				WithContext(ctx).
				WithArguments(kyma.GetName(), kyma.GetNamespace(), module.Name, controlPlaneClient,
					shared.StateWarning).
				Should(Succeed())

[jeremyharisch]

Template-Operator Changes will be done in kyma-project/template-operator#147
Unitl then this issue is blcoked.