If backupentry reconciles before Shoot namespace gets created cluster creation will fail

[Kostov6]

How to categorize this issue?

/area quality
/kind bug

What happened:

gardener/extensions/pkg/controller/backupentry/genericactuator/actuator.go

Lines 51 to 59 in c024f37

	namespace := &corev1.Namespace{}
	if err := a.client.Get(ctx, kubernetesutils.Key(shootTechnicalID), namespace); err != nil {
	if apierrors.IsNotFound(err) {
	log.Info("SeedNamespace for shoot not found. Avoiding etcd backup secret deployment")
	return nil
	}
	log.Error(err, "Failed to get seed namespace")
	return err
	}

There is no dependency in the shoot reconciliation flow between the steps for deploying shoot namespace and deploying backup entry. So if the extension Backupentry gets created when shoot namespace is missing it can succeed without deploying backup secret. After that a second BE reconciliation won't happen. Therefore etcd-backup won't get created and shoot won't get created.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
Add time.Sleep(20*time.Seconds) in the beginning of pkg/gardenlet/operation/botanist/namespaces.go@DeploySeedNamespace and create local shoot

$ k -n garden-local get shoot local -o yaml | yq '.status.lastErrors'
- description: 'task "Deploying main and events etcd" failed: retry failed with context deadline exceeded, last error: Secret "etcd-backup" not found'
  lastUpdateTime: "2024-05-14T14:36:16Z"
  taskID: Deploying main and events etcd

Anything else we need to know?:

Environment:

• Gardener version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others: