New seed bootstrap tokens cannot be created for existing seeds

[vpnachev]

How to categorize this issue?

/area robustness ops-productivity
/kind bug
/priority 3

What happened:
The client certificate of one of the managedseed clusters has expired and it has not been renewed on purpose.
Then, the parent gardenlet can create new bootstrap token and let the child seed use it to request new certificate. According to the code

gardener/pkg/gardenlet/controller/managedseed/managedseed_actuator.go

Lines 687 to 694 in 981aa35

	isAlreadyBootstrapped, err := isAlreadyBootstrapped(ctx, shootClient.Client(), gcc.KubeconfigSecret)
	if err != nil {
	return "", err
	}
	if isAlreadyBootstrapped {
	return "", nil
	}

, to trigger this action the secret from gardenClientConnection.kubeconfigSecret of the child seeds must not exist. I have deleted it, but then the parent gardenlet is failing to generate the token with error like

could not deploy gardenlet into shoot garden/foo: secrets "bootstrap-token-5111ca" is forbidden: User "gardener.cloud:system:seed:bar" cannot get resource "secrets" in API group "" in the namespace "kube-system": no relationship found between seed ''bar' and this object

This gardener landscape has the SeedAuthorizer enabled and it is indeed rejecting the request with

{"level":"info","ts":"2021-09-18T06:46:45.704Z","logger":"seedauthorizer","msg":"SEED DENY: 'bar' authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc001db1e40), Verb:\"get\", Namespace:\"kube-system\", APIGroup:\"\", APIVersion:\"v1\", Resource:\"secrets\", Subresource:\"\", Name:\"bootstrap-token-5111ca\", ResourceRequest:true, Path:\"\"}"} 

What you expected to happen:
A new bootstrap token to be successfully generated and gardenlet re-deployed.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Gardener version: 981aa35
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[rfranzke]

There is no way to know for the SeedAuthorizer that the gardenlet's client certificate might be expired. We might want to think about an annotation which can be set by an operator to allow such calls again.

[rfranzke]

/assign

[timebertt]

This gist provides a script to manually rebootstrap an existing seed as a workaround for this issue.