Infrastructure Manager - implement kubeconfig secret management

[akgalwas]

Description

The Infrastructure Manager must manage dynamic kubeconfigs.

Acceptance criteria:

• Infrastructure Manager can be installed on Gardener cluster.
• Infrastructure Manager creates secrets when a new Gardener Cluster CR is created.  #37
• Implement kubeconfig rotation #48
• Infrastructure Manager removes secrets when Gardener Cluster CR is deleted #39
• Infrastructure Manager is periodically triggered to ensure secrets are rotated when needed.
• It is possible to force a secret rotation with annotation added to the secret.

Reasons

In the long term the Infrastructure Manager will replace Provisioner. In the first step it will be responsible for kubeconfig management in the Kyma Control Plane.

[akgalwas]

The POC code is here: kyma-project/control-plane#3017.
What needs to be done to productise the POC:

• Code fetching the dynamic kubeconfig from Gardener
• Kubeconfig for shoot cluster, and control plane needs to be passed
• Tests, and code implemented as a part of POC must be reviewed, and refined.
• Cluster CR status needs to be set when error occurs, or the operator successfully creates/rotates secret

[akgalwas]

Workplan:

• Secret creation
  • Infrastructure Manager creates secrets when a new Gardener Cluster CR is created.  #37
  • Pass configuration parameters to the Infrastructure Manager #42
  • Fetching dynamic kubeconfig #47 @akgalwas
  • Integrate and test on Gardener cluster @akgalwas
• Secret deletion
  • Infrastructure Manager removes secrets when Gardener Cluster CR is deleted #39
• Secret rotation functionality
  • Implement kubeconfig rotation #48 @Disper, @akgalwas
  • Setup time-based reconciliation. Note: by default Kubebuilder runs reconciliation once in 10 hours. Please see this. If kubeconfig validity time is set to 24h, we are probably safe with the default settings. We should consider this thoroughly to avoid losing events.
• Additional stuff
  • adds configuration for protecode and whitesource scanning #41
  • Run tests on all prs + gitignore enhancement #38

[akgalwas]

There is couple of things to be done as follow up:

• Context management : we probably should pass the context from reconciliation loop everywhere
• Getting secret : there is no need to find the secret by label, as GardenerCluster spec contains secret name

[Disper]

Currently, the secret is rotated both periodically and if operator.kyma-project.io/force-kubeconfig-rotation is added to the GardenCluster CR. That means that secrets

• .data.config is regenerated
• .metadata.annotations.operator.kyma-project.io/last-sync is updated with the time of the rotation.

[Disper]

As of now, if you will get the kubeconfig from that secret using e.g. k get secret kubeconfig-md-im -n kcp-system -ojsonpath={.data.config} | base64 -D > ~/kubeconfig.yaml, you will still be able to access the cluster with that kubeconfig after the secret is rotated. Regardless if the rotation happened periodically or was forced.

@ebensom could you help us understand whether this is acceptable behavior?

[ebensom]

@Disper Yes AFAIK this is expected behavior, as tokens obtained via Gardener TokenRequest are OIDC tokens with exp field. When new token is requested via API, the old tokens are not invalidated, they are still valid until the expiration.

[Disper]

We will do a tiny refactoring in the code to reflect that we're not doing the revocation, but rotation.
But it will not affect the functionality so I'm closing the issue.