Gardener components no longer use admin kubeconfig for local garden dev setup

[rfranzke]

How to categorize this PR?

/area dev-productivity
/kind enhancement

What this PR does / why we need it:
Today, all Gardener components are using the admin kubeconfig for the local garden development setup. This has the downside that the RBAC rules which we maintain in

Which issue(s) this PR fixes:
Softly related to #1723 and #1724.

Special notes for your reviewer:
/squash

Release note:

When using the local garden development environment, the Gardener components do now use dedicated kubeconfigs constrained by RBAC rules (earlier, they were always using the admin kubeconfig).

[rfranzke]

@vpnachev - an idea as follow-up of this PR to also make the gardenlet use the RBAC privileges in the seed that you plan to apply with #3763:

In https://github.com/gardener/gardener/blob/master/hack/local-development/start-gardenlet,

1. create a ServiceAccount in the garden namespace of all the seed clusters (by reading all Seeds and using the kubeconfig referenced by .spec.secretRef)
2. apply the RBAC resources for the gardenlet in the seeds
3. create a new, temporary kubeconfig which uses the token of the ServiceAccount created in 1.
4. update the kubeconfig in the .spec.secretRef of the various seeds

It might have drawbacks, just brain-storming here. WDYT?

[vpnachev]

@vpnachev - an idea as follow-up of this PR to also make the gardenlet use the RBAC privileges in the seed that you plan to apply with #3763:

In https://github.com/gardener/gardener/blob/master/hack/local-development/start-gardenlet,

1. create a `ServiceAccount` in the `garden` namespace of all the seed clusters (by reading all `Seed`s and using the kubeconfig referenced by `.spec.secretRef`)

2. apply the RBAC resources for the gardenlet in the seeds

3. create a new, temporary kubeconfig which uses the token of the `ServiceAccount` created in 1.

4. update the kubeconfig in the `.spec.secretRef` of the various seeds

It might have drawbacks, just brain-storming here. WDYT?

Thanks for the suggestion! However, I am still thinking that it should be as it is today. Why

1. A new seed can be added at any time and gardenlet will use the configured kubeconfig. On restart of gardenlet, it will create the service account and bound the (cluster)roles to it. This can lead to authorization issues if new permissions are required, or if the seed is deleted with no restart of gardenlet such missing permissions will not be detected.
2. It still possible to run gardenlet remotely in productive landscapes with the whatever kubeconfig is configured by the landscape administrator. Should we implement such automation in this case - I think no. It would be a bit weird gardenlet to amend externally configured kubeconfig.

For now I will not add anything like this in #3763 as it has been prolonged too much and a lot of conflicts needs to be resolved. Such automation can be contributed later, if we find a suitable solution.

[rfranzke]

@vpnachev My recommendation was only for the local development environment. Of course, in production code we should not have anything like this. I just think we should find a way to make sure to detect issues in the gardenlet's RBAC rules already when developing locally. We don't have to follow above approach (it was just some short brain-storming/food for thought), and we don't have to do anything now with #3763, but I think we should eventually have some solution for it.

[stoyanr]

/assign

[stoyanr]

Few minor nits and some questions regarding why is this enabled only for local-garden / $NODELESS as opposed to other local environment types, e.g. remote-garden / $REMOTE.

[stoyanr]

/lgtm