Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove SeedSelector from GardenletConfiguration #4306

Merged
merged 6 commits into from Jul 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -161,10 +161,6 @@ data:
{{- if .Values.global.gardenlet.config.featureGates }}
xrstf marked this conversation as resolved.
Show resolved Hide resolved
featureGates:
{{ toYaml .Values.global.gardenlet.config.featureGates | indent 6 }}
{{- end }}
{{- if .Values.global.gardenlet.config.seedSelector }}
seedSelector:
{{ toYaml .Values.global.gardenlet.config.seedSelector | indent 6 }}
{{- end }}
{{- if .Values.global.gardenlet.config.seedConfig }}
seedConfig:
Expand Down
1 change: 0 additions & 1 deletion charts/gardener/gardenlet/values.yaml
Expand Up @@ -155,7 +155,6 @@ global:
# labels:
# istio: ingressgateway-handler-2

# seedSelector: {}
# seedConfig: {}
# logging:
# fluentBit:
Expand Down
1 change: 0 additions & 1 deletion cmd/gardenlet/app/gardenlet.go
Expand Up @@ -323,7 +323,6 @@ func NewGardenlet(ctx context.Context, cfg *config.GardenletConfiguration) (*Gar
}

seedClientMapBuilder := clientmapbuilder.NewSeedClientMapBuilder().
WithInCluster(cfg.SeedSelector == nil).
WithClientConnectionConfig(&cfg.SeedClientConnection.ClientConnectionConfiguration)
shootClientMapBuilder := clientmapbuilder.NewShootClientMapBuilder().
WithClientConnectionConfig(&cfg.ShootClientConnection.ClientConnectionConfiguration)
Expand Down
58 changes: 29 additions & 29 deletions docs/concepts/gardenlet.md
Expand Up @@ -111,28 +111,34 @@ for example, using command `kubectl certificate approve seed-csr-<...>`).
If that doesn’t happen within 15 minutes,
the gardenlet repeats the process and creates another CSR.

## Seed Config versus Seed Selector
xrstf marked this conversation as resolved.
Show resolved Hide resolved

The usage of the gardenlet is flexible:

| Usage | Description |
|:---|:---|
| `seedConfig` | Run one gardenlet per seed cluster inside the seed cluster itself.|
| `seedSelector` | Use one gardenlet to manage multiple seed clusters. The gardenlet can run outside of the seed cluster.|

> For production use it’s recommended to go for the `seedConfig` option,
> because it makes scaling easier and leads to a better distribution of responsibilities.

* Provide a `seedConfig` that contains information about the seed cluster itself
if you want the gardenlet in the standard way,
see [the example gardenlet configuration](../../example/20-componentconfig-gardenlet.yaml).
Once bootstrapped, the gardenlet creates and updates its `Seed` object itself.

* Provide a `seedSelector` that incorporates a label selector for the
targeted seed clusters if you want the gardenlet to manage multiple seeds,
see [the example gardenlet configuration](../../example/20-componentconfig-gardenlet.yaml).
In this case, you have to create the `Seed` objects
together with a `kubeconfig` pointing to the cluster yourself.
## Configuring the Seed to work with

The Gardenlet works with a single seed, which must be configured in the
`GardenletConfiguration` under `.seedConfig`. This must be a copy of the
`Seed` resource, for example (see `example/20-componentconfig-gardenlet.yaml`
for a more complete example):

```yaml
apiVersion: gardenlet.config.gardener.cloud/v1alpha1
kind: GardenletConfiguration
seedConfig:
metadata:
name: my-seed
spec:
provider:
type: aws
# ...
secretRef:
name: my-seed-secret
namespace: garden
```

When using `make start-gardenlet`, the corresponding script will automatically
xrstf marked this conversation as resolved.
Show resolved Hide resolved
fetch the seed cluster's `kubeconfig` based on the `seedConfig.spec.secretRef`
and set the environment accordingly.

On startup, gardenlet registers a `Seed` resource using the given template
in `seedConfig` if it's not present already.

## Component Configuration

Expand Down Expand Up @@ -166,13 +172,7 @@ It’s used as a liveness probe in the `Deployment` of the gardenlet.
If the gardenlet fails to renew its lease
then the endpoint returns `500 Internal Server Error`, otherwise it returns `200 OK`.

> ⚠️<br> In case the gardenlet is managing multiple seeds
> (that is, a seed selector is used) then the `/healthz`
> reports `500 Internal Server Error` if there is at least one seed
> for which it couldn’t renew its lease. Only if it can renew the lease
> for all seeds it reports `200 OK`.

Please note, that the `/healthz` only indicates whether the gardenlet
Please note that the `/healthz` only indicates whether the gardenlet
could successfully probe the Seed's API server and renew the lease with
the Garden cluster.
It does *not* show that the Gardener extension API server (with the Gardener resource groups)
Expand Down
181 changes: 89 additions & 92 deletions docs/deployment/deploy_gardenlet_manually.md

Large diffs are not rendered by default.

15 changes: 5 additions & 10 deletions docs/usage/managed_seed.md
Expand Up @@ -8,18 +8,18 @@ An existing shoot can be registered as a seed by creating a `ManagedSeed` resour
* `gardenlet` deployment parameters, such as the number of replicas, the image, etc.
* The `GardenletConfiguration` resource that contains controllers configuration, feature gates, and a `seedConfig` section that contains the `Seed` spec and parts of its metadata.
* Additional configuration parameters, such as the garden connection bootstrap mechanism (see [TLS Bootstrapping](../concepts/gardenlet.md#tls-bootstrapping)), and whether to merge the provided configuration with the configuration of the parent `gardenlet`.
Either the `seedTemplate` or the `gardenlet` section must be specified, but not both:

Either the `seedTemplate` or the `gardenlet` section must be specified, but not both:

* If the `seedTemplate` section is specified, `gardenlet` is not deployed to the shoot, and a new `Seed` resource is created based on the template.
* If the `gardenlet` section is specified, `gardenlet` is deployed to the shoot, and it registers a new seed upon startup based on the `seedConfig` section of the `GardenletConfiguration` resource.

Note the following important aspects:

* Unlike the `Seed` resource, the `ManagedSeed` resource is namespaced. Currently, managed seeds are restricted to the `garden` namespace.
* The newly created `Seed` resource always has the same name as the `ManagedSeed` resource. Attempting to specify a different name in `seedTemplate` or `seedConfig` will fail.
* The newly created `Seed` resource always has the same name as the `ManagedSeed` resource. Attempting to specify a different name in `seedTemplate` or `seedConfig` will fail.
* The `ManagedSeed` resource must always refer to an existing shoot. Attempting to create a `ManagedSeed` referring to a non-existing shoot will fail.
* A shoot that is being referred to by a `ManagedSeed` cannot be deleted. Attempting to delete such a shoot will fail.
* A shoot that is being referred to by a `ManagedSeed` cannot be deleted. Attempting to delete such a shoot will fail.
* You can omit practically everything from the `seedTemplate` or `gardenlet` section, including all or most of the `Seed` spec fields. Proper defaults will be supplied in all cases, based either on the most common use cases or the information already available in the `Shoot` resource.
* Some `Seed` spec fields, for example the provider type and region, networking CIDRs for pods, services, and nodes, etc., must be the same as the corresponding `Shoot` spec fields of the shoot that is being registered as seed. Attempting to use different values (except empty ones, so that they are supplied by the defaulting mechanims) will fail.

Expand Down Expand Up @@ -55,9 +55,6 @@ spec:
shoot:
name: crazy-botany
seedTemplate:
metadata:
labels:
seed.gardener.cloud/gardenlet: local
spec:
dns:
ingressDomain: ""
Expand All @@ -69,8 +66,6 @@ spec:
region: ""
```

Note the `seed.gardener.cloud/gardenlet: local` label above. It stands for the label that is used in a `seedSelector` field of a `gardenlet` that takes care of multiple seeds. This label can be omitted if the `seedSelector` field selects all seeds. If there is no gardenlet running outside the cluster and selecting the seed, it won't be reconciled and no shoots will be scheduled on it.

For an example that uses non-default configuration, see [55-managed-seed-seedtemplate.yaml](../../example/55-managedseed-seedtemplate.yaml)

## Migrating from the `use-as-seed` Annotation to `ManagedSeeds`
Expand All @@ -97,6 +92,6 @@ Option | Description
`apiServer.autoscaler.maxReplicas` | Controls the maximum number of `kube-apiserver` replicas for the shooted seed cluster.
`apiServer.replicas` | Controls how many `kube-apiserver` replicas the shooted seed cluster gets by default.

For backward compatibility, it is still possible to specify these options via the `shoot.gardener.cloud/managed-seed-api-server` annotation, using exactly the same syntax as before.
For backward compatibility, it is still possible to specify these options via the `shoot.gardener.cloud/managed-seed-api-server` annotation, using exactly the same syntax as before.

If you use any of these fields in any or your `use-as-seed` annotations, instead of removing the annotation completely as mentioned above, simply rename it to `managed-seed-api-server`, keeping these fields, and removing everything else.
2 changes: 1 addition & 1 deletion docs/usage/shooted_seed.md
Expand Up @@ -56,7 +56,7 @@ It is strongly recommended to use such resources directly to register shoots as
Option | Description
--- | ---
`true` | Registers the cluster as a seed cluster. Automatically deploys the gardenlet into the shoot cluster, unless specified otherwise (e.g. setting the `no-gardenlet` flag).
`no-gardenlet` | Prevents the automatic deployment of the gardenlet into the shoot cluster. Instead, the `Seed` object will be created with the assumption that another gardenlet will be responsible for managing it (according to its `seedSelector` configuration).
`no-gardenlet` | Prevents the automatic deployment of the gardenlet into the shoot cluster. Instead, the `Seed` object will be created with the assumption that another gardenlet will be responsible for managing it (according to its `seedConfig` configuration).
`disable-capacity-reservation` | Set `spec.settings.excessCapacity.enabled` in the seed cluster to false (see [/example/50-seed.yaml](../../example/50-seed.yaml)).
`invisible` | Set `spec.settings.scheduling.visible` in the seed cluster to false (see [/example/50-seed.yaml](../../example/50-seed.yaml))
`visible` | Set `spec.settings.scheduling.visible` in the seed cluster to true (see [/example/50-seed.yaml](../../example/50-seed.yaml)) (**default**).
Expand Down
5 changes: 1 addition & 4 deletions example/55-managedseed-seedtemplate.yaml
Expand Up @@ -8,10 +8,7 @@ spec:
name: crazy-botany
# seedTemplate is a template for a Seed object, that should be used to register a given cluster as a Seed.
# When seedTemplate is specified, the ManagedSeed controller will not deploy a gardenlet into the cluster
# and an existing gardenlet reconciling the new Seed is required.
# and an existing gardenlet reconciling the new Seed is required.
seedTemplate:
# metadata:
# labels:
# seed.gardener.cloud/gardenlet: local # Label to use in seedSelector of the existing gardenlet
spec: # Seed spec
# <See `spec` in `50-seed.yaml` for more details>
Expand Up @@ -71,10 +71,6 @@ func ValidateLandscaperImports(imports *imports.Imports) field.ErrorList {
func validateGardenletConfiguration(gardenletConfig *gardenletconfig.GardenletConfiguration, fldPath *field.Path) field.ErrorList {
allErrs := field.ErrorList{}

if gardenletConfig.SeedSelector != nil {
allErrs = append(allErrs, field.Forbidden(fldPath.Child("seedSelector"), "seed selector is forbidden, provide a seedConfig instead."))
}

if gardenletConfig.SeedConfig == nil {
return append(allErrs, field.Required(fldPath.Child("seedConfig"), "the seed configuration has to be provided. This is used to automatically register the seed."))
}
Expand Down
Expand Up @@ -115,31 +115,6 @@ var _ = Describe("ValidateLandscaperImports", func() {
})

Context("validate Gardenlet configuration", func() {
It("should validate that the Seed selector is not set", func() {
// only pick one required Gardenlet component configuration to show that the configuration is indeed verified
// neither seedSelector nor seedConfig are provided. One of them has to be set to
// pass the validation of the GardenletConfiguration
gardenletConfiguration.SeedSelector = &metav1.LabelSelector{
MatchLabels: map[string]string{
"x": "y",
},
}
gardenletConfiguration.SeedConfig = nil
landscaperGardenletImport.ComponentConfiguration = &gardenletConfiguration

errorList := ValidateLandscaperImports(landscaperGardenletImport)
Expect(errorList).To(ConsistOf(
PointTo(MatchFields(IgnoreExtras, Fields{
"Type": Equal(field.ErrorTypeForbidden),
"Field": Equal("componentConfiguration.seedSelector"),
})),
PointTo(MatchFields(IgnoreExtras, Fields{
"Type": Equal(field.ErrorTypeRequired),
"Field": Equal("componentConfiguration.seedConfig"),
})),
))
})

It("should validate that the kubeconfig in the GardenClientConnection is not set", func() {
gardenletConfiguration.GardenClientConnection = &gardenletconfigv1alpha1.GardenClientConnection{
ClientConnectionConfiguration: componentbaseconfigv1alpha1.ClientConnectionConfiguration{
Expand Down
5 changes: 0 additions & 5 deletions pkg/apis/seedmanagement/validation/managedseed.go
Expand Up @@ -293,11 +293,6 @@ func validateImage(image *seedmanagement.Image, fldPath *field.Path) field.Error
func validateGardenletConfiguration(gardenletConfig *config.GardenletConfiguration, bootstrap seedmanagement.Bootstrap, mergeWithParent bool, fldPath *field.Path, inTemplate bool) field.ErrorList {
allErrs := field.ErrorList{}

// Ensure seed selector is not specified
if gardenletConfig.SeedSelector != nil {
allErrs = append(allErrs, field.Forbidden(fldPath.Child("seedSelector"), "seed selector is forbidden"))
}

// Ensure name is not specified since it will be set by the controller
if gardenletConfig.SeedConfig != nil && gardenletConfig.SeedConfig.Name != "" {
allErrs = append(allErrs, field.Forbidden(fldPath.Child("seedConfig", "metadata", "name"), "seed name is forbidden"))
Expand Down
9 changes: 4 additions & 5 deletions pkg/client/kubernetes/clientmap/builder/deleg_builder.go
Expand Up @@ -27,7 +27,7 @@ import (
// based on the type of the key (e.g. a call with keys.ForShoot() will be delegated to the ShootClientMap).
type DelegatingClientMapBuilder struct {
gardenClientMapFunc func(logger logrus.FieldLogger) (clientmap.ClientMap, error)
seedClientMapFunc func(gardenClients clientmap.ClientMap, logger logrus.FieldLogger) (clientmap.ClientMap, error)
seedClientMapFunc func(logger logrus.FieldLogger) (clientmap.ClientMap, error)
shootClientMapFunc func(gardenClients, seedClients clientmap.ClientMap, logger logrus.FieldLogger) (clientmap.ClientMap, error)
plantClientMapFunc func(gardenClients clientmap.ClientMap, logger logrus.FieldLogger) (clientmap.ClientMap, error)

Expand Down Expand Up @@ -69,18 +69,17 @@ func (b *DelegatingClientMapBuilder) WithGardenClientMapBuilder(builder *GardenC

// WithSeedClientMap sets the ClientMap that should be used for Seed clients.
func (b *DelegatingClientMapBuilder) WithSeedClientMap(clientMap clientmap.ClientMap) *DelegatingClientMapBuilder {
b.seedClientMapFunc = func(_ clientmap.ClientMap, _ logrus.FieldLogger) (clientmap.ClientMap, error) {
b.seedClientMapFunc = func(_ logrus.FieldLogger) (clientmap.ClientMap, error) {
return clientMap, nil
}
return b
}

// WithSeedClientMapBuilder sets a ClientMap builder that should be used to build a ClientMap for Seed clients.
func (b *DelegatingClientMapBuilder) WithSeedClientMapBuilder(builder *SeedClientMapBuilder) *DelegatingClientMapBuilder {
b.seedClientMapFunc = func(gardenClients clientmap.ClientMap, logger logrus.FieldLogger) (clientmap.ClientMap, error) {
b.seedClientMapFunc = func(logger logrus.FieldLogger) (clientmap.ClientMap, error) {
return builder.
WithLogger(logger.WithField("ClientMap", "SeedClientMap")).
WithGardenClientMap(gardenClients).
Build()
}
return b
Expand Down Expand Up @@ -139,7 +138,7 @@ func (b *DelegatingClientMapBuilder) Build() (clientmap.ClientMap, error) {
var seedClients, shootClients, plantClients clientmap.ClientMap

if b.seedClientMapFunc != nil {
seedClients, err = b.seedClientMapFunc(gardenClients, b.logger)
seedClients, err = b.seedClientMapFunc(b.logger)
if err != nil {
return nil, fmt.Errorf("failed to construct seed ClientMap: %w", err)
}
Expand Down
4 changes: 2 additions & 2 deletions pkg/client/kubernetes/clientmap/builder/deleg_builder_test.go
Expand Up @@ -73,7 +73,7 @@ var _ = Describe("DelegatingClientMapBuilder", func() {
Context("#seedClientMapFunc", func() {
It("should be set correctly by WithSeedClientMap", func() {
builder := NewDelegatingClientMapBuilder().WithSeedClientMap(fakeSeedClientMap)
Expect(builder.seedClientMapFunc(nil, nil)).To(BeIdenticalTo(fakeSeedClientMap))
Expect(builder.seedClientMapFunc(nil)).To(BeIdenticalTo(fakeSeedClientMap))
})

It("should be set correctly by WithSeedClientMapBuilder", func() {
Expand Down Expand Up @@ -154,7 +154,7 @@ var _ = Describe("DelegatingClientMapBuilder", func() {
builder := NewDelegatingClientMapBuilder().
WithLogger(fakeLogger).
WithGardenClientMap(fakeGardenClientMap)
builder.seedClientMapFunc = func(clientmap.ClientMap, logrus.FieldLogger) (clientmap.ClientMap, error) {
builder.seedClientMapFunc = func(logrus.FieldLogger) (clientmap.ClientMap, error) {
return nil, fakeErr
}
clientMap, err := builder.Build()
Expand Down
33 changes: 0 additions & 33 deletions pkg/client/kubernetes/clientmap/builder/seed_builder.go
Expand Up @@ -15,23 +15,18 @@
package builder

import (
"context"
"fmt"

"github.com/sirupsen/logrus"
baseconfig "k8s.io/component-base/config"

"github.com/gardener/gardener/pkg/client/kubernetes"
"github.com/gardener/gardener/pkg/client/kubernetes/clientmap"
"github.com/gardener/gardener/pkg/client/kubernetes/clientmap/internal"
"github.com/gardener/gardener/pkg/client/kubernetes/clientmap/keys"
)

// SeedClientMapBuilder can build a ClientMap which can be used to construct a ClientMap for requesting and storing
// clients for Seed clusters.
type SeedClientMapBuilder struct {
gardenClientFunc func(ctx context.Context) (kubernetes.Interface, error)
inCluster bool
clientConnectionConfig *baseconfig.ClientConnectionConfiguration

logger logrus.FieldLogger
Expand All @@ -48,29 +43,6 @@ func (b *SeedClientMapBuilder) WithLogger(logger logrus.FieldLogger) *SeedClient
return b
}

// WithGardenClientMap sets the ClientMap that should be used to retrieve Garden clients.
func (b *SeedClientMapBuilder) WithGardenClientMap(clientMap clientmap.ClientMap) *SeedClientMapBuilder {
b.gardenClientFunc = func(ctx context.Context) (kubernetes.Interface, error) {
return clientMap.GetClient(ctx, keys.ForGarden())
}
return b
}

// WithGardenClientSet sets the ClientSet that should be used as the Garden client.
func (b *SeedClientMapBuilder) WithGardenClientSet(clientSet kubernetes.Interface) *SeedClientMapBuilder {
b.gardenClientFunc = func(ctx context.Context) (kubernetes.Interface, error) {
return clientSet, nil
}
return b
}

// WithInCluster sets the inCluster attribute of the builder. If true, the created ClientSets will use in-cluster communication
// (using the provided ClientConnectionConfig.Kubeconfig or fallback to mounted ServiceAccount if unset).
func (b *SeedClientMapBuilder) WithInCluster(inCluster bool) *SeedClientMapBuilder {
b.inCluster = inCluster
return b
}

// WithClientConnectionConfig sets the ClientConnectionConfiguration that should be used by ClientSets created by this ClientMap.
func (b *SeedClientMapBuilder) WithClientConnectionConfig(cfg *baseconfig.ClientConnectionConfiguration) *SeedClientMapBuilder {
b.clientConnectionConfig = cfg
Expand All @@ -82,16 +54,11 @@ func (b *SeedClientMapBuilder) Build() (clientmap.ClientMap, error) {
if b.logger == nil {
return nil, fmt.Errorf("logger is required but not set")
}
if b.gardenClientFunc == nil {
return nil, fmt.Errorf("garden client is required but not set")
}
if b.clientConnectionConfig == nil {
return nil, fmt.Errorf("clientConnectionConfig is required but not set")
}

return internal.NewSeedClientMap(&internal.SeedClientSetFactory{
GetGardenClient: b.gardenClientFunc,
InCluster: b.inCluster,
ClientConnectionConfig: *b.clientConnectionConfig,
}, b.logger), nil
}