Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet: add setting for configuring cloudProvider #1494

Merged
merged 2 commits into from Apr 19, 2021

Conversation

jpculp
Copy link
Member

@jpculp jpculp commented Apr 16, 2021

Issue number:
#1459

Description of changes:

Adds a new setting kubernetes.cloud-provider for configuring whether the cloud provider is aws or external.
Prior to this, the argument was hard-coded to aws.

Adds a migration for the new kubernetes.cloud-provider setting.

Testing done:

  • Build aws-k8s-1.19 ami.
  • Launch and connect to EKS cluster.
  • Deploy something to the node.
  • Connect to admin container and sudo sheltie.
  • Verify that there are no kubelet errors.
  • Repeat the same steps for aws-k8s-1.18.
  • Repeat the same steps for aws-k8s-1.17.
  • Repeat the same steps for aws-k8s-1.16.
  • Verify the migration works as expected.
  • Verify variant without a default cloud provider specified defaults to external.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@jpculp jpculp changed the title Kubelet cloud provider setting kubelet: add setting for configuring cloudProvider Apr 16, 2021
packages/kubernetes-1.16/kubelet-exec-start-conf Outdated Show resolved Hide resolved
packages/kubernetes-1.17/kubelet-env Outdated Show resolved Hide resolved
/// kubelet. It stores the original string and makes it accessible through standard traits.
#[derive(Debug, Clone, Eq, PartialEq, Hash)]
pub struct KubernetesCloudProvider {
inner: String,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe an enum would be better. Not a strong preference but it looks like we have enums for other settings.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd reach for an enum as well. :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not crazy about the enum for something this small. This block of code used KubernetesAuthenticationMode as a reference.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to replacing it with an enum later, but we would want to address #1501 first.

README.md Outdated
@@ -308,6 +308,7 @@ The following settings can be optionally set to customize the node labels and ta
The following settings are optional and allow you to further configure your cluster.
* `settings.kubernetes.cluster-domain`: The DNS domain for this cluster, allowing all Kubernetes-run containers to search this domain before the host's search domains. Defaults to `cluster.local`.
* `settings.kubernetes.standalone-mode`: Whether to run the kubelet in standalone mode, without connecting to an API server. Defaults to `false`.
* `settings.kubernetes.cloud-provider`: The cloud platform for this cluster. Defaults to `aws` for AWS variants, and `external` for other variants.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably say "cloud provider" rather than "cloud platform".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to avoid repeating the name of the setting, but if folks feel strongly or have a less redundant suggestion I'm open to changing it.

Copy link
Contributor

@etungsten etungsten Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep it in line with the term that the k8s official doc uses to remove any ambiguity on what this is for. See https://v1-18.docs.kubernetes.io/docs/concepts/cluster-administration/cloud-providers/

/// kubelet. It stores the original string and makes it accessible through standard traits.
#[derive(Debug, Clone, Eq, PartialEq, Hash)]
pub struct KubernetesCloudProvider {
inner: String,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd reach for an enum as well. :)

@jpculp jpculp force-pushed the kubelet-cloud-provider-setting branch from f022822 to 99d6483 Compare April 16, 2021 20:59
@jpculp
Copy link
Member Author

jpculp commented Apr 16, 2021

  • Removed the environment variable logic.

@jpculp jpculp requested review from webern and zmrow April 16, 2021 22:13
@jpculp
Copy link
Member Author

jpculp commented Apr 16, 2021

  • Tested all other support kubernetes versions.
  • Performed migration test with 1.19 instance.

{{~/unless}}
--cloud-provider {{settings.kubernetes.cloud-provider}} \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need the old logic here so that we pass an empty string to cloud-provider if standalone mode is enabled.

Adds a new setting `kubernetes.cloud-provider` for configuring whether
the cloud provider is `aws` or `external`. Prior to this, the argument
was hard-coded to `aws`.
Adds a migration for the new `kubernetes.cloud-provider` setting.
@jpculp jpculp force-pushed the kubelet-cloud-provider-setting branch from 99d6483 to 104122a Compare April 19, 2021 20:16
@jpculp
Copy link
Member Author

jpculp commented Apr 19, 2021

  • Reverted back to unless+else logic in kubelet-exec-start-conf.
  • Added default setting of external if settings.kubernetes.cloud-provider is absent from the variant's defaults.
  • Replaced the word platform with provider.

@jpculp jpculp requested a review from bcressey April 19, 2021 20:19
Copy link
Contributor

@zmrow zmrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🦺

Code looks good. Before you merge - let's make sure to build an image without the cloud-provider setting and make sure that it defaults to "external" like we think it should.

@jpculp
Copy link
Member Author

jpculp commented Apr 19, 2021

Tested migration with a variant that did not have a default cloud provider setting set. Launched as --cloud-provider aws with 1.0.8 and after upgrading to a modified 1.1.0 variant launched as --cloud-provider external.

@jpculp jpculp merged commit 025ba5d into bottlerocket-os:develop Apr 19, 2021
@jpculp jpculp deleted the kubelet-cloud-provider-setting branch April 19, 2021 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants