Skip to content

@justinsb justinsb released this Oct 24, 2020

This version contains a critical update to etcd-manager: 1 year after creation (or first adopting etcd-manager), clusters will stop responding due to expiration of a TLS certificate. Upgrading kops to 1.18.1 (or the latest versions of the 1.15, 1.16, 1.17 or 1.18 series) and running kops update followed by a kops rolling-update will fix the issue. Please see the advisory for the full details.


kops 1.18.2 is the next patch release in the 1.18 series of kops, offering support for kubernetes 1.18.

Please see the release notes for the full list of changes.

Release notes for kops 1.18 series

Significant changes

  • The default image has been updated to Ubuntu 20.04 (Focal). Consequently, the SSH user changed to ubuntu and the Linux kernel changed to version 5.4.

  • To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Support for RHEL 8 and CentOS 8 has been added.

  • Support for Amazon Linux 2 has been improved and will work with the default Docker version.

  • containerd has been added and can be selected as an alternate container runtime for Kubernetes. Enable by using the --container-runtime containerd flag when creating a cluster or by setting spec.containerRuntime: containerd.

  • Rolling updates now support surging and parallelism within an instance group. For details see the documentation.

  • Cilium CNI can now use AWS networking natively through the AWS ENI IPAM mode. Kops can also run a Kubernetes cluster entirely without kube-proxy using Cilium's BPF NodePort implementation.

  • Cilium CNI can now use a dedicated etcd cluster managed by etcd-manager for synchronizing agent state instead of CRDs.

  • The Terraform target now supports Terraform 0.12 syntax (HCL2) by default. See the Required Actions item below.

  • New clusters in GCE are configured to run the metadata-proxy by default. The proxy runs as a DaemonSet and lands on nodes with the nodeLabel cloud.google.com/metadata-proxy-ready: "true". If you want to enable metadata-proxy on an existing cluster/instance group, add that nodeLabel to your instancegroup specs (kops edit ig ...) and run kops update cluster. When the changes are applied, the proxy will roll out to those targeted nodes.

  • GCE has a new flag: --gce-service-account. This takes the email of an existing GCP service account and launches the instances with it. This setting applies to the whole cluster (ie: it is not currently designed to support Instance Groups with different service accounts). If you do not specify a service account during cluster creation, the default compute service account will be used which matches the prior behavior.

  • Google API client libraries updated from v0.beta to v1.

  • Support for NodeLocalDNS cache.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for Terraform 0.12 support. See Required Actions below.

  • Support for the CoreOS OS distribution has been removed. Users should consider Flatcar as a replacement.

  • Support for the Debian 8 (Jessie) OS distribution has been removed.

  • The Docker health-check service has been disabled by default. It shouldn't be needed anymore, but it can still be enabled by setting spec.docker.healthCheck: true. It is recommended to also check node-problem-detector and draino as replacements. See Required Actions below.

  • Network and internet access for docker run containers has been disabled by default, to avoid any unwanted interaction between the Docker firewall rules and the firewall rules of netwok plugins. This was the default since the early days of Kops, but a race condition in the Docker startup sequence changed this behaviour in more recent years. To re-enable, set spec.docker.ipTables: true and spec.docker.ipMasq: true.

  • Lyft CNI plugin default subnet tags changed from from Type: pod to KubernetesCluster: myclustername.mydns.io. Subnets intended for use by the plugin will need to be tagged with this new tag and additional tag filters may need to be added to the cluster spec in order to achieve the desired set of subnets.

  • Support for basic authentication has been disabled by default for Kubernetes 1.18 and will be removed in Kubernetes 1.19.

  • Support for static tokens has been disabled by default for Kubernetes 1.18 and later. To re-enable, see the Security Notes for Kubernetes. We intend to remove support entirely in a future kops version, so file an issue with your use case if you need this feature.

  • Support for Kubernetes versions prior to 1.9 has been removed.

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. See Required Actions below.

  • Support for the "Legacy" etcd provider has been removed for Kubernetes versions 1.18 and higher. Such clusters will need to migrate to the default "Manager" etcd provider. To migrate, see the etcd migration documentation.

  • A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

  • The kops.k8s.io/v1alpha1 API has been removed. Users of kops replace will need to supply v1alpha2 resources.

  • Please see the notes in the 1.15 release about the apiGroup changing from kops to kops.k8s.io

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to support Terraform 0.12.
    Terraform 0.12 no longer supports resource names starting with digits. In Kops, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.

    • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
    • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

    To prevent downtime, follow these steps with the new version of Kops:

    KOPS_FEATURE_FLAGS=-Terraform-0.12 kops update cluster --target terraform ...
    # Use Terraform <0.12
    terraform plan
    # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
    # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
    terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
    terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
    terraform plan
    # Ensure these resources are no longer being destroyed and recreated
    terraform apply
    

    Kops will now output Terraform 0.12 syntax with the normal workflow:

    kops update cluster --target terraform ...
    # Use Terraform 0.12. This plan should be a no-op
    terraform plan
    
  • Users that need the Docker health-check service will need to explicitly enable it:

  kops edit cluster
  # Add the following section
  spec:
    docker:
      healthCheck: true
  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of Kops.

    To enable the Pod priority feature, follow these steps:

    kops edit cluster
    # Add the following section
    spec:
      kubelet:
        featureGates:
          PodPriority: "true"
    
  • If a custom Kops build was used on a cluster, a kops-controller Deployment may have been created that should get deleted.
    Run kubectl -n kube-system delete deployment kops-controller after upgrading to Kops 1.16.0-beta.1 or later.

Known Issues

  • AWS clusters with an ACM certificate attached to the API ELB (the cluster's spec.api.loadBalancer.sslCertificate is set) will need to reenable basic auth to use the kubeconfig context created by kops export kubecfg. Set spec.kubeAPIServer.disableBasicAuth: false before running kops export kubecfg. See #9756 for more information.

Deprecations

  • Support for Kubernetes versions 1.9 and 1.10 are deprecated and will be removed in kops 1.19.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of Kops.

  • Support for the Romana networking provider is deprecated and will be removed in kops 1.19.

  • Support for legacy IAM permissions is deprecated and will be removed in kops 1.19.


All changes from 1.18.1 to 1.18.2

Please see the release notes for the full list of changes.

Assets 17
Pre-release
Pre-release

@hakman hakman released this Oct 15, 2020

Release notes for kops 1.19 series

(The kops 1.19 release has not been released yet; this is a document to gather the notes prior to the release).

Significant changes

Changes to kubernetes config export

Kops will no longer automatically export the kubernetes config on kops update cluster. In order to export the config on cluster update, you need to either add the --user <user> to reference an existing user, or --admin to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.

Similarly, kops export kubecfg will also require passing either the --admin or --user flag if the context does not already exist.

By default, the credentials of any exported admin user now have a lifetime of 18 hours. The lifetime of the exported
credentials may be specified as a value of the --admin flag. To get the previous behavior, specify --admin=87600h to either kops update cluster or kops export kubecfg.

kops create cluster --yes exports the admin user along with rest of the cluster config, as was the previous behaviour (except for the 18-hour validity).

OpenStack Cinder plugin

Kops will install the Cinder plugin for kops running kubernetes 1.16 or newer. If you already have this plugin installed you should remove it before upgrading.

If you already have a default StorageClass, you should set cloudConfig.Openstack.BlockStorage.CreateStorageClass: false to prevent kops from installing one.

Other significant changes

  • New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.

  • On AWS kops now defaults to using launch templates instead of launch configurations.

  • Clusters using the Amazon VPC CNI provider now perform an ec2.DescribeInstanceTypes call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue.

  • There is now Alpha support for Hashicorp Vault as a store for secrets and keys. See the Vault state store docs.

  • New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.

  • The kops update cluster command will now refuse to run on a cluster that
    has been updated by a newer version of kops unless it is given the --allow-kops-downgrade flag.

  • The lifetimes of certificates used by various components have been substantially reduced.
    The certificates on a node will expire sometime between 455 and 485 days after the node's creation.
    The expiration times vary randomly so that nodes are likely to have their certs expire at different times than other nodes.

  • New command for deleting a single instance: kops delete instance

  • Metrics Server is now available as a configurable addon. Add spec.metricsServer.enabled: true to the cluster spec to enable.

Breaking changes

  • Support for Kubernetes 1.9 and 1.10 has been removed.

  • Support for the Romana networking provider has been removed.

  • Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kops 1.20 by setting the LegacyIAM feature flag.

Required Actions

Deprecations

  • Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kops 1.20.

  • Support for Terraform version 0.11 has been deprecated and will be removed in kops 1.20.

  • Support for feature flag Terraform-0.12 has been deprecated and will be removed in kops 1.20. All generated Terraform HCL2/JSON files will support versions 0.12.26+ and 0.13.0+.

  • The manifest based metrics server addon has been deprecated in favour of a configurable addon.

Partial change list

1.19.0-alpha.4 to 1.19.0-alpha.5

Assets 30
Pre-release
Pre-release

@hakman hakman released this Sep 16, 2020

Release notes for kops 1.19 series

(The kops 1.19 release has not been released yet; this is a document to gather the notes prior to the release).

Significant changes

Changes to kubernetes config export

Kops will no longer automatically export the kubernetes config on kops update cluster. In order to export the config on cluster update, you need to either add the --user <user> to reference an existing user, or --admin to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.

Similarly, kops export kubecfg will also require passing either the --admin or --user flag if the context does not already exist.

By default, the credentials of any exported admin user now have a lifetime of 18 hours. The lifetime of the exported
credentials may be specified as a value of the --admin flag. To get the previous behavior, specify --admin=87600h to either kops update cluster or kops export kubecfg.

kops create cluster --yes exports the admin user along with rest of the cluster config, as was the previous behaviour (except for the 18-hour validity).

OpenStack Cinder plugin

Kops will install the Cinder plugin for kops running kubernetes 1.16 or newer. If you already have this plugin installed you should remove it before upgrading.

If you already have a default StorageClass, you should set cloudConfig.Openstack.BlockStorage.CreateStorageClass: false to prevent kops from installing one.

Other significant changes

  • New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.

  • On AWS kops now defaults to using launch templates instead of launch configurations.

  • Clusters using the Amazon VPC CNI provider now perform an ec2.DescribeInstanceTypes call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue.

  • There is now Alpha support for Hashicorp Vault as a store for secrets and keys. See the Vault state store docs.

  • New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.

  • The kops update cluster command will now refuse to run on a cluster that
    has been updated by a newer version of kops unless it is given the --allow-kops-downgrade flag.

  • The lifetimes of certificates used by various components have been substantially reduced.
    The certificates on a node will expire sometime between 455 and 485 days after the node's creation.
    The expiration times vary randomly so that nodes are likely to have their certs expire at different times than other nodes.

  • New command for deleting a single instance: kops delete instance

Breaking changes

  • Support for Kubernetes 1.9 and 1.10 has been removed.

  • Support for the Romana networking provider has been removed.

  • Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kops 1.20 by setting the LegacyIAM feature flag.

Required Actions

Deprecations

  • Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kops 1.20.

Full change list since 1.18.0 release

1.19.0-alpha.3 to 1.19.0-alpha.4

Assets 30
Sep 9, 2020
Release 1.17.2

@justinsb justinsb released this Sep 9, 2020

This version contains a critical update to etcd-manager: 1 year after creation (or first adopting etcd-manager), clusters will stop responding due to expiration of a TLS certificate. Upgrading kops to 1.18.1 (or the latest versions of the 1.15, 1.16, 1.17 or 1.18 series) and running kops update followed by a kops rolling-update will fix the issue. Please see the advisory for the full details.


kops 1.18.1 is the next patch release in the 1.18 series of kops, offering support for kubernetes 1.18.

Please see the release notes for the full list of changes.

Release notes for kops 1.18 series

Significant changes

  • The default image has been updated to Ubuntu 20.04 (Focal). Consequently, the SSH user changed to ubuntu and the Linux kernel changed to version 5.4.

  • To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Support for RHEL 8 and CentOS 8 has been added.

  • Support for Amazon Linux 2 has been improved and will work with the default Docker version.

  • containerd has been added and can be selected as an alternate container runtime for Kubernetes. Enable by using the --container-runtime containerd flag when creating a cluster or by setting spec.containerRuntime: containerd.

  • Rolling updates now support surging and parallelism within an instance group. For details see the documentation.

  • Cilium CNI can now use AWS networking natively through the AWS ENI IPAM mode. Kops can also run a Kubernetes cluster entirely without kube-proxy using Cilium's BPF NodePort implementation.

  • Cilium CNI can now use a dedicated etcd cluster managed by etcd-manager for synchronizing agent state instead of CRDs.

  • The Terraform target now supports Terraform 0.12 syntax (HCL2) by default. See the Required Actions item below.

  • New clusters in GCE are configured to run the metadata-proxy by default. The proxy runs as a DaemonSet and lands on nodes with the nodeLabel cloud.google.com/metadata-proxy-ready: "true". If you want to enable metadata-proxy on an existing cluster/instance group, add that nodeLabel to your instancegroup specs (kops edit ig ...) and run kops update cluster. When the changes are applied, the proxy will roll out to those targeted nodes.

  • GCE has a new flag: --gce-service-account. This takes the email of an existing GCP service account and launches the instances with it. This setting applies to the whole cluster (ie: it is not currently designed to support Instance Groups with different service accounts). If you do not specify a service account during cluster creation, the default compute service account will be used which matches the prior behavior.

  • Google API client libraries updated from v0.beta to v1.

  • Support for NodeLocalDNS cache.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for Terraform 0.12 support. See Required Actions below.

  • Support for the CoreOS OS distribution has been removed. Users should consider Flatcar as a replacement.

  • Support for the Debian 8 (Jessie) OS distribution has been removed.

  • The Docker health-check service is now disabled by default. It shouldn't be needed anymore, but it can still be enabled by setting spec.docker.healthCheck: true. It is recommended to also check node-problem-detector and draino as replacements. See Required Actions below.

  • Lyft CNI plugin default subnet tags changed from from Type: pod to KubernetesCluster: myclustername.mydns.io. Subnets intended for use by the plugin will need to be tagged with this new tag and additional tag filters may need to be added to the cluster spec in order to achieve the desired set of subnets.

  • Support for basic authentication has been disabled by default for Kubernetes 1.18 and will be removed in Kubernetes 1.19.

  • Support for static tokens has been disabled by default for Kubernetes 1.18 and later. To re-enable, see the Security Notes for Kubernetes. We intend to remove support entirely in a future kops version, so file an issue with your use case if you need this feature.

  • Support for Kubernetes versions prior to 1.9 has been removed.

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. See Required Actions below.

  • Support for the "Legacy" etcd provider has been removed for Kubernetes versions 1.18 and higher. Such clusters will need to migrate to the default "Manager" etcd provider. To migrate, see the etcd migration documentation.

  • A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

  • The kops.k8s.io/v1alpha1 API has been removed. Users of kops replace will need to supply v1alpha2 resources.

  • Please see the notes in the 1.15 release about the apiGroup changing from kops to kops.k8s.io

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to support Terraform 0.12.
    Terraform 0.12 no longer supports resource names starting with digits. In Kops, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.

    • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
    • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

    To prevent downtime, follow these steps with the new version of Kops:

    KOPS_FEATURE_FLAGS=-Terraform-0.12 kops update cluster --target terraform ...
    # Use Terraform <0.12
    terraform plan
    # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
    # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
    terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
    terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
    terraform plan
    # Ensure these resources are no longer being destroyed and recreated
    terraform apply
    

    Kops will now output Terraform 0.12 syntax with the normal workflow:

    kops update cluster --target terraform ...
    # Use Terraform 0.12. This plan should be a no-op
    terraform plan
    
  • Users that need the Docker health-check service will need to explicitly enable it:

  kops edit cluster
  # Add the following section
  spec:
    docker:
      healthCheck: true
  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of Kops.

    To enable the Pod priority feature, follow these steps:

    kops edit cluster
    # Add the following section
    spec:
      kubelet:
        featureGates:
          PodPriority: "true"
    
  • If a custom Kops build was used on a cluster, a kops-controller Deployment may have been created that should get deleted.
    Run kubectl -n kube-system delete deployment kops-controller after upgrading to Kops 1.16.0-beta.1 or later.

Deprecations

  • Support for Kubernetes versions 1.9 and 1.10 are deprecated and will be removed in kops 1.19.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of Kops.

  • Support for the Romana networking provider is deprecated and will be removed in kops 1.19.

  • Support for legacy IAM permissions is deprecated and will be removed in kops 1.19.


All changes from 1.18.0 to 1.18.1

Please see the release notes for the full list of changes.

Assets 17

@justinsb justinsb released this Sep 9, 2020

This version contains a critical update to etcd-manager: 1 year after creation (or first adopting etcd-manager), clusters will stop responding due to expiration of a TLS certificate. Upgrading kops to 1.17.2 (or the latest versions of the 1.16, 1.17 or 1.18 series) and running kops update followed by a kops rolling-update will fix the issue. Please see the advisory for the full details.


kops 1.17.2 is the next patch release in the kops 1.17 series, supporting kubernetes version 1.17.x and earlier.

Please see the release notes for the full list of changes.

Significant changes

  • The default Docker version has been changed to 19.03.4. Optional support for Docker 19.03.8 has been added and will be the default in future versions. Enable by setting spec.docker.version: 19.03.8.

  • The default instance type for AWS has been changed to t3.medium. This should provide better performance and reduced costs in clusters where the average CPU usage is low.

  • Support for Ubuntu 20.04 (Focal) has been added.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for future Terraform 0.12 support. See Required Actions below.

  • Please see the notes in the 1.15 release about the apiGroup changing from kops
    to kops.k8s.io

  • Since 1.16, a controller is now used to apply labels to nodes. If
    you are not using AWS, GCE or OpenStack your (non-master) nodes may
    not have labels applied correctly.

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to prepare for future Terraform 0.12 support.
    Terraform 0.12 no longer supports resource names starting with digits. In Kops, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.

    • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
    • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

    To prevent downtime, follow these steps with the new version of Kops:

    kops update cluster --target terraform ...
    terraform plan
    # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
    # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
    terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
    terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
    terraform plan
    # Ensure these resources are no longer being destroyed and recreated
    terraform apply
    
  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of Kops.

    To enable the Pod priority feature, follow these steps:

    kops edit cluster
    # Add the following section
    spec:
      kubelet:
        featureGates:
          PodPriority: "true"
    
  • If either a Kops 1.17 alpha release or a custom Kops build was used on a cluster,
    a kops-controller Deployment may have been created that should get deleted because it has been replaced with a DaemonSet.
    Run kubectl -n kube-system delete deployment kops-controller after upgrading to Kops 1.17.0-alpha.2 or later.

Deprecations

  • Support for Kubernetes releases prior to 1.9 is deprecated and will be removed in kops 1.18.

  • The kops/v1alpha1 API is deprecated and will be removed in kops 1.18. Users of kops replace will need to supply v1alpha2 resources.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of Kops.

  • Support for Debian 8 (Jessie) has been deprecated and will be removed in future versions of Kops.

  • Support for CoreOS has been deprecated and will be removed in future versions of Kops. Those affected should consider using Flatcar as a replacement.

  • Support for the "Legacy" etcd provider has been deprecated. It will not be supported for Kubernetes 1.18 or later. To migrate to the default "Manager" etcd provider see the etcd migration documentation.

Known Issues

  • None at the present time

Changes from 1.17.1 to 1.17.2

Please see the release notes for the full list of changes.

Assets 20
Pre-release
Pre-release

@hakman hakman released this Sep 7, 2020

Release notes for kops 1.19 series

(The kops 1.19 release has not been released yet; this is a document to gather the notes prior to the release).

Significant changes

Changes to kubernetes config export

Kops will no longer automatically export the kubernetes config on kops update cluster. In order to export the config on cluster update, you need to either add the --user <user> to reference an existing user, or --admin to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.

Similarly, kops export kubecfg will also require passing either the --admin or --user flag if the context does not already exist.

By default, the credentials of any exported admin user now have a lifetime of 18 hours. The lifetime of the exported
credentials may be specified as a value of the --admin flag. To get the previous behavior, specify --admin=87600h to either kops update cluster or kops export kubecfg.

kops create cluster --yes exports the admin user along with rest of the cluster config, as was the previous behaviour (except for the 18-hour validity).

OpenStack Cinder plugin

Kops will install the Cinder plugin for kops running kubernetes 1.16 or newer. If you already have this plugin installed you should remove it before upgrading.

If you already have a default StorageClass, you should set cloudConfig.Openstack.BlockStorage.CreateStorageClass: false to prevent kops from installing one.

Other significant changes

  • New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.

  • On AWS kops now defaults to using launch templates instead of launch configurations.

  • Clusters using the Amazon VPC CNI provider now perform an ec2.DescribeInstanceTypes call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue.

  • There is now Alpha support for Hashicorp Vault as a store for secrets and keys. See the Vault state store docs.

  • New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.

  • The kops update cluster command will now refuse to run on a cluster that
    has been updated by a newer version of kops unless it is given the --allow-kops-downgrade flag.

  • The lifetimes of certificates used by various components have been substantially reduced.
    The certificates on a node will expire sometime between 455 and 485 days after the node's creation.
    The expiration times vary randomly so that nodes are likely to have their certs expire at different times than other nodes.

Breaking changes

  • Support for Kubernetes 1.9 and 1.10 has been removed.

  • Support for the Romana networking provider has been removed.

  • Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kops 1.20 by setting the LegacyIAM feature flag.

Required Actions

Deprecations

  • Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kops 1.20.

1.19.0-alpha.2 to 1.19.0-alpha.3

Assets 30

@justinsb justinsb released this Aug 2, 2020

This version contains a critical update to etcd-manager: 1 year after creation (or first adopting etcd-manager), clusters will stop responding due to expiration of a TLS certificate. Upgrading kops to 1.18.0 (or the latest versions of the 1.15, 1.16, 1.17 or 1.18 series) and running kops update followed by a kops rolling-update will fix the issue. Please see the advisory for the full details.


kops 1.18.0 is the first stable release in the 1.18 series of kops, offering support for kubernetes 1.18.

Please see the release notes for the full list of changes.

Release notes for kops 1.18 series

Significant changes

  • The default image has been updated to Ubuntu 20.04 (Focal). Consequently, the SSH user changed to ubuntu and the Linux kernel changed to version 5.4.

  • To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Support for RHEL 8 and CentOS 8 has been added.

  • Support for Amazon Linux 2 has been improved and will work with the default Docker version.

  • containerd has been added and can be selected as an alternate container runtime for Kubernetes. Enable by using the --container-runtime containerd flag when creating a cluster or by setting spec.containerRuntime: containerd.

  • Rolling updates now support surging and parallelism within an instance group. For details see the documentation.

  • Cilium CNI can now use AWS networking natively through the AWS ENI IPAM mode. Kops can also run a Kubernetes cluster entirely without kube-proxy using Cilium's BPF NodePort implementation.

  • Cilium CNI can now use a dedicated etcd cluster managed by etcd-manager for synchronizing agent state instead of CRDs.

  • The Terraform target now supports Terraform 0.12 syntax (HCL2) by default. See the Required Actions item below.

  • New clusters in GCE are configured to run the metadata-proxy by default. The proxy runs as a DaemonSet and lands on nodes with the nodeLabel cloud.google.com/metadata-proxy-ready: "true". If you want to enable metadata-proxy on an existing cluster/instance group, add that nodeLabel to your instancegroup specs (kops edit ig ...) and run kops update cluster. When the changes are applied, the proxy will roll out to those targeted nodes.

  • GCE has a new flag: --gce-service-account. This takes the email of an existing GCP service account and launches the instances with it. This setting applies to the whole cluster (ie: it is not currently designed to support Instance Groups with different service accounts). If you do not specify a service account during cluster creation, the default compute service account will be used which matches the prior behavior.

  • Google API client libraries updated from v0.beta to v1.

  • Support for NodeLocalDNS cache.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for Terraform 0.12 support. See Required Actions below.

  • Support for the CoreOS OS distribution has been removed. Users should consider Flatcar as a replacement.

  • Support for the Debian 8 (Jessie) OS distribution has been removed.

  • The Docker health-check service is now disabled by default. It shouldn't be needed anymore, but it can still be enabled by setting spec.docker.healthCheck: true. It is recommended to also check node-problem-detector and draino as replacements. See Required Actions below.

  • Lyft CNI plugin default subnet tags changed from from Type: pod to KubernetesCluster: myclustername.mydns.io. Subnets intended for use by the plugin will need to be tagged with this new tag and additional tag filters may need to be added to the cluster spec in order to achieve the desired set of subnets.

  • Support for basic authentication has been disabled by default for Kubernetes 1.18 and will be removed in Kubernetes 1.19.

  • Support for static tokens has been disabled by default for Kubernetes 1.18 and later. To re-enable, see the Security Notes for Kubernetes. We intend to remove support entirely in a future kops version, so file an issue with your use case if you need this feature.

  • Support for Kubernetes versions prior to 1.9 has been removed.

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. See Required Actions below.

  • Support for the "Legacy" etcd provider has been removed for Kubernetes versions 1.18 and higher. Such clusters will need to migrate to the default "Manager" etcd provider. To migrate, see the etcd migration documentation.

  • A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

  • The kops.k8s.io/v1alpha1 API has been removed. Users of kops replace will need to supply v1alpha2 resources.

  • Please see the notes in the 1.15 release about the apiGroup changing from kops to kops.k8s.io

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to support Terraform 0.12.
    Terraform 0.12 no longer supports resource names starting with digits. In Kops, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.

    • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
    • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

    To prevent downtime, follow these steps with the new version of Kops:

    KOPS_FEATURE_FLAGS=-Terraform-0.12 kops update cluster --target terraform ...
    # Use Terraform <0.12
    terraform plan
    # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
    # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
    terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
    terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
    terraform plan
    # Ensure these resources are no longer being destroyed and recreated
    terraform apply
    

    Kops will now output Terraform 0.12 syntax with the normal workflow:

    kops update cluster --target terraform ...
    # Use Terraform 0.12. This plan should be a no-op
    terraform plan
    
  • Users that need the Docker health-check service will need to explicitly enable it:

  kops edit cluster
  # Add the following section
  spec:
    docker:
      healthCheck: true
  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of Kops.

    To enable the Pod priority feature, follow these steps:

    kops edit cluster
    # Add the following section
    spec:
      kubelet:
        featureGates:
          PodPriority: "true"
    
  • If a custom Kops build was used on a cluster, a kops-controller Deployment may have been created that should get deleted.
    Run kubectl -n kube-system delete deployment kops-controller after upgrading to Kops 1.16.0-beta.1 or later.

Deprecations

  • Support for Kubernetes versions 1.9 and 1.10 are deprecated and will be removed in kops 1.19.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of Kops.

  • Support for the Romana networking provider is deprecated and will be removed in kops 1.19.

  • Support for legacy IAM permissions is deprecated and will be removed in kops 1.19.


All changes from v1.18.0-beta.2 to v1.18.0

Please see the release notes for the full list of changes.

Assets 17
Pre-release
Pre-release

@justinsb justinsb released this Jul 31, 2020

(The kops 1.19 series has not been released yet; this is a pre-release).

kops 1.19.0-alpha.2 is the next alpha in the 1.19 series for kops.

Please see the release notes for the full list of changes.

Significant changes

Changes to kubernetes config export

Kops will no longer automatically export the kubernetes config on kops update cluster. In order to export the config on cluster update, you need to either add the --user <user> to reference an existing user, or --admin to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.

Similarly, kops export kubecfg will also require passing either the --admin or --user flag if the context does not already exist.

kops create cluster --yes exports the admin user along with rest of the cluster config, as is existing behaviour.

Other significant changes

  • New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.

  • On AWS kops now defaults to using launch templates instead of launch configurations.

  • Clusters using the Amazon VPC CNI provider now perform an ec2.DescribeInstanceTypes call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue.

  • Alpha support for Hashicorp Vault as store for secrets and keys. See the Vault state store docs.

  • New clusters running Cilium will have enabled BPF NodePort by default if kubernetes version is 1.12 or newer.

  • The kops update cluster command will now refuse to run on a cluster that
    has been updated by a newer version of kops unless it is given the --allow-kops-downgrade flag.

Breaking changes

  • Support for Kubernetes 1.9 and 1.10 has been removed.

  • Support for the Romana networking provider has been removed.

  • Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kops 1.20 by setting the LegacyIAM feature flag.

Required Actions

Deprecations

  • Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kops 1.20.

Change list

Changes from 1.19.0-alpha.1 to 1.19.0-alpha.2

Please see the release notes for the full list of changes.

Assets 16
Pre-release
Pre-release

@justinsb justinsb released this Jul 13, 2020

This version contains a critical update to etcd-manager: 1 year after creation (or first adopting etcd-manager), clusters will stop responding due to expiration of a TLS certificate. Upgrading kops to 1.18.0-beta.1 (or the latest versions of the 1.15, 1.16, 1.17 or 1.18 series) and running kops update followed by a kops rolling-update will fix the issue. Please see the advisory for the full details.


kops 1.18.0 beta.2 is the next beta in the 1.18 series for kops. We intend for the next release in the 1.18 series to be 1.18.0, so this can be treated as a release candidate for 1.18.0 (1.18.0-rc.1).

Please see the release notes for the full list of changes.

  • The default image has been updated to Ubuntu 20.04 (Focal). Consequently, the SSH user changed to ubuntu and the Linux kernel changed to version 5.4.

  • To address the issue of IPv4 only clusters being susceptible to MitM attacks via IPv6 rogue router advertisements, the affected components have been upgraded as follows:

  • Support for RHEL 8 and CentOS 8 has been added.

  • Support for Amazon Linux 2 has been improved and will work with the default Docker version.

  • containerd has been added and can be selected as an alternate container runtime for Kubernetes. Enable by using the --container-runtime containerd flag when creating a cluster or by setting spec.containerRuntime: containerd.

  • Rolling updates now support surging and parallelism within an instance group. For details see the documentation.

  • Cilium CNI can now use AWS networking natively through the AWS ENI IPAM mode. Kops can also run a Kubernetes cluster entirely without kube-proxy using Cilium's BPF NodePort implementation.

  • Cilium CNI can now use a dedicated etcd cluster managed by etcd-manager for synchronizing agent state instead of CRDs.

  • The Terraform target now supports Terraform 0.12 syntax (HCL2) by default. See the Required Actions item below.

  • New clusters in GCE are configured to run the metadata-proxy by default. The proxy runs as a DaemonSet and lands on nodes with the nodeLabel cloud.google.com/metadata-proxy-ready: "true". If you want to enable metadata-proxy on an existing cluster/instance group, add that nodeLabel to your instancegroup specs (kops edit ig ...) and run kops update cluster. When the changes are applied, the proxy will roll out to those targeted nodes.

  • GCE has a new flag: --gce-service-account. This takes the email of an existing GCP service account and launches the instances with it. This setting applies to the whole cluster (ie: it is not currently designed to support Instance Groups with different service accounts). If you do not specify a service account during cluster creation, the default compute service account will be used which matches the prior behavior.

  • Google API client libraries updated from v0.beta to v1.

  • Support for NodeLocalDNS cache.

Breaking changes

  • Support for Docker versions 1.11, 1.12 and 1.13 has been removed because of the dockerproject.org shut down. Those affected must upgrade to a newer Docker version.

  • Terraform users on AWS may need to rename some resources in their state file in order to prepare for Terraform 0.12 support. See Required Actions below.

  • Support for the CoreOS OS distribution has been removed. Users should consider Flatcar as a replacement.

  • Support for the Debian 8 (Jessie) OS distribution has been removed.

  • The Docker health-check service is now disabled by default. It shouldn't be needed anymore, but it can still be enabled by setting spec.docker.healthCheck: true. It is recommended to also check node-problem-detector and draino as replacements. See Required Actions below.

  • Lyft CNI plugin default subnet tags changed from from Type: pod to KubernetesCluster: myclustername.mydns.io. Subnets intended for use by the plugin will need to be tagged with this new tag and additional tag filters may need to be added to the cluster spec in order to achieve the desired set of subnets.

  • Support for basic authentication has been disabled by default for Kubernetes 1.18 and will be removed in Kubernetes 1.19.

  • Support for static tokens has been disabled by default for Kubernetes 1.18 and later. To re-enable, see the Security Notes for Kubernetes. We intend to remove support entirely in a future kops version, so file an issue with your use case if you need this feature.

  • Support for Kubernetes versions prior to 1.9 has been removed.

  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. See Required Actions below.

  • Support for the "Legacy" etcd provider has been removed for Kubernetes versions 1.18 and higher. Such clusters will need to migrate to the default "Manager" etcd provider. To migrate, see the etcd migration documentation.

  • A controller is now used to apply labels to nodes. If you are not using AWS, GCE or OpenStack your (non-master) nodes may not have labels applied correctly.

  • The kops.k8s.io/v1alpha1 API has been removed. Users of kops replace will need to supply v1alpha2 resources.

  • Please see the notes in the 1.15 release about the apiGroup changing from kops to kops.k8s.io

Required Actions

  • Terraform users on AWS may need to rename resources in their terraform state file in order to support Terraform 0.12.
    Terraform 0.12 no longer supports resource names starting with digits. In Kops, both the default route and additional VPC CIDR associations are affected. See #7957 for more information.

    • The default route was named aws_route.0-0-0-0--0 and will now be named aws_route.route-0-0-0-0--0.
    • Additional CIDR blocks associated with a VPC were similarly named the hyphenated CIDR block with two hyphens for the /, for example aws_vpc_ipv4_cidr_block_association.10-1-0-0--16. These will now be prefixed with cidr-, for example aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16.

    To prevent downtime, follow these steps with the new version of Kops:

    KOPS_FEATURE_FLAG=-Terraform-0.12 kops update cluster --target terraform ...
    # Use Terraform <0.12
    terraform plan
    # Observe any aws_route or aws_vpc_ipv4_cidr_block_association resources being destroyed and recreated
    # Run these commands as necessary. The exact names may differ; use what is outputted by terraform plan
    terraform state mv aws_route.0-0-0-0--0 aws_route.route-0-0-0-0--0
    terraform state mv aws_vpc_ipv4_cidr_block_association.10-1-0-0--16 aws_vpc_ipv4_cidr_block_association.cidr-10-1-0-0--16
    terraform plan
    # Ensure these resources are no longer being destroyed and recreated
    terraform apply
    

    Kops will now output Terraform 0.12 syntax with the normal workflow:

    kops update cluster --target terraform ...
    # Use Terraform 0.12. This plan should be a no-op
    terraform plan
    
  • Users that need the Docker health-check service will need to explicitly enable it:

  kops edit cluster
  # Add the following section
  spec:
    docker:
      healthCheck: true
  • Kubernetes 1.9 users will need to enable the PodPriority feature gate. This is required for newer versions of Kops.

    To enable the Pod priority feature, follow these steps:

    kops edit cluster
    # Add the following section
    spec:
      kubelet:
        featureGates:
          PodPriority: "true"
    
  • If a custom Kops build was used on a cluster, a kops-controller Deployment may have been created that should get deleted.
    Run kubectl -n kube-system delete deployment kops-controller after upgrading to Kops 1.16.0-beta.1 or later.

Deprecations

  • Support for Kubernetes versions 1.9 and 1.10 are deprecated and will be removed in kops 1.19.

  • Support for Ubuntu 16.04 (Xenial) has been deprecated and will be removed in future versions of Kops.


All changes from v1.18.0-beta.1 to v1.18.0-beta.2

Please see the release notes for the full list of changes.

Assets 17
You can’t perform that action at this time.