v1.19.0-beta.3
Pre-releaseRelease notes for kOps 1.19 series
kOps 1.19.0 beta.3 is the next beta in the 1.19 series for kOps. We intend for the next release in the 1.19 series to be 1.19.0, so this can be treated as a release candidate for 1.19.0 (1.19.0-rc.1).
Significant changes
Changes to kubernetes config export
kOps will no longer automatically export the kubernetes config on kops update cluster
. In order to export the config on cluster update, you need to either add the --user <user>
to reference an existing user, or --admin
to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.
Similarly, kops export kubecfg
will also require passing either the --admin
or --user
flag if the context does not already exist.
By default, the credentials of any exported admin user now have a lifetime of 18 hours. The lifetime of the exported
credentials may be specified as a value of the --admin
flag. To get the previous behavior, specify --admin=87600h
to either kops update cluster
or kops export kubecfg
.
kops create cluster --yes
exports the admin user along with rest of the cluster config, as was the previous behaviour (except for the 18-hour validity).
ARM64 support
kOps will install ARM64 artifacts and containerd images when the instance group supports ARM64 for both machine type and OS image. At the moment this is known to work with AWS m6g
, c6g
, r6g
and t4g
instances, with the latest Ubuntu 20.04 OS images for ARM64.
OpenStack Cinder plugin
kOps will install the Cinder plugin for kOps running kubernetes 1.16 or newer. If you already have this plugin installed you should remove it before upgrading.
If you already have a default StorageClass
, you should set cloudConfig.Openstack.BlockStorage.CreateStorageClass: false
to prevent kOps from installing one.
Other significant changes by kind
General
-
New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.
-
On AWS kOps now defaults to using launch templates instead of launch configurations.
-
There is now Alpha support for Hashicorp Vault as a store for secrets and keys. See the Vault state store docs.
-
The lifetimes of certificates used by various components have been substantially reduced.
The certificates on a node will expire sometime between 455 and 485 days after the node's creation.
The expiration times vary randomly so that nodes are likely to have their certs expire at different times than other nodes. -
kOps now supports using an AWS Network Load Balancer (NLB) for API access.
See the documentation for more info. -
Allow users to partially compress user-data, check the instance groups docs for more details.
CLI
-
The
kops update cluster
command will now refuse to run on a cluster that
has been updated by a newer version of kOps unless it is given the--allow-kops-downgrade
flag. -
New command for deleting a single instance: kops delete instance
CNI
-
Clusters using the Amazon VPC CNI provider now perform an
ec2.DescribeInstanceTypes
call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue. -
Clusters using Calico with
CrossSubnet
enabled will switch to the new awsSrcDstCheck for disabling the AWS source/destination checks. The previous implementation using k8s-ec2-srcdst is now deprecated. -
Clusters using Calico can now enable the eBPF dataplane mode for Ubuntu 20.04 (Focal) hosts. Add
spec.networking.calico.bpfEnabled: true
andspec.kubeProxy.enabled: false
to the cluster spec to enable. -
Clusters using Calico can now encrypt pod-to-pod traffic with WireGuard for Ubuntu hosts. Add
spec.networking.calico.wireguardEnabled: true
to the cluster spec to enable. -
New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.
Addons
-
Metrics Server is now available as a configurable addon. Add
spec.metricsServer.enabled: true
to the cluster spec to enable. -
Cluster Autoscaler is now availalble as a configurable addon. Add
spec.clusterAutoscaler.enabled: true
to the cluster spec to enable. -
AWS Node Termination Handler is now available as a configurable addon. Add
spec.nodeTerminationHandler.enabled: true
to the cluster spec to enable.
Breaking changes
-
Support for Kubernetes 1.9 and 1.10 has been removed.
-
Support for the Romana networking provider has been removed.
-
Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kOps 1.20 by setting the
LegacyIAM
feature flag.
Required Actions
-
See note about Openstack Cinder plugin above.
-
Terraform 0.12 users on AWS, in order to prevent downtime you will have to remove the state of any existing ELB or TargetGroup attachments from your Terraform state file. This is due to migrating the attachments to the in-line
aws_autoscaling_group
fields. See the terraform documentation for more information about the difference. This migration is required due to a bug described in #9913.To prevent downtime, follow these steps with the new version of Kops:
kops update cluster --target terraform ... terraform plan terraform state list | grep aws_autoscaling_attachment | xargs -L1 terraform state rm terraform plan # Ensure these resources are no longer being destroyed and recreated terraform apply
-
Terraform 0.12 users on AWS migrating clusters from Launch Configurations to Launch Templates may need to remove the state of the old Launch Configuration. This is due to potential errors with Terraform attempting to delete the Launch Configuration before updating the AutoScalingGroup to use the Launch Template. The Launch Configurations will need to be manually deleted afterwards.
More information including detailed remediation steps is available in #10017.kops update cluster --target terraform ... terraform state list | grep aws_launch_configuration | xargs -L1 terraform state rm terraform plan # Ensure launch configurations are not being destroyed terraform apply
-
If you are using Terraform with an additional .tf file and using "aws_autoscaling_attachment" to attach additional Load Balancers or ALB/NLB Target Groups you'll need to migrate to attaching them through the InstanceGroup spec instead.
-
AWS clusters using an ACM Certificate on the API ELB (
.spec.api.loadBalancer.sslCertificateID
) will need to migrate from Classic LoadBalancer (CLB) to Network LoadBalancer (NLB) prior to upgrading to Kubernetes 1.19 by setting.spec.api.loadBalancer.class: Network
.
Any kubeconfig files using kOps' admin client credentials will need to be regenerated withkops export kubecfg --admin
.
For more information see this page.
Deprecations
-
Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kOps 1.20.
-
Support for Terraform version 0.11 has been deprecated and will be removed in kOps 1.20.
-
Support for feature flag
Terraform-0.12
has been deprecated and will be removed in kOps 1.20. All generated Terraform HCL2/JSON files will support versions0.12.26+
and0.13.0+
. -
The manifest based metrics server addon has been deprecated in favour of a configurable addon.
-
The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.
Partial change list since 1.19.0-beta.2 release
1.19.0-beta.2 to 1.19.0-beta.3
- [weave] Add support for default version override @dntosas,@hakman #10273
- Automated cherrypick of #10275 onto release-1.19 @rdrgmnzs #10306
- Update Calico to v3.17.0 @hakman #10310
- Tolerate missing detached EC2 instances @hwoarang #10319
- Remove copyright notice from nodeup scripts to reduce the user-data size. @rdrgmnzs #10333
- Update containerd and Docker versions @hakman #10341
- Allow using gp3 for root volumes @olemarkus #10345
- Bump aws-vpc-cni version to 1.7.6 @MoShitrit #10337
- [Digital Ocean] Upgrade godo sdk to v1.54 @srikiz,@timoreimann #10320
- Update etcd-manager to 3.0.20201202 @justinsb #10351
- Add paramaeters related to Taint based Evictions in kube-apiserver @h3poteto #10339
- Remove support for using legacy ELB name @hakman #10296
- Give users the option to gzip and base64 encode the heredocs in the nodeup.sh user-data @rdrgmnzs #10357
- Remove resource limits from cluster autoscaler @olemarkus #10375
- Remove dependency on TravisCI @hakman #10366
- Add option to reuse existing Elastic IPs for NAT gateways @hakman #10374
- Upgrade docker client @olemarkus #10193
- Update aws-sdk-go to v1.36.0 @hakman #10347
- Add option for setting the volume encryption key in AWS @hakman #10359
- Add support for AWS IMDS v2 @bharath-123 #10324
- Update etcd-manager to 20201209 @justinsb #10394
- Explicitly specify http_endpoint in terraform launch template @bharath-123 #10398
- Update k8s dependencies to v1.19.5 @hakman #10385
- Mount /lib64 for Protokube only on AMD64 @hakman #10396
- Update cilium to 1.8.6 @olemarkus #10406
- Allow override of registry and tag for Calico images @hakman #10316
- Bump aws-cni to 1.7.7 @MoShitrit #10416
- Add support for containerd v1.4.3 ARM64 @hakman #10418
- Update container runtime service files @hakman #10428
- Expose metrics port when PrometheusMetricsEnabled set to true in Calico @avdhoot #10414
- Bump AWS-CNI to version 1.7.8 @MoShitrit #10447
- Allow Calico to run on systems with loose reverse path forwarding @hakman #10442
- Calico: Allow operators to choose which encapsulation mode to use @seh #10404
- protokube - query host by label when setting tags @rdrgmnzs #10413
- Drop support for containerd 1.2 @hakman #10483
- Added event-qps and event-burst flags to kubelet @DOboznyi #10486
- Add new-pod-scale-up-delay in Cluster Autoscaler spec @akshedu #10471
- Add config options for container runtime package URL and Hash @hakman #10473
- Release 1.19.0-beta.3 @hakman #10494