Provide Flatcar Container Linux CAPA images on AWS

[t-lo]

/kind feature

This issue is to track the status of a recurring CAPA office hours item - to provide CAPA images of Flatcar Container Linux on AWS (see e.g. here). Image availability will foster e2e testing aims to provide a point of reference for using Flatcar Container Linux with the CAPI AWS provider.

This issue aims to consolidate communication re: image building to a single point of reference.

Current status

The Flatcar maintainers team publishes CAPA images after each release to a separate AWS account ( @dongsupark drives this initiative and can provide more detail ) . Availability of Flatcar CAPA for e2e testing thus currently depends on a 3rd party team.

Future / ideal state

Flatcar Container Linux CAPA images are published via the CAPA account (managed by VMWare). Images follow Flatcar Stable releases and are updated regularly.

Additional information

Flatcar Container Linux AWS images can be built via image-builder; see README-flatcar.md for details.

Builds should use the "stable" channel; the Flatcar release version to build is auto-detected and will default to the latest release.

Environment:

• Cluster-api-provider-aws version: upcoming (Flatcar support will only become available with CAPI v1.1-beta1)
• Kubernetes version: (use kubectl version): Various / TBD.
• OS (e.g. from /etc/os-release): (Note that the below has been edited so it's release version agnostic)

  NAME="Flatcar Container Linux by Kinvolk"
  ID=flatcar
  ID_LIKE=coreos
  VERSION=[VERSION]
  VERSION_ID=[VERSION]
  PRETTY_NAME="Flatcar Container Linux by Kinvolk [VERSION] (Oklo)"
  ANSI_COLOR="38;5;75"
  HOME_URL="https://flatcar-linux.org/"
  BUG_REPORT_URL="https://issues.flatcar-linux.org"
  FLATCAR_BOARD="amd64-usr"
  CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:[VERSION]:*:*:*:*:*:*:*"
  

[k8s-ci-robot]

@t-lo: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[t-lo]

CC: @sedefsavas

[dongsupark]

Thanks for creating the issue.
I will try to add more details.

Each Flatcar CAPA AMI name has a format of capa-ami-flatcar-[channel]-[k8s-version]-00-[timestamp], e.g. capa-ami-flatcar-stable-1.23.1-00-1639733395.

So far AMIs were created for all Flatcar channels, i.e. stable, beta, and alpha.
AMIs for each channel have been created for 3 recent Kubernetes versions, e.g. 1.21, 1.22, and 1.23.
As a result, 9 AMIs in total.
Every time when Flatcar releases happen, like yesterday Jan 26th, Flatcar team runs image-builder for Flatcar 9 times.

Though I am not sure if anyone is relying on Beta and Alpha AMIs. For now publishing only Stable channel should be also fine.

So far the result AMIs have been created with the default Flatcar infra account of AWS, 075585003325.
If we are going to do that with a different account, then we need to update the line above.

Image-builder by default builds only for k8s 1.20. So we need to manually update those as well.

[sedefsavas]

Published Kubernetes 1.23.3 Flatcar-stable images:

➜  cluster-api-provider-aws git:(flatcar) ✗ clusterawsadm ami list --kubernetes-version=1.23.3 --os=flatcar-stable

KUBERNETES VERSION   REGION           OS               NAME                                           AMI ID
1.23.3               ap-northeast-1   flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0230a91d677b1c3c7
1.23.3               ap-northeast-2   flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0eb262c6143a0e58b
1.23.3               ap-south-1       flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0d348faa340d389cd
1.23.3               ap-southeast-1   flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-05a5ad0567dd082c5
1.23.3               ap-southeast-2   flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0eab38252f3903fd8
1.23.3               ca-central-1     flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0319ec41e34dd0b64
1.23.3               eu-central-1     flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-035306ef732f51213
1.23.3               eu-west-1        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-01f47ef292804e13b
1.23.3               eu-west-2        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-03d36345126442b94
1.23.3               eu-west-3        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0c50cec7651f942ef
1.23.3               sa-east-1        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-059fb6b9147ab6341
1.23.3               us-east-1        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0844aa998d4d28ead
1.23.3               us-east-2        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-0719aa823b65cf48f
1.23.3               us-west-1        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-00a767cfcd4ef2665
1.23.3               us-west-2        flatcar-stable   capa-ami-flatcar-stable-1.23.3-00-1644206816   ami-08544763d05196023

Given that these are reference images, publishing and testing stable releases would be more sustainable IMO compared to publishing also alpha, beta releases which probably will not be used by end users. We can discuss this further during the office hours.

[sedefsavas]

There were few things I needed to change in the config to build them. Added these to the config:

"containerd_version": "1.5.9",
"containerd_sha256": "f64c8e3b736b370c963b08c33ac70f030fc311bc48fcfd00461465af2fff3488",
"kubernetes_source_type": "http"
"kubernetes_cni_source_type": "http"

Could not make it work without setting the checksum.

The complete config I used to build the images is here: https://gist.github.com/sedefsavas/9e145e8ee0cf81cfdbb439edc8663e66

[invidian]

I'm trying images in e2e tests right now with #2271. I've noticed that only K8s v1.23.3 images are published, while CI use 1.23.0 right now. However, just updating everything to 1.23.3 doesn't seem to make it work, so I'm investigating further.

Should we update e2e tests to use v1.23.3 or could we build Flatcar for 1.23.0 as well?

[invidian]

so I'm investigating further.

So, it seems new images are missing e.g. kubernetes-sigs/image-builder@f4864de patch (and likely others), which we rely on to simplify the user configuration for CAPA. This is why e2e tests are failing right now.

@sedefsavas do you know what image-builder version was used to build new AMIs? Can we update/rebase it somehow?

[sedefsavas]

Not sure why it fails, I see kindest node for v1.23.3 is published.

But about to publish 1.23.0 images.

[invidian]

But about to publish 1.23.0 images.

Thanks, but if they do not include kubernetes-sigs/image-builder@f4864de, they won't work anyway :/

[sedefsavas]

I am using 0.10.0 image builder.
Will republish 1.23.3 images with v0.1.11 image builder.
It does have the change, right?

We only build using released image-builder versions normally.

[invidian]

Hmm, kubernetes-sigs/image-builder@f4864de is only included in master and [v0.1.11](https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.11), according to GitHub 🤔

[sedefsavas]

ACK, will republish using v0.1.11 image builder.

[sedefsavas]

New v1.23.3 images published with image-builder v0.1.11:

--> flatcar-stable: AMIs were created:
ap-northeast-1: ami-0644b9a8c22af19f0
ap-northeast-2: ami-081a93f256696060e
ap-south-1: ami-078caddc6c8d8b3e8
ap-southeast-1: ami-01a56a3c987b4f56a
ap-southeast-2: ami-0e303cafee81a9466
ca-central-1: ami-07fbef0a507436f69
eu-central-1: ami-0c1408c802a43f808
eu-west-1: ami-0b0ff0d076a82b34f
eu-west-2: ami-031c52bb805306ae2
eu-west-3: ami-0cc078f8a4ad3dab9
sa-east-1: ami-06f990af996cc288a
us-east-1: ami-0f77d92147284a138
us-east-2: ami-05b686e544dd31374
us-west-1: ami-058b7bed1f97d479d
us-west-2: ami-02b65e82db7b5f8b9

[invidian]

New v1.23.3 images published with image-builder v0.1.11:

Awesome, with new AMIs e2e tests are passing again 🎉 With k8s v1.23.3, will test with 1.23.0. Thanks!