[RFE] Offer Flatcar (and CAPI Flatcar) images via Azure Community Gallery

[invidian]

Current situation

Azure has recently put into public preview Community Galleries support for Compute galleries, which allows sharing the images between subscriptions and tenants and use existing SIG images building process for publishing new images, which should be significantly easier than publishing to marketplace, which as far as I'm aware requires some manual steps at the moment and carries a delay in the release process (#675, #411).

The consumption of the image would also become easier, as it would no longer be need to provide Plan information in VM creation requests.

Perhaps Flatcar could offer images to users using Community Gallery for simpler consumption and to speed up, automate and simplify the release process for Azure in the future.

In addition, existing CAPI work (#445) for CAPZ (kubernetes-sigs/cluster-api-provider-azure#1729) is currently blocked on availability of some CAPI images, which could be consumed from the CI for the e2e tests. CAPZ upstream has already some reference images published in the Marketplace, but adding Flatcar there may not be the optimal option (some details in the CAPZ community call: https://www.youtube.com/watch?v=GjRCsdvvr0E), so Community Galleries sounds like a best alternative.

There has been some effort and tests performed for this, namely we're in the process of adding Community Galleries support to CAPZ to support it: kubernetes-sigs/cluster-api-provider-azure#2277

Ideal future situation

I would like CAPI Flatcar images to be available for consumption in some official Community Gallery.

Implementation options

After discussing this internally, the suggestion was to create 2 Community Galleries, one with plain Flatcar images and another one for CAPI images, as they will likely have different image definitions.

Having a gallery e.g. per release channel doesn't really make sense, as public gallery names have UUID appended to the prefix, which makes them not easy to swap. This applies to both regular and CAPI images.

Generally we should aim to avoid image definition explosion because of too many combinations.

Initial naming and publishing scheme to open the discussion:

• Flatcar Community Gallery:
  • PublicNamePrefix: Flatcar
  • Public gallery name (computed): Flatcar-52089376-cc7d-456e-8541-3c634dd271a2
  • Image definitions (in the "Community Images" section of "Select an image" portal page, I think it make sense to have Flatcar in image definition name, as it makes a first column in the table):
    • flatcar-stable
    • flatcar-stable-gen2
    • flatcar-beta
    • flatcar-alpha
    • flatcar-LTS
  • Versions of each image definition will then follow Flatcar release versions (so users can make use of latest version, deploying latest Flatcar release in a given channel)
• Flatcar CAPI Community Gallery:
  • PublicNamePrefix: Flatcar-CAPI
  • Public gallery name (computed): Flatcar-CAPI-b7420630-2f5a-4258-9fc9-c88da42bfe14
  • Image definitions:
    • flatcar-<Flatcar channel>-capi-<Kubernetes version>, e.g. flatcar-stable-capi-v1.22.11 (for V2 images)
    • flatcar-<Flatcar channel>-capi-<Kubernetes version>-gen1, e.g. flatcar-stable-capi-v1.22.11-gen1 (for V1)
  • Versions of each image definition will also follow Flatcar release versions, for consistency with the other gallery.
  • We may consider disabling latest for all versions of CAPI images to enforce on user selection of Flatcar version, as CAPI images also disable Flatcar updates, so perhaps updates there should be more explicit overall.
  • One more thing to consider is that existing name is capi-flatcar-stable-<Flatcar version>, so if we downcase, we can keep the scheme almost like it is, except swapping Flatcar version with Kubernetes version.

~~I'm also fine with down-casing all the names where - can be used, e.g. Flatcar-CAPI-stable-<Kubernetes version> -> flatcar-capi-stable-<Kubernetes version>. ~~ Names are case-insensitive, so it make sense downcasing.

As a reference, CAPA images are currently published with the names following scheme: capa-ami-flatcar-stable-1.23.5-00-1647494826, with no indicator of Flatcar version used. Perhaps we can use capz-flatcar-stable- format. We should also decide if mentioned K8s version should include v or not.

Additional information

As far as I understood, creation of the gallery and image definitions could be done as a pre-release step in https://github.com/flatcar-linux/mantle/tree/flatcar-master/cmd/plume.

We should also figure how to automate the process of building CAPI images as part of the release process. As far as I know Flatcar builds right now produce VHD images, so we should check if image-builder can build the SIG image from VHD images or if we can build SIG from other SIG images.

CC @t-lo @jepio @sayanchowdhury

[invidian]

As far as I know Flatcar builds right now produce VHD images, so we should check if image-builder can build the SIG image from VHD images or if we can build SIG from other SIG images.

According to https://www.packer.io/plugins/builders/azure/arm#azure-arm-builder-specific-options:

The Azure builder can create either a VHD or a managed image. If you are creating a VHD, you must start with a VHD. Likewise, if you want to create a managed image you must start with a managed image.

I've previously tested building VHD from VHD which works fine and I just tested building a SIG image from another SIG image, which also works without issues. This is not something supported by image-builder project at the moment though, I had to apply the following patch:

diff --git images/capi/packer/azure/packer.json images/capi/packer/azure/packer.json
index f60d85632..07f88fd08 100644
--- images/capi/packer/azure/packer.json
+++ images/capi/packer/azure/packer.json
@@ -46,11 +46,14 @@
       },
       "client_id": "{{user `client_id`}}",
       "client_secret": "{{user `client_secret`}}",
-      "image_offer": "{{user `image_offer` }}",
-      "image_publisher": "{{user `image_publisher` }}",
-      "image_sku": "{{user `image_sku`}}",
-      "image_version": "{{user `image_version`}}",
       "location": "{{user `azure_location`}}",
+      "shared_image_gallery": {
+        "subscription": "{{user `subscription_id`}}",
+        "resource_group":"{{user `resource_group_name`}}",
+        "gallery_name": "{{user `shared_image_gallery_name`}}",
+        "image_name": "{{user `image_name`}}",
+        "image_version": "0.3.1651499183"
+      },
       "managed_image_name": "{{user `image_name`}}-{{user `build_timestamp`}}",
       "managed_image_resource_group_name": "{{user `resource_group_name`}}",
       "name": "sig-{{user `build_name`}}",

So if there is a process to covert VHD image to SIG image in place for Flatcar already, perhaps that would be a simpler way of implementing the solution for this issue for the time being. I can also have a look into adding official support for this in image builder though.

[invidian]

While working on this, we should also decide on publisher, offer and SKU names for SIG image definitions. Currently image-builder use publisher capz, offer capz-demo and SKU flatcar-stable-2905.2.3.

Seems SKU can be as well e.g. Flatcar-CAPI-stable-v1.22.11, following image definition name.

If we would plan to continue building images from the marketplace offer, it would perhaps make sense to use existing publisher: kinvolk, offer: flatcar-container-linux-free and sku: stable to make it easier for users to discover what plan information they need to specify. However, this won't work as 2 image definitions cannot use the same combination of values above and we plan to have separate image definition for each Kubernetes version.

However, as the goal is to use Flatcar Community SIG images as a base for CAPI images, the publisher, offer and SKU becomes less relevant, as they won't be needed by user. I would still put some sensible values there.

EDIT: Excluding from latest all versions works as expected and it's possible via image-builder using small patch.

While finalizing the schema we also need to consider how to identify gen2 images.

[invidian]

This is how images look like using Azure CLI with proposed schema:

$ az sig image-definition list-community -l westeurope --public-gallery-name capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be -o table | grep -v capi-flatcar-stable
HyperVGeneration    Location    Name                               OsState      OsType    UniqueId
------------------  ----------  ---------------------------------  -----------  --------  ----------------------------------------------------------------------------------------------------------------
V1                  westeurope  Flatcar-CAPI-stable-v1.22.11       Generalized  Linux     /CommunityGalleries/capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be/Images/Flatcar-CAPI-stable-v1.22.11
V2                  westeurope  Flatcar-CAPI-stable-v1.22.11-gen2  Generalized  Linux     /CommunityGalleries/capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be/Images/Flatcar-CAPI-stable-v1.22.11-gen2
V1                  westeurope  Flatcar-CAPI-stable-v1.22.9        Generalized  Linux     /CommunityGalleries/capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be/Images/Flatcar-CAPI-stable-v1.22.9
$ az sig image-definition show-community -l westeurope --public-gallery-name capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be --gallery-image-definition Flatcar-CAPI-stable-v1.22.11 2>/dev/null
{
  "disallowed": null,
  "endOfLifeDate": null,
  "features": null,
  "hyperVGeneration": "V1",
  "identifier": {
    "offer": "capz-demo",
    "publisher": "capz",
    "sku": "Flatcar-CAPI-stable-v1.22.11"
  },
  "location": "westeurope",
  "name": "Flatcar-CAPI-stable-v1.22.11",
  "osState": "Generalized",
  "osType": "Linux",
  "purchasePlan": null,
  "recommended": {
    "memory": null,
    "vCpUs": null
  },
  "type": null,
  "uniqueId": "/CommunityGalleries/capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be/Images/Flatcar-CAPI-stable-v1.22.11"
}
$ az sig image-version list-community -l westeurope --public-gallery-name capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be --gallery-image-definition Flatcar-CAPI-stable-v1.22.11 2>/dev/null
[
  {
    "endOfLifeDate": null,
    "excludeFromLatest": true,
    "location": "westeurope",
    "name": "3033.2.4",
    "publishedDate": "2022-07-05T11:28:05.778645+00:00",
    "storageProfile": {
      "dataDiskImages": null,
      "osDiskImage": {
        "diskSizeGB": 30,
        "hostCaching": "ReadWrite",
        "sizeInGb": null
      }
    },
    "type": null,
    "uniqueId": "/CommunityGalleries/capiMatTestWE2-3282f15c-906a-4c4b-b206-eb3c51adb5be/Images/Flatcar-CAPI-stable-v1.22.11/Versions/3033.2.4"
  }
]

And using Azure Portal:

Here is the branch with a bunch of changes I needed to make this happen: https://github.com/kinvolk/image-builder/tree/invidian/flatcar-community-sig

And here is my extra packer.json variables file:

{
  "replication_regions": "westeurope,eastus,eastus2,northeurope,uksouth,westus2",
  "image_name": "Flatcar-CAPI-{{env `FLATCAR_CHANNEL`}}-{{user `kubernetes_semver`}}",
  "sig_image_version": "{{env `FLATCAR_VERSION`}}",
  "exclude_from_latest": "true"
}

[jepio]

I'm also fine with down-casing all the names where - can be used, e.g. Flatcar-CAPI-stable- -> flatcar-capi-stable-.

The names are case-insensitive, so lower-case makes sense.

If we would plan to continue building images from the marketplace offer, it would perhaps make sense to use existing publisher: kinvolk, offer: flatcar-container-linux-free and sku: stable to make it easier for users to discover what plan information they need to specify. However, this won't work as 2 image definitions cannot use the same combination of values above and we plan to have separate image definition for each Kubernetes version.

I saw two conflicting ideas for how to use publisher:offer:sku in galleries: either it's a hint to the user what information they need to enter as plan info when spawning instances from gallery image. Or it's a placeholder for what the values would be if one were to publish the image on the marketplace. Don't know how that helps us, but if they have to be unique (across galleries? or only within one?) then our hands are tied and it needs to be the second use.

RE: naming. How about grouping related information: flatcar-stable-capi-1.22.11-gen2. I agree on the need to repeat "flatcar" in the name for discoverability. Also keep in mind architecture (arm64).

[invidian]

Thanks for feedback @jepio, greatly appreciated.

The names are case-insensitive, so lower-case makes sense.

Make sense, adopted.

RE: naming. How about grouping related information: flatcar-stable-capi-1.22.11-gen2. I agree on the need to repeat "flatcar" in the name for discoverability. Also keep in mind architecture (arm64).

Yeah, I think that make sense. flatcar-stable-capi-v1.22.11-gen2 sounds good.

Do you have some details regarding arm64 images? How can you build those? I'm not even sure actually if image-builder supports arm64. If so, would the image name must be different as well, like flatcar-stable-capi-v1.22.11-arm64?

[invidian]

Notes from today's meeting:

• We may still want to consider -amd64 prefix for images so we would be consistent with -arm64 and possibly other architectures in the future.
• We should check if CAPZ supports using gen2 images. If so, we can consider only publishing gen2 images and dropping -gen2 prefix all together.
  It has also been mentioned that arm64 images only supports gen2, so dropping -gen2 prefix would avoid us combining -arm64 prefix with other prefixes.
  EDIT: it seems CAPZ supports gen2, here is the issue which is almost ready to close: Azure Generation 2 VM support kubernetes-sigs/cluster-api-provider-azure#1003
  • I've updated our CAPZ PR to use gen2 images to verify that it works.
  • So the initial test fails, but it might be just about the new Flatcar images in general, so reverted to gen1 now to see what happens.
  • So gen 2 images works just fine. I was just hitting the issue with sku for gen2 images, which is stable-gen2 instead of just stable.
• Check if image-builder can produce arm64 images.
  So it seems image-builder has some bits for parametrizing CPU architecture, but I don't think it has a full suport. I've created an issue to clarify and track it: Support building arm64 CAPI images kubernetes-sigs/image-builder#936
• Even for testing, we should create both Flatcar and Flatcar-CAPI galleries and try building CAPI images from Flatcar SIG images, so we produce plan-free CAPI images from the beginning, which should help with adoption.
  EDIT: I'm trying this right now with kubernetes-sigs/cluster-api-provider-azure@47d21e3.
  This seems to work just fine, cool.

[invidian]

RE: vanilla Flatcar SIG images. As we have Flatcar VHD available as a blob object, we can simply do az sig image-version create --os-vhd-storage-account --os-vhd-uri to add it to the SIG, which is very simple. Then we can use this for building the CAPI image, although image-builder right now only supports building from the marketplace image. We would have to extend it to allow it building from the private or community SIG.

[invidian]

Based on the recent checks (#734 (comment)), I propose the following changes to the proposed schema:

• Start with creating just V2 image definitions without -gen2 prefix. If the need arise for V1 images, we create them with -gen1 suffix. Hopefully V3 won't be introduced anytime soon.
• Create both Flatcar and Flatcar CAPI SIGs as community shared SIGs from the beginning and build CAPI images using Flatcar SIG to save users from the hassle of having to specify plan information.
• Image definitions will be named flatcar-<Flatcar channel>-<CPU arch> and flatcar-<Flatcar channel>-capi-<Kubernetes version>-<CPU arch>.
• Use publisher flatcar.
• Use offer: stable-capi for CAPI images.
• As SKU must be unique per image definition, we simply re-use image definition name.

I am still not sure about:

• The -amd64 suffix for both Flatcar and CAPI images. It would be nice to have it for consistency with possible -arm64 images, though they are currently not planned.
  If we do so, we will have flatcar-stable-amd64 and flatcar-stable-capi-v1.22.11-amd64 images.
  @jepio @pothos thoughts? I'm personally think it's a good idea.
• What offer do we use for regular Flatcar images? Maybe just stable?

If we are fine with the changes and we solve the above, I will:

• Create new Azure resource group for both Flatcar and Flatcar CAPI SIGs.
• Create both Flatcar and Flatcar CAPI SIGs as community shared SIGs.
• Manually create flatcar-stable image definition.
• Using existing VHD import initial Flatcar version into flatcar-stable.
• Use Flatcar SIG to build initial CAPI image for CAPZ consumption.

[pothos]

Using the explicit -amd64 and -arm64 suffixes is nice (assuming there has to be one). In the past we only had -arm64 suffixes and it's a mess in the code base and it's easy to overlook that both have to be handled.

Starting with the stable offer is good, but I expect lts and maybe beta if not alpha to also be available.

[invidian]

Using the explicit -amd64 and -arm64 suffixes is nice. In the past we only had -arm64 suffixes and it's a mess in the code base and it's easy to overlook that both have to be handled.

I definitely like always having the suffix. But I'm considering flatcar-stable-amd64-capi-v1.22.11 for the format, so then the base remains the same as flatcar-stable-amd64 and capi-v1.22.11 becomes a "flavor". And over time we can make flatcar-stable-amd64-gen3, flatcar-stable-amd64-capi-v1.22.11-gen3 and whatever else we need.

Same for the offer, I'd make them stable and stable-capi.

And the gallery name can be downcased for consistency with everything else.

[invidian]

To summarize, I think things would look roughly like following:

galleries:

- publicNamePrefix: flatcar
  publicGalleryName: flatcar-52089376-cc7d-456e-8541-3c634dd271a2
  imageDefinitions:
  - name: flatcar-stable-amd64
    publisher: flatcar
    offer: stable
    sku: flatcar-stable-amd64
    versions:
    - 3227.2.1 # latest
    - 3227.2.0
  #- name: flatcar-stable-amd64-gen1
  #  publisher: flatcar
  #  offer: stable-gen1
  #  sku: flatcar-stable-amd64-gen1
  #  versions:
  #  - 3227.2.1 # latest
  #  - 3227.2.0
  #- name: flatcar-stable-arm64
  #  publisher: flatcar
  #  offer: stable
  #  sku: flatcar-stable-arm64
  #  versions:
  #  - 3227.2.1 # latest
  #  - 3227.2.0
  #- name: flatcar-beta-amd64
  #  publisher: flatcar
  #  offer: beta
  #  sku: flatcar-beta-amd64
  #  versions:
  #  - 3277.1.1 # latest
  #  - 3277.1.0

- publicNamePrefix: flatcar-capi
  publicGalleryName: flatcar-capi-ea790d66-b5e8-44ae-9100-41db645c249e
  imageDefinitions:
  - name: flatcar-stable-amd64-capi-v1.22.11
    publisher: flatcar
    offer: stable-capi
    sku: flatcar-stable-amd64-capi-v1.22.11
    versions:
    - 3227.2.1
    - 3227.2.0
  #- name: flatcar-stable-amd64-capi-v1.22.11-gen1
  #  publisher: flatcar
  #  offer: stable-capi-gen1
  #  sku: flatcar-stable-amd64-capi-v1.22.11-gen1
  #  versions:
  #  - 3227.2.1
  #  - 3227.2.0
  #- name: flatcar-beta-amd64-capi-v1.22.11
  #  publisher: flatcar
  #  offer: beta-capi
  #  sku: flatcar-beta-amd64-capi-v1.22.11
  #  versions:
  #  - 3277.1.1
  #  - 3277.1.0

[invidian]

Some updates:

• Initial galleries has been created. flatcar-ae8eca70-1f72-48a7-9295-c25706de26db and flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 galleries has been created.
  It was not possible to use flatcar-capi as a gallery prefix, so we used flatcar4capi instead.
• Initial set of images is published. Flatcar one is actually right now only replicated to West Europe, where we do builds. We should consider replicating it to all regions.
  CAPI images are initially replicated to regions used by CAPZ CI.
• The build process is manual right now. I'm working on documenting the process and in the future we should consider automating it.

"demo":

$ az sig image-definition list-community --public-gallery-name flatcar-ae8eca70-1f72-48a7-9295-c25706de26db --location westeurope
HyperVGeneration    Location    Name                  OsState      OsType    UniqueId
------------------  ----------  --------------------  -----------  --------  --------------------------------------------------------------------------------------------
V2                  westeurope  flatcar-stable-amd64  Generalized  Linux     /CommunityGalleries/flatcar-ae8eca70-1f72-48a7-9295-c25706de26db/Images/flatcar-stable-amd64
$ az sig image-definition list-community --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --location westeurope

HyperVGeneration    Location    Name                                OsState      OsType    UniqueId
------------------  ----------  ----------------------------------  -----------  --------  ---------------------------------------------------------------------------------------------------------------
V2                  westeurope  flatcar-stable-amd64-capi-v1.23.13  Generalized  Linux     /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.23.13

Work items:

• Set replication for Flatcar images.
• Finish documentation.

[invidian]

Updates:

• Flatcar gallery had to be re-shared which changed it's name to flatcar-23485951-527a-48d6-9d11-6931ff0afc2e. CAPI gallery remains intact.
• Internal documentation and automation has been added so while still manually, the publishing process for new Flatcar and Kubernetes versions should be just one script invocation from now on.

[invidian]

We already hit issues several times because images are not replicated to all regions. Any objections from enabling it for both Flatcar and CAPI images?

[pothos]

Sounds good, it takes long to run anyway

[invidian]

I'll let the idea soak for some time and enable it then, probably next week or so :)

[invidian]

I will enable replication to the regions returned by this query:

BLACKLISTED_TARGET_REGIONS=${BLACKLISTED_TARGET_REGIONS:-polandcentral australiacentral2 brazilsoutheast centraluseuap eastus2euap eastusstg francesouth germanynorth jioindiacentral norwaywest southafricawest switzerlandwest uaecentral}
DEFAULT_TARGET_REGIONS=$(az account list-locations -o json | jq -r '.[] | select( .metadata.regionType != "Logical" ) | .name' | sort | grep -v -E "(${BLACKLISTED_TARGET_REGIONS// /|})" | tr \\n ' ')

Which currently are:
australiacentral australiaeast australiasoutheast brazilsouth canadacentral canadaeast centralindia centralus eastasia eastus eastus2 francecentral germanywestcentral japaneast japanwest jioindiawest koreacentral koreasouth northcentralus northeurope norwayeast qatarcentral southafricanorth southcentralus southeastasia southindia swedencentral switzerlandnorth uaenorth uksouth ukwest westcentralus westeurope westindia westus westus2 westus3

[invidian]

Image for Flatcar stable version 3510.2.2 has been now published to 37 Azure locations. Given that, I'm going to close this issue as completed.