Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decision Record: capi images mandatory #582

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Decision Record: capi images mandatory #582

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a9c20ce
Adds Decision record document
mxmxchere May 2, 2024
7f4c85d
Fix syntax
mxmxchere May 2, 2024
2a9099a
Fix syntax
mxmxchere May 2, 2024
159dfb1
include review aspects
mxmxchere May 2, 2024
d07a2d9
Merge branch 'main' into require-k8s-images
garloff Jun 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
47 changes: 47 additions & 0 deletions Standards/scs-0115-v0-capi-images.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: Cluster-API images
type: Decision Record
status: Draft
track: Iaas
---

<!---
This is a template striving to provide a starting point for
creating a decision record adhering to scs-0001.
Replace at least all text in the sections not marked as OPTIONAL.
See https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0001-v1-sovereign-cloud-standards.md
--->

## Abstract

The problem that brings us here is how to update and keep track with upstream kubernetes versions when using the cluster-stacks approach on OpenStack. As the [SCS K8S Version Policy](https://docs.scs.community/standards/scs-0210-v2-k8s-version-policy#motivation) formulates, SCS is quite eager to keep track with the upstream work and does not want to fall behind:

> We want to achieve an up-to-date policy, meaning that providers should be mostly in sync with the upstream and don't fall behind the official Kubernetes releases.

## Context

The following scenario: 1 CSP with 100 customers each deploying at least one cluster.
One image will be downloaded (from upstream) and uploaded (to glance) and stored 99 times.

## Decision

Mandate the CSP to keep cluster-api Images up-to-date.

From the meeting notes:

- will save time and money for both CSP and customer
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also saves physical resources & power.

- would guarantee quality and timelness of image
- regardless of CSP taste, this KaaS tech is part of SCS
- CSP won't have noticable disadvantages from providing the image

This would lead to the following standard changes:
Create a copy of the [default image list](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs-0104-v1-images.yaml) and change the capi block from "recommended" to "mandatory".

Also, strictly speaking, we have to change [the standard image format definition](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0104-v1-standard-images.md#image-specification-class-of-images) so that "mandatory" for an image class does not mean that at least one image MUST be present in glance but all that match the [SCS K8S Version Policy](https://docs.scs.community/standards/scs-0210-v2-k8s-version-policy#motivation). Also the checking script has to be adapted.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So let me get this right, as a CSP -who has no interest in offering k8s services based on CAPI- I'm forced to provide not one but how every many Images that are needed only in the scope of CAPI and keep them "maintained"?

I would rather like to not mix standards and keep the separation between IaaS, KaaS + Ops "clean".

We're already looking into how to align our Gardener based k8s Setup with the KaaS Standards and thus would follow the mandatory versions and thus at this point in time we see no benefit in accepting additional "mandatory" Images.

Copy link
Contributor Author

@mxmxchere mxmxchere May 24, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. Yes, the standard, in the above form, would force a CSP to maintain k8s CAPI images, even in a timely manner.

Maybe a compromise could be that this standard is only mandatory when your are not providing a SCS-compliant k8s service (which you plan to do, so this would not affect you). However that would also connect IaaS and KaaS layers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea behind this was to prepare an SCS-standardised infrastructure as well as possible for the use of SCS cluster stacks. A customer can use the SCS cluster stacks or cluster API in general completely independently of the Kubernetes as a Service preferred and offered by the CSP. If it is specified in the standard that a CSP must provide the cluster API images, then the SCS cluster stacks could be used without additional effort (providing an image for each project, which is definitely a disadvantage if you have to do that). I think it's basically the same as with the special flavors with local storage. Their existence was necessary due to the problems with Kubernetes managed by cluster API / SCS cluster stacks.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the summary @berendt - this is exactly what I'd have written.
From my point of view the impact for the CSP is fairly low while having a large impact for the customer whose Cluster-API stuff simply works.

Copy link
Contributor

@mbuechse mbuechse Jun 3, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the local storage, there was nothing a user could do if they needed the performance. The images, however, can be uploaded by the user. Yes, this does require some additional effort and it might waste some time and energy and produce emissions etc., but it is possible. So from my POV it should be up to the CSP to decide whether they want to meet their customers halfway here or not (if they do, they can advertise as much). I don't see the need to make it mandatory.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being late, but how are these images provided? Are they downloadable for the CSP? Or does the CSP have to create those images for the operated environments? That might be difficult for CSPs that don't use/provide cluster-api services - even to test those images...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment we provide them at https://github.com/osism/k8s-capi-images. We do not test them at the moment. There is an issue open for this. We are currently depending on the feedback from the SCS K8s team for the tests. If there is no negative feedback from them, we assume that the images are working.

Building the images themselves is trivial with the https://github.com/kubernetes-sigs/image-builder/ .

Copy link
Member

@fkr fkr Jun 3, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've discussed this in today's PB again and the overall consensus was that these images should be RECOMMENDED and not MANDATORY.
This allows CSPs to follow the recommendation, without having the burden to maintain the images if they really don't want to carry them. Imho we should make sure the discussion is reflected in the decision record.

Copy link
Contributor Author

@mxmxchere mxmxchere Jun 4, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommended instead of mandatory is the status quo, from my point of view we need no decision record for that.

The only thing that can be "improved" in the standards is the concatenation between the SCS K8S Version Policy 0210-v2 and the recommended issues. But as these standards are neither binding nor would an additional concatenation document add any new informational value I see no point in doing that either. In the worst case we build in a contradiction somewhere and create confusion. So from my point of view we can just close this PR (unmerged)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mxmxchere I suggested the decision record so that the discussion is manifested outside of this PR. ;)


## Consequences

CSPs will have the additional burden to provide cluster-api images in glance (and update them regularly).

The CSP will save bandwitdth and diskspace when images are only downloaded and stored once, instead of per Customer.
It will be easier and quicker for customers to update their clusters.