KEP-21: Upgrading KUDO #1124

ANeumann82 · 2019-12-02T10:35:39Z

What this PR does / why we need it:
At the moment, we do not have a process or code for upgrading KUDO.

This is an early draft to invite discussion.

Added newlines

alenkacz

Just did a first pass, I need to process/think about it now. Few high level ideas that might simplify some parts:

I think a lot of migration code could be just updating the resources in the cluster, pretty much doing kudo init -o yaml --dry-run | kubectl apply -f -
I think using conversion webhooks will simplify this a lot and if we commit to always use it for newer version it will make our life easier, I think the supported k8s version is not limiting us here

Also probably worth some thought - what if we supported upgrades via helm? Would that actually help us in any way? How would that look like?

alenkacz · 2019-12-03T10:37:41Z

keps/0021-kudo-upgrade.md

+
+## Open Questions
+- Lowest supported K8s version
+- Do we want to support downgrades? 


I would say no right now, definitely not in this kep - on the other hand, what if I upgrade but need to rollback again? 🤔

Yeah, I don't really want to support it either, but I think it's important enough to have a (small) discussion about it. And if we decide to go with the conversion webhooks, it might not be that difficult to do.

oh we have to do conversion webhooks - do you feel like there's any reason not to do them?

alenkacz · 2019-12-03T10:38:50Z

keps/0021-kudo-upgrade.md

+## Open Questions
+- Lowest supported K8s version
+- Do we want to support downgrades? 
+- Split versions between KUDO manager and KUDO CLI?


could you write more about why?

They're basically two different programs that only communicate via CRDs. It might make something easier having split versions, for example bug fixes in manager only without having a new KUDO CLI version that does not change anything in the CLI.

But it would introduce a new compatibility matrix, and having "empty" version changes on manager or CLI isn't that big of an issue. I lean towards having a single version for both parts of KUDO.

yeah, me too I think until there's a compelling reason to split them, having them as one is a big win, I would almost go as far as resolve this question with - no :) especially since there's no real as for this and the scope of this KEP is already pretty big

alenkacz · 2019-12-03T10:40:40Z

keps/0021-kudo-upgrade.md

+Versioned: No, but closely tied to KUDO manager
+
+The KUDO manager has a set of prerequisites that need to be present for the manager to run successfully. They are
+the least likely to change, but probably the most specific. If we make changes here, we need to implement custom


by the way - new features might actually touch those, e.g. take a look at my webhook PR. I wonder what is a good strategy for those. I mean in there I:

added new prereqs, that's easy, you just create those on upgrade

updated manager - could we just patch the manager in that case?

Also could you maybe show how would the custom migration code look like? I think the webhook is a great example

alenkacz · 2019-12-03T12:06:07Z

keps/0021-kudo-upgrade.md

+Each migration should have a validate-step that checks if the migration is possible.
+
+**Alternative update process**
+- Can we just delete them all and reinstall them? Probably not


what about updating them? just as you do with any other resource, what are the challenges with that?

Yes, updating is probably the only possible way. The idea with "Delete all and install fresh" would be to not do migrations but to always just install the expected state, but I don't think that's really possible.

alenkacz · 2019-12-03T12:07:14Z

keps/0021-kudo-upgrade.md

+
+- Use semantic versioning for the manager binary
+  - ToBeDiscussed: Use different versioning for Manager and CLI?
+- The manager needs to be shut down before updating the CRDs


does it? why? also are we talking about updating crd of the same version or updating to newer version or both?

Well - this assumes we're not using the conversion web hook, and therefore can't serve different versions of a CR and need to migrate them manually. In that case, having a manager running that tries to read or modify the old version of CRs would be problematic.

Regarding updating CRDs with the same version - Is that something that is commonly used? I can imagine updating a CRD with new fields, assuming it's backwards compatible and defaults are provided, but in which case is the version of a CRD updated then? Only in case of breaking changes?

I just read the K8s documentation on API changes. (https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) Explains a lot. I'll update the KEP.

alenkacz · 2019-12-03T12:08:46Z

keps/0021-kudo-upgrade.md

+fields. 
+- Existing CRs need to be migrated to new versions
+- K8s CRD support
+  - Which K8s version do we want/need to support?


I guess this is kind of given by what kubernetes version we support in which version of manager? Maybe we should somehow define that as well?

Well, it mostly comes down to: Do we want to use conversion webhooks or not.

alenkacz · 2019-12-03T12:10:12Z

keps/0021-kudo-upgrade.md

+    - MultiVersion is supported since 1.11 (manual conversion)
+    - WebHook conversion since 1.16
+  - WebHook conversion would allow us to transparently switch to a new CRD version without manually migrating all existing CRs
+- CRD versioning


I would probably rather stick with one version if we can for now to not make things compilcated.

Hmmmm, if we don't enforce the same requirements on CRD stability as k8s itself, this might be possible. We would have to enforce that KUDO CLI is at least the same version as the installed KUDO manager though, otherwise and older CLI version could drop fields from a CR that it doesn't know about...

alenkacz · 2019-12-03T12:11:23Z

keps/0021-kudo-upgrade.md

+#### Proposal for update process
+Integrated into `kudo init --upgrade`
+
+To support older K8s versions, no WebHook conversion is used. We only serve a single CRD-Version and migrate existing CRs in the update process


I think we're fine in terms of kube version here Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13.

Which means, the operating cluster needs to have webhook conversion explicitly enabled for KUDO to work. I'm not sure if we want to make this a requirement for KUDO, tbh.

alenkacz · 2019-12-03T12:13:18Z

keps/0021-kudo-upgrade.md

+- Do we allow an older KUDO CLI to be used with a newer KUDO installation?
+
+#### Proposal for update process
+User has to download newest KUDO version.


yeah I think replacing the binary or dowing upgrade via brew is good enough here 👍

Yeah, i think this is the most simple part of the update process :)

alenkacz · 2019-12-11T09:07:17Z

keps/0021-kudo-upgrade.md

+    - Every migration step after the installed version is executed
+    - Every migration step only migrates the prerequisites from the previous version to the marked version of the migration
+
+2. Have only one setup/update version in KUDO


I think we should start with this and introduce migrations when there's very compelling reason to do so. I can see all our current migration needs in the last versions to be covered with this.

Added section about operator compatibility Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

runyontr · 2020-01-22T18:47:10Z

keps/0021-kudo-upgrade.md

+We may arrive at a point where we need to implement custom migration logic between KUDO versions, but so far this would mean a lot of
+implementation effort without a defined use case.    
+
+### KUDO Manager


Is there any expectation on the manager to re-process plans that have been run on previous versions? I'm thinking of scenarios where there might be a bug in a previous version that would have caused the plan to not be executed correctly, and want to re-process the plans to correct the mistake.

I don't think that would be a good idea to do automatically. For some plans that may be ok, but others not as much - restore plans, node replaces, etc.

There will be some kind of framework to write migrations, I guess, to fix major issues that need to be fixed, but nothing like just re-processing plans.

kensipe · 2020-03-10T14:11:19Z

@ANeumann82 ping... what is next here

kensipe · 2020-04-09T13:44:08Z

retargeting for 0.13.0

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

… scope here Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

ANeumann82 added 6 commits November 28, 2019 13:11

Initial commit for KUDO-Upgrade-KEP

fc64ac0

Added details to upgrade KEP

4469d03

More work on upgrades

3c85c76

More work on upgrades

aee9d1d

Updated KEP-Number

51fc13a

Updated TOC

c340d8e

Added newlines

ANeumann82 added the kind/kep Kudo Enhancement label Dec 2, 2019

Added question about lowest supported KUDO version for updates

611e543

alenkacz reviewed Dec 3, 2019

View reviewed changes

ANeumann82 added 5 commits December 4, 2019 14:36

Some work on prereq migrations

8a79ba6

Some work on prereq migrations

3324dad

Improvements how to update prerequisites

6a467cd

Updated document

d3bd795

Added list of potential action items/issues

1be6bc3

alenkacz reviewed Dec 11, 2019

View reviewed changes

Updated CRD version section

fc237f2

kensipe mentioned this pull request Jan 6, 2020

Upgrade KUDO KEP #1151

Open

gerred added the needs review label Jan 6, 2020

ANeumann82 added 2 commits January 7, 2020 13:53

Merge branch 'master' into an/kep-kudo-upgrades

d405332

Added details, cleaned up

7605478

Added section about operator compatibility Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

ANeumann82 marked this pull request as ready for review January 7, 2020 16:22

ANeumann82 requested review from gerred, kensipe, nfnt and zen-dog as code owners January 7, 2020 16:22

ANeumann82 added the epic/upgrade-kudo label Jan 8, 2020

runyontr reviewed Jan 22, 2020

View reviewed changes

kensipe added this to the 0.12.0 milestone Mar 10, 2020

kensipe assigned ANeumann82 Mar 10, 2020

kensipe modified the milestones: 0.12.0, 0.13.0 Apr 9, 2020

ANeumann82 added 3 commits May 26, 2020 13:53

Add new KEP draft

5365775

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

x

c212dc3

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

Some additions

5ac1f84

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

kensipe changed the base branch from master to main June 24, 2020 00:41

ANeumann82 added 3 commits September 22, 2020 14:21

Cleanup KEP to show current state - remove multiple CRD versions from…

17d420b

… scope here Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

Merge branch 'main' into an/kep-kudo-upgrades

b0cd2f2

Update TOC

00554bd

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

alenkacz approved these changes Sep 22, 2020

View reviewed changes

Updated status and overview

81aed0a

Signed-off-by: Andreas Neumann <aneumann@mesosphere.com>

ANeumann82 merged commit 198260f into main Sep 22, 2020

ANeumann82 deleted the an/kep-kudo-upgrades branch September 22, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-21: Upgrading KUDO #1124

KEP-21: Upgrading KUDO #1124

ANeumann82 commented Dec 2, 2019

alenkacz left a comment •

edited

Loading

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 11, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 11, 2019 •

edited

Loading

alenkacz Dec 3, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 3, 2019

ANeumann82 Dec 4, 2019

alenkacz Dec 11, 2019

runyontr Jan 22, 2020

ANeumann82 Apr 15, 2020

kensipe commented Mar 10, 2020

kensipe commented Apr 9, 2020

KEP-21: Upgrading KUDO #1124

KEP-21: Upgrading KUDO #1124

Conversation

ANeumann82 commented Dec 2, 2019

alenkacz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alenkacz Dec 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kensipe commented Mar 10, 2020

kensipe commented Apr 9, 2020

alenkacz left a comment •

edited

Loading

alenkacz Dec 11, 2019 •

edited

Loading