Storage upgrade mechanism

[lavalamp]

Before we remove an API object version, we need to be certain that all stored objects have been upgraded to a version that will be readable in the future. The old plan for that was to run the cluster/upgrade-storage-objects.sh script after each upgrade. Unfortunately, there are a number of problems with this:

• The script is old and incomplete, as api authors haven't been consistently adding their API objects to it (no doubt due to a lack of documentation in the right places).
• Instructions to run the script in the release notes have been spotty.
• We believe not all distributions have been running the script after upgrades.

Therefore, we need to design a robust solution for this problem, which works in the face of HA installations, user-provided apiservers, rollbacks, patch version releases, etc. Probably this won't look much like a script that you run after an upgrade, instead it may look more like a system where apiservers come to consensus on the desired storage version, plus a controller that does the job that script was supposed to do after a consensus change.

In the mean time, it is not safe to remove API object versions, so we are instating a moratorium on API object version removal until this system is in place. (Deprecations are still fine.)

[thockin]

Thanks, Daniel. Good summary. To echo:

it is not safe to remove API object versions, so we are instating a moratorium on API object version removal

Emphasis on removal. It sucks to carry old APIs forward, but this is a liability I think we can't ignore. We've gotten lucky this far.

@kubernetes/sig-network-api-reviews @cmluciano @dcbw @danwinship

@kubernetes/api-approvers @kubernetes/api-reviewers

[smarterclayton]

For context the openshift approach is we force a migration before every upgrade starts, and the migration has to complete. Migration is a kubectl cmd that reads all objects and writes a no-op change (which forces storage turn over). We use this for protobuf migration, field defaulting, new versions, storage versions conversion, self link fixup, etc.

It's basically a production version of upgrade-storage-objects.sh - depending on how complicated we want to get, i'd say it's the minimum viable path, while the cluster consensus is probably fine but more complicated.

https://github.com/openshift/origin/blob/master/pkg/oc/admin/migrate/storage/storage.go

[thockin]

Is this blocking the removal of Alpha APIs, too?

[lavalamp]

@smarterclayton Is that an upstream command?

[lavalamp]

@thockin I think it depends on how the alpha API is storing its objects.

[smarterclayton]

Downstream, it is a bunch of utility code for a migration framework (we have other migrators for things like RBAC, alpha annotations to beta fields, image references) and then a set of simple commands. I don't know that I think of this as kubectl as much as kubeadm or a separate command for admins (in openshift, oc adm is all admin focused commands and oc is all end user focused, but we don't quite have the equiv in kube)

[lavalamp]

@smarterclayton Gotcha. I will take a look at it but that still doesn't sound like an ideal way to solve the problem. Like, it's good if you're going to run one or two of these things with super-well-trained humans doing the upgrade, but Kubernetes isn't installed like that :)

[smarterclayton]

It's just part of general cluster upgrade operations. Someone has to roll the masters. Someone has to lay down new config. I agree there can be magic involved. But magic may have higher cost in some scenarios. On Sep 8, 2017, at 4:28 PM, Daniel Smith <notifications@github.com> wrote: @smarterclayton <https://github.com/smarterclayton> Gotcha. I will take a look at it but that still doesn't sound like an ideal way to solve the problem. Like, it's good if you're going to run one or two of these things with super-well-trained humans doing the upgrade, but Kubernetes isn't installed like that :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52185 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9i4xB-URdFNh97YstqxPPuQg8_cks5sgaN8gaJpZM4PReU1> .

[enj]

I have worked closely with OpenShift's installer/upgrade team on their use of the oc adm migrate storage command. I will say running it automatically pre and post upgrade is the trivial part. The command is surprisingly good at finding random edge cases. It could be turned into a controller of sort, though I am not sure what state the controller would try to drive the system from/to.

[timothysc]

Per conversation on sig-api-machinery, there was general consensus that it makes more sense to have this as a controllers responsibility with some set of strategies.

[bgrant0607]

We need to figure out downgrades, also, which are similar. Right now, a downgrade will strand storage of any new APIs that are created.

[liggitt]

We need to figure out downgrades, also, which are similar. Right now, a downgrade will strand storage of any new APIs that are created.

Brand new resources will get orphaned on downgrade, but should be inert.

New versions of existing resources shouldn't start persisting with the new version until n+1 versions from when they are released (or rolling HA API upgrades won't work). If we only support single version downgrade, there shouldn't be any issues.

[bgrant0607]

Related: #4855, #3562, #46073, #23233

[bgrant0607]

@liggitt Inert resources have some of the timebomb characteristics of annotations discussed in #30819, upon re-upgrade.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[enj]

/lifecycle frozen

[thockin]

Any status update here? We'd like to remove the deprecated v1beta1 NetworkPolicy.

[liggitt]

I haven't seen a better suggestion than a no-op read/write cycle via the API on the affected resources.

Enforcing that is done seems like the responsibility of the deployment mechanism, so it can be staged after all apiservers in a HA deployment are upgraded.

[liggitt]

In the meantime, you can start disabling serving the deprecated object by default. The types still remain, so they can be read from storage, and we still have the in tree debt, but it pushes people to use the new types as new clusters are deployed.

[bgrant0607]

Speaking of HA, today each apiserver has its own independent default storage versions, which are imposed immediately upon apiserver upgrade. That doesn't seem desirable.

[lavalamp]

https://docs.google.com/document/d/1eoS1K40HLMl4zUyw5pnC05dEF3mzFLp5TPEEt4PFvsM/edit#heading=h.xgjl2srtytjt

[caesarxuchao]

KEP: https://github.com/kubernetes/community/pull/2524/files.

[athenabot]

@lavalamp @caesarxuchao
If this issue has been triaged, please comment /remove-triage unresolved.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the help-wanted label.

🤖 I am a bot run by vllry. 👩‍🔬

[caesarxuchao]

/remove-triage unresolved

[dims]

long-term-issue (note to self)

[andrewsykim]

/assign

[shaneutt]

This is marked as priority/important-soon but given the last updates, that wouldn't seem to be the case. Can someone catch me up on what the status is here? Is this simply stale since we don't have anyone to move it forward?

[enj]

/lifecycle frozen

The foundational pieces are being worked on here:

kubernetes/enhancements#2339
kubernetes/enhancements#4192

[enj]

The #sig-api-machinery-storageversion-dev slack channel is available as well.