Bug 1789581: Fix osImageURL upgrade race #1357

runcom · 2020-01-08T14:28:29Z

This patches does two things:

~~makes sure we create the osimageurl configmap before the new MCO deployment manifest is applied~~
version the osimageurl configmap

NOTE

This patch requires a 4.2.z patch to version the osimageurl config map there as well (done)
~~it might be needed to version the ControllerConfig as well (???)~~

related bug https://bugzilla.redhat.com/show_bug.cgi?id=1786993

runcom · 2020-01-08T14:30:39Z

/hold

cgwalters · 2020-01-08T14:34:43Z

Sad we're not 🦊 ing it...

ashcrow · 2020-01-08T14:38:34Z

pkg/operator/sync.go

@@ -686,6 +686,11 @@ func (optr *Operator) getOsImageURL(namespace string) (string, error) {
 	if err != nil {
 		return "", err
 	}
+	releaseVersion := cm.Data["releaseVersion"]
+	optrVersion, _ := optr.vStore.Get("operator")
+	if releaseVersion != optrVersion {


Is there any valid cases where the configmap from a previous setup would either be unversioned or not match while transitioning to versioned?

Previous configmaps won't be versioned; but that's fine because Golang doesn't have Option<> so the releaseVersion will be the empty string, which won't match. Which is what we want - we'll ignore the previous unversioned configmap.

notice there would still be a possible race where the 4.2 MCO reads the new 4.3 osimageurl configmap and could generate a new rendered mc with a newer osimageurl, that's why I believe we need to patch 4.2 as well to take versioning into account

notice there would still be a possible race where the 4.2 MCO reads the new 4.3 osimageurl configmap and could generate a new rendered mc with a newer osimageurl, that's why I believe we need to patch 4.2 as well to take versioning into account

I would rather we address that race condition directly rather than trying to patch 4.2 which requires us to enforce that 4.2.x (unpatched) upgrades through 4.2.x+1(patched) before going to 4.3. That's possible in theory but seems to be cumbersome in practice.

Some thoughts:

Version the key in the osimageurl configmap (e.g. osImageURL-0.0.1-snapshot: ...) so that each MCO release has its own image reference. This would address this upgrade issue by changing the key name in 4.3 and protects us from any future upgrade race-conditions and failures. It would also allow you to do some weird cross-version testing, like if you wanted to have multiple MCOs referencing different osimages for some reason.

Set the osimageurl value on the MCO deployment directly - it can be a flag passed in to the operator, the release infra will replace the image reference in the deployment manifest the same as it does in the configmap. If there are other reasons for using a configmap, the MCO deployment can be changed to reference a configmap mounted in just like all the other images. This would reduce any version issues to mismatches in the CC/MCC which we've dealt with before.

Make MCO updates atomic by putting everything that always and only updates together together. The osimageurl, MCO, MCC controllers, and MC templates. Simplify controller architecture, merge "operator" and "controller" #878

cgwalters · 2020-01-08T14:47:00Z

I haven't traced through the code and the logs from the race, but I completely believe you have and this looks highly likely to fix the problem.

I looked at previous variants of this like #1198 and the original was 835278f

Looking over the manifests we install, I think this is the last thing that is input to the MachineConfig that isn't versioned. The other CVO-installed manifests are pure kube level stuff like RBAC rules etc.

/approve
/lgtm

ashcrow · 2020-01-08T20:11:25Z

Prometheus was not happy.

/test ci/prow/e2e-aws

ashcrow · 2020-01-08T20:30:52Z

/test ci/prow/e2e-aws

kikisdeliveryservice · 2020-01-08T22:14:28Z

/test e2e-aws

This patch also guards against unversioned osImageURL config maps in the future. Signed-off-by: Antonio Murdaca <runcom@linux.com>

Signed-off-by: Antonio Murdaca <runcom@linux.com>

openshift-ci-robot · 2020-01-09T20:12:56Z

@runcom: This pull request references Bugzilla bug 1789581, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug 1789581: Fix osImageURL upgrade race

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2020-01-11T03:20:45Z

/retest