4.12 -> 4.13 upgrade fails "Release not accepted"

[volkermauel]

Describe the bug
Server Version: 4.12.0-0.okd-2023-04-16-041331
Kubernetes Version: v1.25.0-2824+27e744f55d2e99-dirty

Tried to do an upgrade via CLI to

Name: 4.13.0-0.okd-2023-05-22-052007
Digest: sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d
Created: 2023-05-22T10:51:47Z
OS/Arch: linux/amd64
Manifests: 665
Metadata files: 1

Pull From: registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d
Version
IPI vSphere 8.0

UI
UI says "Release not accepted"

Retrieving payload failed version="4.13.0-0.okd-2023-05-22-052007" image="registry.ci.openshift.org/origin/release@sha256:16b722…able to verify sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d against keyrings: verifier-public-key-ci

---

cluster version operator log

I0527 11:11:57.596273       1 cvo.go:570] Started syncing cluster version "openshift-cluster-version/version", spec changes, status, and payload (2023-05-27 11:11:57.596249162 +0000 UTC m=+1276.455999505)
I0527 11:11:57.596393       1 cvo.go:595] Desired version from spec is v1.Update{Version:"4.13.0-0.okd-2023-05-22-052007", Image:"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d", Force:false}
I0527 11:11:57.596424       1 sync_worker.go:468] Initializing prior known value of enabled capabilities from ClusterVersion status.
I0527 11:11:57.596453       1 sync_worker.go:261] syncPayload: 4.13.0-0.okd-2023-05-22-052007 (force=false)
I0527 11:11:57.596657       1 event.go:285] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RetrievePayload' Retrieving and verifying payload version="4.13.0-0.okd-2023-05-22-052007" image="registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d"
I0527 11:11:58.060106       1 sync_worker.go:437] loadUpdatedPayload syncPayload err=The update cannot be verified: unable to verify sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d against keyrings: verifier-public-key-ci
I0527 11:11:58.060245       1 event.go:285] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="4.13.0-0.okd-2023-05-22-052007" image="registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d" failure=The update cannot be verified: unable to verify sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d against keyrings: verifier-public-key-ci
I0527 11:11:58.060270       1 status.go:170] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:3, Failure:error(nil), Done:0, Total:0, Completed:0, Reconciling:false, Initial:false, VersionHash:"", Architecture:"", LastProgress:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Actual:v1.Release{Version:"4.13.0-0.okd-2023-05-22-052007", Image:"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d", URL:"", Channels:[]string(nil)}, Verified:false, loadPayloadStatus:cvo.LoadPayloadStatus{Step:"RetrievePayload", Message:"Retrieving payload failed version=\"4.13.0-0.okd-2023-05-22-052007\" image=\"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d\" failure=The update cannot be verified: unable to verify sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d against keyrings: verifier-public-key-ci", AcceptedRisks:"", Failure:(*payload.UpdateError)(0xc000a38fc0), Update:v1.Update{Version:"4.13.0-0.okd-2023-05-22-052007", Image:"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d", Force:false}, Verified:false, Local:false, LastTransitionTime:time.Time{wall:0xc114973f8394d397, ext:1276919835359, loc:(*time.Location)(0x2f4af80)}}, CapabilitiesStatus:cvo.CapabilityStatus{Status:v1.ClusterVersionCapabilitiesStatus{EnabledCapabilities:[]v1.ClusterVersionCapability(nil), KnownCapabilities:[]v1.ClusterVersionCapability(nil)}, ImplicitlyEnabledCaps:[]v1.ClusterVersionCapability(nil)}}
I0527 11:11:58.060471       1 status.go:90] merge into existing history completed=false desired=v1.Release{Version:"4.13.0-0.okd-2023-05-22-052007", Image:"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d", URL:"", Channels:[]string(nil)} last=&v1.UpdateHistory{State:"Partial", StartedTime:time.Date(2023, time.May, 26, 6, 14, 56, 0, time.Local), CompletionTime:<nil>, Version:"4.13.0-0.okd-2023-05-22-052007", Image:"registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d", Verified:false, AcceptedRisks:""}
I0527 11:11:58.060644       1 cvo.go:572] Finished syncing cluster version "openshift-cluster-version/version" (464.388455ms)

[LorbusChris]

Try pulling from quay.io/openshift/okd@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d

[volkermauel]

running into the same issue

oc adm upgrade --to-image=quay.io/openshift/okd@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d --allow-explicit-upgrade --allow-upgrade-with-warnings

Retrieving payload failed version="" image="quay.io/openshift/okd@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c8…able to verify sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d against keyrings: verifier-public-key-ci

[vrutkovs]

See #1612 (comment)

[vrutkovs]

Are you sure you previous version was 4.12.0-0.okd-2023-04-16-041331 and the previous upgrade was finished?

[vrutkovs]

Actually, its probably indeed broken - release-controller uses the new key, but cluster-update was not updated - openshift/cluster-update-keys#50 should fix it

[volkermauel]

99,9% certain

the cluster was installed with the last stable version available, what at the time of installation was 4.12 and hadn't done any upgrade since no newer version was available yet.

In the console it says

"Last completed version
4.12.0-0.okd-2023-04-16-041331"

anything else we could check to verify?

[madpearl]

I've got the same warning and my step to go through
oc adm upgrade
oc adm upgrade --clear
oc adm upgrade --to 4.13.0-0.okd-2023-05-22-052007 --force

[volkermauel]

This worked, thanks!