What's the deal with 4.13.0-0.okd-2023-09-03-112738

[ibotty]

It's not listed in the releases.

[vrutkovs]

Its a regular nightly, why would it be listed in the releases?

[ibotty]

That's the question. Two of my clusters listed this release as soon as the other one from 2023-09-03 had been installed.

[madpearl]

I get also upgrade release to 4.13.0-0.okd-2023-09-03-112738
`oc adm upgrade
Cluster version is 4.13.0-0.okd-2023-09-03-082426

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

VERSION IMAGE
4.13.0-0.okd-2023-09-03-112738 registry.ci.openshift.org/origin/release@sha256:733ea877551d8de77541b62b7e404d8e522ca20fede56ad9d5237c07d929021d`

[vrutkovs]

Interesting. It indeed was showing in "Updates to" list for latest stable.

I removed this nightly, so it should disappear for now. Lets see if this happens again - perhaps our release-controller is misconfigured again

[madpearl]

Now is Okay
`
oc adm upgrade
Cluster version is 4.13.0-0.okd-2023-09-03-082426

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.
`

[ibotty]

That's unfortunate for my test cluster that I already updated to 4.13.0-0.okd-2023-09-03-112738. One node cannot pull some images now.

Sep 05 08:42:47 ... bash[6471]: Trying to pull registry.ci.openshift.org/origin/4.13-2023-09-03-112738@sha256:25919ca30f4a1d410b2301d191b8f268f131372da04c4646a45c1127c9492e22...
Sep 05 08:42:48 ... bash[6471]: Error: initializing source docker://registry.ci.openshift.org/origin/4.13-2023-09-03-112738@sha256:25919ca30f4a1d410b2301d191b8f268f131372da04c4646a45c1127c9492e22: reading manifest sha256:25919ca30f4a1d410b2301d191b8f268f131372da04c4646a45c1127c9492e22 in registry.ci.openshift.org/origin/4.13-2023-09-03-112738: received unexpected HTTP status: 500

[vrutkovs]

There are no breaking changes between these, so you can recover by "downgrading" back to stable

[ibotty]

That's harder if the kubelet on the node does not start. I think I'll just reprovision the node.

[ssams]

That's harder if the kubelet on the node does not start.

Last time this problem occurred (#1433), I managed to get around some similar problems and recover such a broken cluster - the related discussion #1435 would contain some hints that may be useful when you need to recover kubelet in this situation (i.e. in case reprovisioning is not an option).

[ibotty]

I'll close this. There is nothing to do here afaict.