Lab updates 2025-12-16

[DanielFroehlich]

• Pin Update Warning Message to slack channel
• sf1 - dnf update
• inf31/sf2 - dnf update
• ISAR Cluster - cluster update to latest channel (stop mad-metro/regional DR Cluster VMs)
• ISAR Cluster - Update operators
• Upgrade keycloak operator (check channel)
• Upgrade LVM Storage (check channel)
• Upgrade ODF (check channel)
• Upgrade ACM (check channel)
• Upgrade Quay (check channel)
• Upgrade OCPVirt (check channel)
• NetApp CSI driver (check channel)
• Create follow up issue and calendar blocker for next update on the first tue of following month
• Un-Pin Update Warning Message in slack channel
• Backup switch config

Follow up to #298

[DanielFroehlich]

Move from Dec to Jan due to lack of cluster upgrades and allow for Roberts customer demos

[rbo]

Move back into december :-)

[rbo]

ISAR Upgrade

Let's check RuncShareProcessNamespace

https://issues.redhat.com/browse/RUN-3748

export NODE_NAME=storm2; oc debug "node/${NODE_NAME}" -- chroot /host bash -c 'crio status config | grep "default_runtime"'
Starting pod/storm2-debug-ztxxs ...
To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`.
time="2025-12-15T07:57:57.614729339Z" level=info msg="Updating config from single file: /etc/crio/crio.conf"
time="2025-12-15T07:57:57.61476105Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf"
time="2025-12-15T07:57:57.615823163Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d"
time="2025-12-15T07:57:57.615890814Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/00-default"
time="2025-12-15T07:57:57.616279134Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime"
    default_runtime = "runc"

Removing debug pod ...

% oc get pods -A -o yaml | grep shareProcessNamespace | sort | uniq -c
   5     shareProcessNamespace: false
  28     shareProcessNamespace: true

❌ We cannot process the update.

List of first versions affected

• 4.20.4
• 4.19.19
• 4.18.29 (only in candidate channels)
• 4.17.44
• 4.16.53 (only in candidate channels)
• 4.14.59
• 4.12.83 (only in candidate channels)

Currently we have not promoted 4.15.z or 4.13.z with the offending change.

Let's check another version:

NetworkManagerOVNBridgeMapping

https://issues.redhat.com/browse/CORENET-6483

We're using OVN Bridge as well.

❌ We cannot process the update.

Let's update to 4.19.21; that is the only clean / bug free update. And for an update to 4.20 we need >4.19.13 anyway.

[rbo]

Update 4.19.21 -> 40.20.8 work in progress.

[rbo]

NetApp CSI driver (check channel)

% oc get installplan -A | grep -i trid
openshift-operators                   install-79s4l   trident-operator.v25.6.2                          Manual      false
openshift-operators                   install-frjx7   trident-operator.v25.6.2                          Manual      false
% oc get subs.operator -A | grep -i  trident
openshift-operators                   trident-operator                                                             trident-operator                       certified-operators   stable

[rbo]

Upgrade ACM (check channel)

Changed from 2.14 channel to 2.15

[DanielFroehlich]

Upgrade ODF to 4.20.3. is in progress now...
LVM Storage re-installed into new namespace openshift-lvm-storage, see #328
... ODF upgrade completed!

[rbo]

NetApp

The Subscription is not visible in the WebUI

[rbo]

Quay

as well

Found : https://console-openshift-console.apps.isar.coe.muc.redhat.com/k8s/ns/openshift-operators/operators.coreos.com~v1alpha1~Subscription/quay-operator

Changed to 3.16

[DanielFroehlich]

strange that the Subsriptions do not show up in the console - is that a bug???

[rbo]

strange that the Subsriptions do not show up in the console - is that a bug???

Looks like... Really strange. I have to investigate...