oVirt/RHV disabled nodeip-configuration.service

[mriedmann]

Description

Our RHV IPI installed clusters (4.6.21, 4.6.24) are showing the nodeip-configuration.service as disabled (enabled: false) in 00-master and 00-worker machine config resources. We are currently investigating a major cluster stability problem since our upgrade from 4.6.18 to 4.6.22 in one cluster. This lead us to another issue (https://bugzilla.redhat.com/show_bug.cgi?id=1940939) that discribes a problem with this service, but is explicitly mentioning that the service is enabled (but not working properly). The --node-ip parameter of kubelet is empty, resulting in a flip-flop behavior on nodes binding 2 ips (like the ingress or api IPs).

As far as I understand the code at hand the service should be enabled, according to following source-code parts, because ingress and API IPs are set in the referenced resource (controllerconfig/machine-config-controller):

• machine-config-operator/templates/common/on-prem/units/nodeip-configuration.service.yaml

  Line 2 in 9c6c2bf

  enabled: {{if (onPremPlatformAPIServerInternalIP .)}}true{{else}}false{{end}}

• machine-config-operator/pkg/controller/template/render.go

  Line 442 in 9c6c2bf

  func onPremPlatformAPIServerInternalIP(cfg RenderConfig) (interface{}, error) {

• machine-config-operator/pkg/operator/render.go

  Line 236 in 9c6c2bf

  func onPremPlatformAPIServerInternalIP(cfg mcfgv1.ControllerConfigSpec) (interface{}, error) {

A current workaround is to start the nodeip-configuration service manually, calling systemctl daemon-reload and restarting kubelet. After that the --node-ip argument is showing the right ip and the cluster starts to stabalize.

Steps to reproduce the issue:

1. Install OCP 4.6.21 against RHV
2. Inspect machine-config 00-master
3. Notice that the nodeip-configuration.service is disabled (enabled: false)

Describe the results you received:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 00-master
...
          enabled: false
          name: nodeip-configuration.service
...

Describe the results you expected:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 00-master
...
          enabled: true
          name: nodeip-configuration.service
...

Additional information you deem important (e.g. issue happens only occasionally):

We installed a new OCP 4.6.21 Cluster yesterday, which shows the same misconfiguration, but runs stable for now. Also the unstable cluster was updated to 4.6.24 today showing no change in behavior.

Output of oc adm release info --commits | grep machine-config-operator:

machine-config-operator                        https://github.com/openshift/machine-config-operator                        d5dc2b519aed5b3ed6a6ab9e7f70f33740f9f8af

Additional environment details (platform, options, etc.):

RHV 4.4

[kikisdeliveryservice]

Could you PTAL since I think you are more familiar?
/assign @patrickdillon

[kikisdeliveryservice]

also for ovirt

/assign @rgolangh

[rgolangh]

/assign @Gal-Zaidman

[hardys]

Seems like #2567 is related

[patrickdillon]

Seems like #2567 is related

+1 Looks like it should fix this.

[mriedmann]

Just for the record: Enabling the nodeip-configuration.service using systemctl enable --now nodeip-configuration.service followed by a systemctl daemon-reload && systemctl restart kubelet.service seems to fix the problem.
We will likely redeploy a 4.6 cluster in the near future, so we should be able to also test the mentioned fix after it is released.

I will report back and close the issue if it works without manual intervention.

[kikisdeliveryservice]

Closing issue reopen if required.