Error on reconciliation of odh-model-controller

[vaibhavjainwiz]

Below issue is coming on reconciliation of odh-model-controller.

2023-09-20T00:50:14Z	ERROR	Reconciler error	{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"caikit-example-isvc","namespace":"kserve-demo"}, "namespace": "kserve-demo", "name": "caikit-example-isvc", "reconcileID": "f399dd1f-1ee1-45c2-badc-c37507bc51fe", "error": "servicemonitors.monitoring.coreos.com \"caikit-example-isvc-metrics\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

Currently this error is not visible in logs because error object is not send back. We capture the error and then just ignore it. In below line we are capturing err but never return it to calling function.
https://github.com/opendatahub-io/odh-model-controller/blob/main/controllers/inferenceservice_controller.go#L99

[vaibhavjainwiz]

depends on changes of #84

[heyselbi]

@rpancham will retest it. Once test passes, @vaibhavjainwiz please make sure it is in RHODS 2.4.

[bdattoma]

@vaibhavjainwiz how can I try to reproduce the issue?

[vaibhavjainwiz]

These error will not be visible in logs but you could see the impacts it on ServiceMeshMemberRole.
If you delete the Namespace from ServiceMeshMemberRole in running application then ideally it would add back by operator but beacause of this error it would not added back.

[bdattoma]

These error will not be visible in logs but you could see the impacts it on ServiceMeshMemberRole. If you delete the Namespace from ServiceMeshMemberRole in running application then ideally it would add back by operator but beacause of this error it would not added back.

I tried, and don't see any error in odh-model-controller pod.
However, the ServiceMeshMemberRoll doesn't get reconcilied automatically.

[israel-hdez]

I updated the label to rhods-2.5, because the fix is missing on 2.4.
Once there is a build for 2.5 I'll re-check and close if I the fix is correctly in place.

[bdattoma]

I updated the label to rhods-2.5, because the fix is missing on 2.4. Once there is a build for 2.5 I'll re-check and close if I the fix is correctly in place.

sounds good, thank you!

[israel-hdez]

The issue seems to be gone in 2.5-RC
Closing.