Kuberhealthy OwnerRefInvalidNamespace Bug

[joshulyne]

We set ownerRef for all our khcheck pods to the kuberhealthy pod (since the kuberhealthy pod provisions/manages all these khcheck pods). However, for khchecks outside the kuberhealthy namespace (or whichever namespace kuberhealthy is running), we run into this OwnerRefInvalidNamespace error for kubernetes 1.20+ and the khcheck pods get automatically deleted:

per k8s 1.20 release notes:

Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.

• A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.

• A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]

[integrii]

It would be worth taking a look at the Kubernetes roadmap to see if cross-namespace checks are coming back in the future. For now, we probably have to drop owner references?

Cross-namespace owner references are disallowed by design.

[AshutoshNirkhe]

Hi @joshulyne @integrii I am testing out k8s 1.20 upgrade in sandbox cluster and encountered exactly this. Was about to file a bug and saw this one :)
I tried setting 'KH_POD_NAMESPACE' in external check def to point to Kuberhealty namespace but that didn't really help as it simply overwrites it with the namespace set in metadata of checker pod.
Do you think there is any easy way out for this as of now other than running all checks in same namespace as that of Kuberhealthy ?
I see this marked for 2.6/2.7. If we don't support this early, issue is that we won't be able to allow our users to maintain/run their checks in their own namespaces. Thanks in advance!

[integrii]

It seems like we might need to quickly make a new build that drops owner references when the pod is being built in a different namespace. Unfortunately, it seems like owner references are not going to be supported by Kubernetes. I have raised this an existing issue in the kubernetes project (see reference above).

[integrii]

@deads2k wrote the documentation on this one in the Kubernetes project and mentioned that in any future implementation, it needs to be "double opt-in" by both namespace owners somehow.

[integrii]

Looks like we need to check if the checker pod is going into the same namespace as the kuberhealthy pod spawning it before adding the owner reference to it here