Readiness probe fails because no cgroup scope found

[harche]

What happened?

On a few instances we are observing the readiness probes fail with following error,

"2023-02-25T06:59:57Z" level=error msg="exec failed: unable to start container process: error adding pid 20296 to cgroups: failed to write 20296: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1
event happened 118 times, something is wrong: ns/openshift-ovn-kubernetes pod/ovnkube-master-rmw2n node/ci-op-ywn42842-e5103-znj25-master-2 - reason/Unhealthy (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = command error: time="2023-02-25T07:09:57Z" level=error msg="exec failed: unable to start container process: error adding pid 33082 to cgroups: failed to write 33082: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1
event happened 186 times, something is wrong: ns/openshift-ovn-kubernetes pod/ovnkube-master-rmw2n node/ci-op-ywn42842-e5103-znj25-master-2 - reason/Unhealthy (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = command error: time="2023-02-25T07:19:57Z" level=error msg="exec failed: unable to start container process: error adding pid 44463 to cgroups: failed to write 44463: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1
event happened 220 times, something is wrong: ns/openshift-ovn-kubernetes pod/ovnkube-master-rmw2n node/ci-op-ywn42842-e5103-znj25-master-2 - reason/Unhealthy (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = command error: time="2023-02-25T07:24:57Z" level=error msg="exec failed: unable to start container process: error adding pid 50149 to cgroups: failed to write 50149: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1
event happened 254 times, something is wrong: ns/openshift-ovn-kubernetes pod/ovnkube-master-rmw2n node/ci-op-ywn42842-e5103-znj25-master-2 - reason/Unhealthy (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = command error: time="2023-02-25T07:29:57Z" level=error msg="exec failed: unable to start container process: error adding pid 55876 to cgroups: failed to write 55876: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1
event happened 323 times, something is wrong: ns/openshift-ovn-kubernetes pod/ovnkube-master-rmw2n node/ci-op-ywn42842-e5103-znj25-master-2 - reason/Unhealthy (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = command error: time="2023-02-25T07:39:57Z" level=error msg="exec failed: unable to start container process: error adding pid 68698 to cgroups: failed to write 68698: open /sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f246da_32e1_4f3c_818f_843c17a6e9ca.slice/crio-c64c0d7a76d118f60aa14b880ff15896b8a6753268adfbfdb6fe144074f84b54.scope/cgroup.procs: no such file or directory"
, stdout: , stderr: , exit code -1

It seems like the doProbe() keeps getting outdated pod status which prevents it from exiting.

I think the ongoing effort to use pod worker to fetch status should fix this issue. There is WIP PR which should update the code for probe worker as well.

cc @rphillips @smarterclayton @bobbypage

/sig node

What did you expect to happen?

Prober worker should get status update from pod worker.

How can we reproduce it (as minimally and precisely as possible)?

Happens occasionally

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[harche]

/assign

[SergeyKanzhelev]

/triage accepted
/priority backlog

may be important for graceful termination

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.