oc exec hangs indefinitely

[wozniakjan]

Description

The CI job deploys origin using cri-o as its container runtime and then it deploys aggregated logging stack. After successful deployment, it runs the logging PR tests where it sometimes hangs indefinitely on a command oc exec logging-fluentd-7hhgt -- date +%Z (the fluentd pod name can be different). It is possible to oc rsh to the pod and run any commands but running oc exec will hang for a command if it hanged previously at least once for any oc exec.

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_aggregated_logging_ansible_crio/7/console

Steps to reproduce the issue:

1. run the CI job at https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_aggregated_logging_ansible_crio/

Describe the results you received:
oc exec hangs indefinitelly

Describe the results you expected:
oc exec not to hang and return the output of the command

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

crio version 1.9.8-dev
commit: "adb9891e31983c41da502ccfc4a379db7cc5e98f"

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS - https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_aggregated_logging_ansible_crio/

The exact same test suite passes when openshift is using docker

[mrunalp]

Thanks! We are looking into this. Meanwhile, could you try oc exec -it and see if it works for you? (Just as a debugging point.)

[mrunalp]

#1443 has a fix for this. Thanks!