TRT-1445: Monitor test hangs during interrupt handling

[xueqzhan]

This problem is caused by sending to a channel (junitch) that is full. That blocked the code from finishing up.

The junitch was created with a fix limit (2*len(monitortest) in the case of CollectData). It was not expected to create more than 2 junit per monitortest. But flake case is added, one monitortest could potentially add three junit (one created by the monitor test, one for a failure and one for a success) to the channel. The channel is not processed until all monitor tests finished collecting data. Therefore, some monitortest got stuck with a full junitch and blocked.

[openshift-ci-robot]

@xueqzhan: This pull request references TRT-1445 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.16.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@xueqzhan: This pull request references TRT-1445 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.16.0" version, but no target version was set.

In response to this:

This problem is caused by sending to a channel (junitch) that is full. That blocked the code from finishing up.

The junitch was created with a fix limit (2*len(monitortest) in the case of CollectData). It was not expected to create more than 2 junit per monitortest. But flake case is added, one monitortest could potentially add three junit (one created by the monitor test, one for a failure and one for a success) to the channel. The channel is not processed until all monitor tests finished collecting data. Therefore, some monitortest got stuck with a full junitch and blocked.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[neisw]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw, xueqzhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/OWNERS [neisw,xueqzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD ddba0ac and 2 for PR HEAD 729aab4 in total

[jluhrsen]

/retest

[jluhrsen]

/retest

[openshift-ci]

@xueqzhan: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node-serial	729aab4	link	false	/test e2e-aws-ovn-single-node-serial
ci/prow/e2e-aws-ovn-single-node-upgrade	729aab4	link	false	/test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-enterprise-tests-container-v4.16.0-202402090739.p0.g53c8fcb.assembly.stream.el8 for distgit openshift-enterprise-tests.
All builds following this will include this PR.