[CI/CD][flaky test] fileconsumer test hits timeout on actuated ARM runner #32391
Comments
crobert-1
added
needs triage
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Unfortunately increasing the timeout has not fixed all frequencies of this, just hit another that timed out at 10 minutes. Reopening instead of creating a new issue to keep context together, but let me know if a new issue would be preferable. |
**Description:** Respect context cancellation in emittest.Sink to prevent deadlock if the emit callback is invoked during manager shutdown. **Link to tracking Issue:** <Issue number if applicable> Related to #32391. Increasing the poll interval in TestCopyTruncate to 5s reliably causes the "timeout: expected: 2700, actual: ..." failure in the linked issue; the test never terminates, and this blocks other tests from running. **Testing:** N/A, fixing test code **Documentation:** N/A
|
I assume this type of timeouts is due to rate-limiting of Docker artifacts when pulling from Docker hub. Actuated offers a way to run a docker registry local to the runner that will limit the impact greatly: https://docs.actuated.dev/tasks/registry-mirror/ |
**Description:** Respect context cancellation in emittest.Sink to prevent deadlock if the emit callback is invoked during manager shutdown. **Link to tracking Issue:** <Issue number if applicable> Related to open-telemetry#32391. Increasing the poll interval in TestCopyTruncate to 5s reliably causes the "timeout: expected: 2700, actual: ..." failure in the linked issue; the test never terminates, and this blocks other tests from running. **Testing:** N/A, fixing test code **Documentation:** N/A
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Hosted runners use a token from an agreement between Docker and Azure to log into the Docker Hub and avoid rate limits. The alternative for self hosted runners is to use the pull through mirror which is preinstalled or to authenticate with your own token prior to accessing the Hub. We do support via Slack not via GitHub so haven't been aware of any recent issues for OTEL. |
**Description:** Respect context cancellation in emittest.Sink to prevent deadlock if the emit callback is invoked during manager shutdown. **Link to tracking Issue:** <Issue number if applicable> Related to open-telemetry#32391. Increasing the poll interval in TestCopyTruncate to 5s reliably causes the "timeout: expected: 2700, actual: ..." failure in the linked issue; the test never terminates, and this blocks other tests from running. **Testing:** N/A, fixing test code **Documentation:** N/A
Component(s)
pkg/stanza
Describe the issue you're reporting
CI/CD link
As with #32298, it looks like runs on actuated arm builds simply take a little longer sometimes, hitting timeout errors. It may be worthwhile to investigate ways to make the test suite finish quicker. Note: The specific test that this is failing on is likely irrelevant to the actual issue, it was simply the test being executed when the timeout occurred.
Failure output:
The text was updated successfully, but these errors were encountered: