Disable two flaky tests (BEAM-8035, BEAM-9164) #11614
Conversation
|
@robertwb Thanks for approval, do we have a way of visualizing ignored tests? I'm a little afraid these test might get ignored for ever, which might be unfortunate. |
| @@ -242,6 +243,7 @@ public void close() {} | |||
| * <p>This test verifies that watermarks are correctly forwarded. | |||
| */ | |||
| @Test(timeout = 30_000) | |||
| @Ignore("https://issues.apache.org/jira/browse/BEAM-9164") | |||
There was a problem hiding this comment.
This one is apparently only flaky on Flink can we better exclude it manually only for Flink?
There was a problem hiding this comment.
This is not @Category(ValidatesRunner) test, looks like it is Flink-specific already?
There was a problem hiding this comment.
@iemejia can I merge this? Is it fine for you?
There was a problem hiding this comment.
IMHO disabling tests should be the last resort. Please inform the mailing list or at least the JIRA issue when you disabled tests. This may be causing regressions. For example, I changed logic around UnboundedSourceWrapper a couple days ago and was not aware of tests being disabled.
There was a problem hiding this comment.
@mxm I started a discussion thread here. Agree that we should have a better way of tracking flaky and disabled tests. There was also additional thread about precommit stability. The two tests disabled in this PR were causing practically permanent failure in precommits.
There was a problem hiding this comment.
Thanks for starting the discussion. I'll try to enable these tests again. The recent changes in UnboundedSourceWrapper should have improved the test stability.
There was a problem hiding this comment.
I'll run some tests here: #11679
|
I'm not sure we have more visibility into disabled tests than the jira entry (which shouldn't get closed until the tests are fixed and/or deemed unneeded and deleted. |
|
@robertwb Looks like we should introduce some measures to solve this (not sure which measures these should be), because like we are accumulating these ignored tests: Most of associated issues remain open, which is what I would expect without better visibility. I'll start a discussion thread on ML. |
Disables two flaky tests until resolution is found.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.