Managing test flakes #637
Merged
Managing test flakes #637
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jglick
commented
May 8, 2023
| @@ -200,6 +201,7 @@ public void testContinuedInterruptionOfClassReferenceCreation(ChannelRunner chan | |||
| }); | |||
| } | |||
|
|
|||
| @Disabled("TODO first call consistently flakes: Remote call on north failed") | |||
There was a problem hiding this comment.
| @@ -50,6 +51,7 @@ public class ClassRemotingTest { | |||
| static final String TESTCALLABLE_TRANSFORMED_CLASSNAME = "hudson.rem0ting.TestCallable"; | |||
| static final String TESTLINKAGE_TRANSFORMED_CLASSNAME = "hudson.rem0ting.TestLinkage"; | |||
|
|
|||
| @Disabled("TODO flakes: Artificial testing interrupt.") | |||
There was a problem hiding this comment.
Not sure what is going on with these. Interference from
(in a different source!) suggests something structurally wrong with the tests.| @@ -36,7 +35,6 @@ | |||
| /** | |||
| * @author Kohsuke Kawaguchi | |||
| */ | |||
| @Ignore | |||
There was a problem hiding this comment.
Disabled in #387, yet consistently passing for me, and this annotation does not even work since #593, so trying to run it again. Whatever it is that it is testing; from
remoting/src/test/java/hudson/remoting/PrefetchTest.java
Lines 46 to 47 in b5eba82
|
|
||
| @Ignore("This is not a test just a benchmark and is here for ease of running") | ||
| @Disabled("This is not a test just a benchmark and is here for ease of running") |
| @@ -114,7 +116,7 @@ public void ready(boolean accept, boolean connect, boolean read, boolean write) | |||
| } catch (IOException e) { | |||
| // ignore | |||
| } | |||
| hub.hub().addInterestAccept(key.get()); | |||
| h.addInterestAccept(key.get()); | |||
There was a problem hiding this comment.
Trying to make sure the same hub is being used.
| @@ -203,14 +205,16 @@ public void onClosedChannel(ClosedChannelException e) { | |||
| } | |||
| } | |||
|
|
|||
| @Disabled("TODO flakes: Read timed out; or expected java.net.SocketTimeoutException to be thrown, but nothing was thrown") | |||
There was a problem hiding this comment.
Judging by 247f15b this test is just inherently flaky.
java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
at java.base/java.io.Reader.read(Reader.java:229)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1432)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1411)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1208)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1022)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:2839)
at org.jenkinsci.remoting.protocol.IOHubTest.noReadyCallbackIfInterestRemoved(IOHubTest.java:256)
or
java.lang.AssertionError: expected java.net.SocketTimeoutException to be thrown, but nothing was thrown
at org.junit.Assert.assertThrows(Assert.java:1028)
at org.junit.Assert.assertThrows(Assert.java:981)
at org.jenkinsci.remoting.protocol.IOHubTest.noReadyCallbackIfInterestRemoved(IOHubTest.java:267)
| @@ -447,7 +448,6 @@ public void socketBasicBackToBackWithAckSSLEngineHeaders() throws Exception { | |||
| } | |||
|
|
|||
| @Test | |||
| @Repeat(16) | |||
There was a problem hiding this comment.
Otherwise can get timeouts. Anyway, why did we need to run these 16 times to “prove” that the code was correct?
| @@ -750,6 +747,7 @@ public void socketChannelFullProtocolNIO_serverRejects() throws Exception { | |||
| assertThat(se.getCause(), instanceOf(ConnectionRefusalException.class)); | |||
| } | |||
|
|
|||
| @Disabled("TODO flake: ConnectionRefusal Incorrect acknowledgement received, expected 0x000341436b got 0x0000000000") | |||
There was a problem hiding this comment.
The exception is expected, but sometimes gets thrown too early (in release). Not sure what test should be doing.
Vlatombe
approved these changes
Jun 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some failures observed in CI, but mainly just running
mvn test -DforkCount=.45C(about five concurrent tests on my machine) over and over. Surprisingly hard to get a clean run; #142 masks poor tests and we do not yet have the infrastructure to track flakes (maybe Launchable could).