8339510: [TestBug] Convert system tests to JUnit 5 #1569

andy-goryachev-oracle · 2024-09-16T17:57:07Z

Converting system tests to junit5.

Please see migration notes:
https://github.com/andy-goryachev-oracle/Test/blob/main/doc/Tests/JUnit5Migration.md

Notes:

I see shutdown timeout on linux, this will be addressed by JDK-8340403

QPathTest > executionError FAILED
    org.opentest4j.AssertionFailedError: Exceeded timeout limit of 10000 msec
        at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
        at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
        at app//test.util.Util.runAndWait(Util.java:156)
        at app//test.util.Util.runAndWait(Util.java:127)
        at app//test.util.Util.shutdown(Util.java:365)
        at app//test.com.sun.marlin.QPathTest.teardownOnce(QPathTest.java:146)

This test might fail intermittently (?) on linux only:

SetSceneScalingTest > testShowAndSetScene() FAILED
    org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
        at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
        at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
        at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:35)
        at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:179)
        at app//test.robot.javafx.stage.SetSceneScalingTest$TestShowSetSceneApp.test(SetSceneScalingTest.java:129)
        at app//test.robot.javafx.stage.SetSceneScalingTest$TestApp.runTest(SetSceneScalingTest.java:89)
        at app//test.robot.javafx.stage.SetSceneScalingTest.testShowAndSetScene(SetSceneScalingTest.java:193)

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

JDK-8339510: [TestBug] Convert system tests to JUnit 5 (Enhancement - P4)

Reviewers

Kevin Rushforth (@kevinrushforth - Reviewer)
Lukasz Kostyra (@lukostyra - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jfx.git pull/1569/head:pull/1569
$ git checkout pull/1569

Update a local copy of the PR:
$ git checkout pull/1569
$ git pull https://git.openjdk.org/jfx.git pull/1569/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1569

View PR using the GUI difftool:
$ git pr show -t 1569

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jfx/pull/1569.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2024-09-16T17:57:34Z

👋 Welcome back angorya! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-09-16T17:58:35Z

@andy-goryachev-oracle This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8339510: [TestBug] Convert system tests to JUnit 5

Reviewed-by: kcr, lkostyra

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 5 new commits pushed to the master branch:

4173840: 8339512: [TestBug] Convert graphics tests to JUnit 5
5bec3f8: 8339511: [TestBug] Convert Non parametrized base tests to JUnit 5
5171753: 8338468: [TestBug] Convert controls tests to JUnit 5
addf085: 8340405: JavaFX shutdown hook can hang preventing app from exiting
bc5adfa: 8340208: Additional WebKit 619.1 fixes from WebKitGTK 2.44.4

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

andy-goryachev-oracle · 2024-09-17T20:59:27Z

/reviewers 2

openjdk · 2024-09-17T20:59:50Z

@andy-goryachev-oracle
The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

kevinrushforth

I spot checked many of the classes, and most of the changes look reasonable.

One question, though: Why did you replace the per-test timeout with an in-method annotation in many places (especially in the early classes I looked at)? There is a direct replacement in JUnit 5 for the per-test timeout, which seems a more straightforward change. I also think an explicit timeout on the test (or class, if all @Test methods have the same timeout) will work more consistently with my in-progress fix to add a global timeout.

tests/system/src/test/java/test/com/sun/javafx/application/ConcurrentStartupTest.java

tests/system/src/test/java/test/com/sun/javafx/application/InitializeJavaFXLaunch1Test.java

tests/system/src/test/java/test/util/Util.java

kevinrushforth · 2024-09-18T00:20:39Z

Testing recommendation: do a test run with gradle -PUNSTABLE_TEST=true so that all the tests that aren't disabled are run.

andy-goryachev-oracle · 2024-09-19T21:42:19Z

Here is a patch to remove unused imports.

there were more. I don't have this warning enabled in Eclipse because, if enabled, I see 410+ such warnings.

but thank you for noticing - I did forget to optimize imports in a few places.

kevinrushforth

The changes you made look good. I still see some unexpected test failures in the (unstable) monocle tests suite. I ran the following command:

gradle --continue --info -PTEST_ONLY=true -PFULL_TEST=true -PUSE_ROBOT=true -PUNSTABLE_TEST=true :systemTests:test --tests "test.robot.com.sun.glass.ui.monocle*"

On master, I get:

1773 tests : 56 failures : 495 ignored

With this PR, I get:

1326 tests : 422 failures: 12 ignored

I note that we are actually running more tests than before: 1773-495=1278 vs 1326-12=1314

As for the failures, since this is an unstable set of tests, the ones that timeout aren't a concern; which ones fail may vary slightly from run to run so a small difference is OK. In this case, the number of additional failures is much higher than I would expect: 56 vs 422.

The ones that fail with an exception other than a timeout suggest a problem with the conversion of parameterized tests. See InputDevicePropertyTest, for one example. All 45 tests pass with master and all 45 fail with a "Toolkit not initialized" with this patch.

kevinrushforth · 2024-09-20T14:52:26Z

tests/system/src/test/java/test/javafx/scene/Snapshot1Test.java

-        }, null);
+            // Should throw IllegalStateException
+            tmpScene.snapshot(p -> {
+                throw new RuntimeException("Should never get here");


In that case, throw new AssertionError seems the most equivalent change, in that it preserves the behavior of throwing an Error rather than a RuntimeException -- typically an Error is better for a case of "can't get here". I'll admit that it doesn't matter in this particular test, so it's just a suggestion. Feel free to ignore it.

kevinrushforth · 2024-09-20T14:55:25Z

tests/system/src/test/java/test/javafx/scene/UIRenderSnapToPixelTest.java

+                Assertions.assertEquals(0, ((sp.snappedTopInset() * scale) + epsilon) % 1, 0.0001, "Top inset not snapped to pixel");
+                Assertions.assertEquals(0, ((sp.snappedBottomInset() * scale) + epsilon) % 1, 0.0001, "Bottom inset not snapped to pixel");
+                Assertions.assertEquals(0, ((sp.snappedLeftInset() * scale) + epsilon) % 1, 0.0001, "Left inset not snapped to pixel");
+                Assertions.assertEquals(0, ((sp.snappedRightInset() * scale) + epsilon) % 1, 0.0001, "Right inset not snapped to pixel");


Yeah, don't worry about the line length, that was just an observation.

I do prefer having the class qualifier over static import (though static imports make more sense in the tests).

So do I in most cases. For tests, I prefer static imports of the methods in Assertions and Assumptions, which is also the pattern recommended by the JUnit docs.

andy-goryachev-oracle · 2024-09-20T16:41:28Z

I still see some unexpected test failures in the (unstable) monocle tests suite.

Speaking of that, I noticed that the number of failed tests varies between invocations on the same machine. For example with RotateTest on macOS running the command
gradle -PFULL_TEST=true -PUSE_ROBOT=true :systemTests:cleanTest :systemTests:test -PUNSTABLE_TEST=true --tests test.robot.com.sun.glass.ui.monocle.RotateTest I get 5 failures, 10 failures, 6 failures, 7...

Examples:

RotateTest > testRotateLeftBigSteps(TestTouchDevice) > [8] NTrigDevice FAILED
    com.sun.glass.ui.monocle.TestLogShim$TestLogAssertion: Timed out after 3005ms waiting for 'TouchPoint: MOVED 285, 337'
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLog(TestLogShim.java:211)
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLogContaining(TestLogShim.java:237)
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLogContaining(TestLogShim.java:246)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.Rotate(RotateTest.java:154)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.Rotate(RotateTest.java:221)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.testRotateLeftBigSteps(RotateTest.java:316)

RotateTest > testRotateLeftFromMinus140Degrees(TestTouchDevice) > [8] NTrigDevice FAILED
    com.sun.glass.ui.monocle.TestLogShim$TestLogAssertion: Timed out after 3005ms waiting for 'TouchPoint: MOVED 640, 760'
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLog(TestLogShim.java:211)
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLogContaining(TestLogShim.java:237)
        at app/javafx.graphics@24-internal/com.sun.glass.ui.monocle.TestLogShim.waitForLogContaining(TestLogShim.java:246)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.Rotate(RotateTest.java:154)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.Rotate(RotateTest.java:217)
        at app//test.robot.com.sun.glass.ui.monocle.RotateTest.testRotateLeftFromMinus140Degrees(RotateTest.java:407)

3 second timeout seems to be sufficient for the purpose, and I don't see any output captured by the TestLogShim. I haven't looked into the actual tests for long since it's out of scope, but maybe I am doing something wrong.

andy-goryachev-oracle · 2024-09-20T17:11:18Z

Side note: TeshLogShim.log appears to be improperly synchronized in getLog(), countLog(), checkLog(), checkLogContaining(), but even when synchronization is fixed the tests still randomly fail...

kevinrushforth · 2024-09-20T20:32:35Z

The latest change look correct and fix most of the failures where the entire test class was failing. I still see a discrepancy between number of tests run before / after and there are more failures now.

On master, I get:

1773 tests : 56 failures : 495 ignored

With the latest version of this PR, I get:

1326 tests : 101 failures: 78 ignored

So we are now running fewer tests than before: 1773-495=1278 vs 1326-78=1248. As for the failures, I ran it twice and got 102 failures the first time and 101 the second time. So there is likely some failure that isn't due to test instability.

andy-goryachev-oracle · 2024-09-20T23:53:32Z

so on macOS, running this command

gradle --continue -PFULL_TEST=true -PUSE_ROBOT=true :systemTests:cleanTest :systemTests:test -PUNSTABLE_TEST=true

I got 2274 tests completed, 51 failed, 130 skipped

I'll run the same command against master next week.

edit:
with the master, I got 2727 tests completed, 49 failed, 553 skipped = 2174
as opposed to this PR 2274 tests completed, 51 failed, 130 skipped = 2144

gives a difference of 30 tests lost somewhere. will investigate and report.

kevinrushforth · 2024-09-21T14:34:21Z

Not sure why I'm seeing something different then. I tried running it on a second macOS system today (macOS 13.x Intel) and got similar results to my earlier runs. I ran the following command to just run the monocle tests:

gradle --continue --info -PTEST_ONLY=true -PFULL_TEST=true -PUSE_ROBOT=true -PUNSTABLE_TEST=true :systemTests:test --tests "test.robot.com.sun.glass.ui.monocle*"

master

1773 tests : 56 failures : 495 ignored -- tests run: 1773-495=1278

this PR

1326 tests : 106 failures: 78 ignored -- tests run: 1326-78=1248

lukostyra · 2024-09-23T13:19:36Z

tests/system/src/test/java/test/javafx/scene/Snapshot1Test.java

@@ -255,7 +262,7 @@ public void testBadSceneCallback2() {
            Callback cb = (Callback<String, Integer>) param -> {
                // Should not get here
                latch.countDown();
-                throw new AssertionFailedError("Should never get here");
+                throw new AssertionError("Should never get here");


Should probably be fail

won't compile with fail(), needs an exception

lukostyra · 2024-09-23T13:19:44Z

tests/system/src/test/java/test/javafx/scene/Snapshot1Test.java

@@ -381,7 +384,7 @@ public void testBadNodeCallback2() {
            Callback cb = (Callback<String, Integer>) param -> {
                // Should not get here
                latch.countDown();
-                throw new AssertionFailedError("Should never get here");
+                throw new AssertionError("Should never get here");


Should probably be fail

won't compile with fail(), needs an exception

andy-goryachev-oracle · 2024-09-23T19:41:21Z

The latest commit fixes the number of tests issue. Note: we should not look at the number of tests reported by gradle at the end of the test run, instead we should look at the HTML report.

master: tests=2728, failures=49, ignored=554 (tests - ignored = 2174)
this PR: tests=2281, failures=96, ignored=107 (test - ignored = 2174)

The actual number of failed tests varies from run to run, and also, being "unstable", differs between master and PR. For example:

FuzzyTapTest: fails with timeout (TO) vs. passing in master
SingleTouchTest: passing vs. failing with TO in master
TouchButtonTest pass vs failures with TO in master

I also noticed a bug in USKeyboardTest, see https://bugs.openjdk.org/browse/JDK-8340693

kevinrushforth · 2024-09-23T20:32:01Z

I ran the latest and I now see the missing 30 tests being run.

gradle --continue --info -PTEST_ONLY=true -PFULL_TEST=true -PUSE_ROBOT=true -PUNSTABLE_TEST=true \
    :systemTests:test --tests "test.robot.com.sun.glass.ui.monocle*"

master

1773 tests : 56 failures : 495 ignored -- tests run: 1773-495=1278

this PR

run 1: 1326 tests : 72 failures: 48 ignored -- tests run: 1326-48=1278
run 2: 1326 tests : 37 failures: 48 ignored -- tests run: 1326-48=1278

All good.

lukostyra

LGTM

andy-goryachev-oracle · 2024-09-24T14:22:21Z

/integrate

openjdk · 2024-09-24T14:24:56Z

Going to push as commit 21601f8.
Since your change was applied there have been 5 commits pushed to the master branch:

4173840: 8339512: [TestBug] Convert graphics tests to JUnit 5
5bec3f8: 8339511: [TestBug] Convert Non parametrized base tests to JUnit 5
5171753: 8338468: [TestBug] Convert controls tests to JUnit 5
addf085: 8340405: JavaFX shutdown hook can hang preventing app from exiting
bc5adfa: 8340208: Additional WebKit 619.1 fixes from WebKitGTK 2.44.4

Your commit was automatically rebased without conflicts.

openjdk · 2024-09-24T14:25:09Z

@andy-goryachev-oracle Pushed as commit 21601f8.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

part 1

3f2d51f

andy-goryachev-oracle added 14 commits September 16, 2024 11:25

part 2

4c0c6a7

part 3

03d7630

part 4

dadf3c5

part 5

dcd18cf

part 6

c83d469

part 7

d8d9e72

part 8 tricky

1ac8f91

cleanup

08f158a

Merge remote-tracking branch 'origin/master' into 8339510.junit5.system

282996b

part 9 monocle

2fb7491

part 10 swing

5c9678f

part 11 scene

9770d0e

fixes

a0fe8be

part 12 last

87c048a

kevinrushforth self-requested a review September 17, 2024 22:35

fixes 1

996c82a

kevinrushforth reviewed Sep 18, 2024

View reviewed changes

kevinrushforth self-requested a review September 18, 2024 00:20

kevinrushforth mentioned this pull request Sep 18, 2024

8338468: [TestBug] Convert controls tests to JUnit 5 #1561

Closed

3 tasks

andy-goryachev-oracle added 3 commits September 18, 2024 08:04

fixes 2

b692d57

fixes 3

16ececf

fixes 4

96cd197

andy-goryachev-oracle marked this pull request as ready for review September 18, 2024 20:05

openjdk bot added the rfr Ready for review label Sep 18, 2024

kevinrushforth mentioned this pull request Sep 19, 2024

8328629: JUnit test without a timeout value can hang indefinitely #1575

Closed

3 tasks

unused imports

2542609

review comments

d09dfb6

openjdk bot removed the rfr Ready for review label Sep 19, 2024

rotate test

7349f48

openjdk bot added the rfr Ready for review label Sep 19, 2024

kevinrushforth reviewed Sep 20, 2024

View reviewed changes

andy-goryachev-oracle added 2 commits September 20, 2024 11:25

more fixes

5905fb4

Merge remote-tracking branch 'origin/master' into 8339510.junit5.system

777620e

lukostyra reviewed Sep 23, 2024

View reviewed changes

30 more

bbf4034

kevinrushforth approved these changes Sep 23, 2024

View reviewed changes

lukostyra approved these changes Sep 24, 2024

View reviewed changes

openjdk bot added the ready Ready to be integrated label Sep 24, 2024

openjdk bot added the integrated Pull request has been integrated label Sep 24, 2024

openjdk bot closed this Sep 24, 2024

openjdk bot removed ready Ready to be integrated rfr Ready for review labels Sep 24, 2024

andy-goryachev-oracle deleted the 8339510.junit5.system branch September 24, 2024 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8339510: [TestBug] Convert system tests to JUnit 5 #1569

8339510: [TestBug] Convert system tests to JUnit 5 #1569

andy-goryachev-oracle commented Sep 16, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Sep 16, 2024

openjdk bot commented Sep 16, 2024 •

edited

Loading

andy-goryachev-oracle commented Sep 17, 2024

openjdk bot commented Sep 17, 2024

kevinrushforth left a comment

kevinrushforth commented Sep 18, 2024

andy-goryachev-oracle commented Sep 19, 2024

kevinrushforth left a comment

kevinrushforth Sep 20, 2024

kevinrushforth Sep 20, 2024

andy-goryachev-oracle commented Sep 20, 2024

andy-goryachev-oracle commented Sep 20, 2024

kevinrushforth commented Sep 20, 2024

andy-goryachev-oracle commented Sep 20, 2024 •

edited

Loading

kevinrushforth commented Sep 21, 2024

lukostyra Sep 23, 2024

andy-goryachev-oracle Sep 23, 2024

lukostyra Sep 23, 2024

andy-goryachev-oracle Sep 23, 2024

andy-goryachev-oracle commented Sep 23, 2024

kevinrushforth commented Sep 23, 2024

lukostyra left a comment

andy-goryachev-oracle commented Sep 24, 2024

openjdk bot commented Sep 24, 2024

openjdk bot commented Sep 24, 2024

8339510: [TestBug] Convert system tests to JUnit 5 #1569

8339510: [TestBug] Convert system tests to JUnit 5 #1569

Conversation

andy-goryachev-oracle commented Sep 16, 2024 • edited by openjdk bot Loading

Notes:

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Sep 16, 2024

openjdk bot commented Sep 16, 2024 • edited Loading

andy-goryachev-oracle commented Sep 17, 2024

openjdk bot commented Sep 17, 2024

kevinrushforth left a comment

Choose a reason for hiding this comment

kevinrushforth commented Sep 18, 2024

andy-goryachev-oracle commented Sep 19, 2024

kevinrushforth left a comment

Choose a reason for hiding this comment

kevinrushforth Sep 20, 2024

Choose a reason for hiding this comment

kevinrushforth Sep 20, 2024

Choose a reason for hiding this comment

andy-goryachev-oracle commented Sep 20, 2024

andy-goryachev-oracle commented Sep 20, 2024

kevinrushforth commented Sep 20, 2024

andy-goryachev-oracle commented Sep 20, 2024 • edited Loading

kevinrushforth commented Sep 21, 2024

master

this PR

lukostyra Sep 23, 2024

Choose a reason for hiding this comment

andy-goryachev-oracle Sep 23, 2024

Choose a reason for hiding this comment

lukostyra Sep 23, 2024

Choose a reason for hiding this comment

andy-goryachev-oracle Sep 23, 2024

Choose a reason for hiding this comment

andy-goryachev-oracle commented Sep 23, 2024

kevinrushforth commented Sep 23, 2024

master

this PR

lukostyra left a comment

Choose a reason for hiding this comment

andy-goryachev-oracle commented Sep 24, 2024

openjdk bot commented Sep 24, 2024

openjdk bot commented Sep 24, 2024

andy-goryachev-oracle commented Sep 16, 2024 •

edited by openjdk bot

Loading

openjdk bot commented Sep 16, 2024 •

edited

Loading

andy-goryachev-oracle commented Sep 20, 2024 •

edited

Loading