LUCENE-10531: Add @RequiresGUI test group for GUI tests #893

mocobeta · 2022-05-16T09:24:42Z

Mark TestScripts.testLukeCanBeLaunched as @RequiresGUI and add an Actions workflow for distribution tests that runs all distribution tests including GUI tests. The workflow should run on all major OSs: Linux, MacOS, and Windows.

On mandatory test runs:

./gradlew -p lucene/distribution.tests/ test -Dtests.gui=false
:lucene:distribution.tests:test (SUCCESS): 8 test(s), 1 skipped  // TestScripts is skipped.

On test runs where tests.gui sysprop is set to true:

./gradlew -p lucene/distribution.tests/ test -Dtests.gui=true
:lucene:distribution.tests:test (SUCCESS): 8 test(s)

On CI build, tests.gui sysprop is forced to set true to run GUI tests.

mocobeta · 2022-05-16T09:34:25Z

...stribution.tests/src/test/org/apache/lucene/distribution/AbstractLuceneDistributionTest.java

+  @Inherited
+  @Retention(RetentionPolicy.RUNTIME)
+  @TestGroup(enabled = false, sysProperty = SYSPROP_NIGHTLY)
+  public @interface Nightly {}


Off-the-shelf com.carrotsearch.randomizedtesting.annotations.Nightly would be okay (tests.nightly system property is used for the test group as default), but I re-defined Nightly annotation here as LuceneTestCase does so.

I would rename the annotation "RequiresGui" or something like this. It really is not a nightly test - just one that has additional requirements/ side effects.

Right... but I think if we introduce a new test group and system property, Jenkins and smoke tester configurations also need to be tweaked?

Thanks for cleaning up those workflow files, @mocobeta ! I don't haven that much experience with github actions so it's definitely cruft that accumulated over time.

Adding a new test group itself (disabled by default) doesn't require any special actions. If we want to run it on nightly runs then yes - jenkins would have to be configured properly for this... Alternatively, the value of the property triggering the test group could be computed inside gradle scripts (much like Uwe recently modified the errorprone to run on CI servers). Does this sound better?

Thanks for your suggestion. I'd like to run this test on the Jenkis Server with other nightly tests, and also in the smoke tester before every release.
I'll look at the alternatives once I will have made it work on Windows VM (currently it works only on Unix-like shells, of course).

@dweiss there will be lots of retries to make it work on both Linux shell and Windows command (I don't have other ideas to test Actions); I think "Unsubscribe" this PR is recommended.

@dweiss I renamed the annotation (test group) as suggested, added tests.gui system property for the test group, and made the sysprop forcibly set to true on CI builds - does this change make sense?

I also corrected the pull request description to reflect the renaming.

mocobeta · 2022-05-16T10:00:06Z

The added workflow takes 3 minutes on ubuntu-latest, and it takes 5 minutes on macos-latest.

.github/workflows/distribution.yml

.github/workflows/hunspell.yml

mocobeta · 2022-05-16T14:48:34Z

This worked on Windows VM - looks fairly slow (the whole workflow takes 7.5 minutes) as compared to Ubuntu (3 minutes) though.

mocobeta · 2022-05-16T15:21:30Z

surprisingly, it worked on the first try... I'll open this and try to address #893 (comment)

mocobeta · 2022-05-16T17:03:14Z

lucene/distribution.tests/build.gradle

+      if (isCIBuild) {
+        // force to run GUI tests
+        args += ["-Dtests.gui=true"]
+      }


I simply use isCIBuild variable that was introduced in #890; this might be hacky?

mocobeta · 2022-05-16T18:24:26Z

while waiting for review... here's Randomizedtesting Duke, rough-and-tumble boy (based on the "Tumble Duke") - the latest addition to my fan-art collection

mocobeta · 2022-05-17T08:46:17Z

The previous CI run result looks good to me. I also enabled GUI tests in the smoke tester.

mocobeta · 2022-05-17T08:51:28Z

How about renaming the action's directory name .github/actions/yarn-caches/ to .github/actions/gradle-caches/? @dweiss

dweiss · 2022-05-17T08:53:29Z

up to you, entirely. :)

dweiss · 2022-05-17T08:54:07Z

The yarn-caches name is wrong - it's something else I was working on (!).

mocobeta · 2022-05-17T08:57:48Z

I'll update the directory name later.

This time, the test timed out on Windows... I think this could occasionally happen :/

dweiss · 2022-05-17T09:02:43Z

Increase the timeout, maybe? Windows boxes on github are slow.

mocobeta · 2022-05-17T09:05:09Z

I increased the timeout to 120 seconds.

mocobeta · 2022-05-17T09:54:30Z

sorry couldn't resist.

"Build Duke"

mocobeta · 2022-05-17T10:06:19Z

Passed all checks and I think we've done all I wanted to do here - we disabled the GUI test in the mandatory test runs, instead, we enabled it on nightly CI runs (Jenkins, GH Actions) and the smoke tester.

mocobeta · 2022-05-18T00:25:27Z

I'm merging this only to main - let me know if it's worth backporting.

dweiss · 2022-05-18T06:14:57Z

I'd apply this to 9x as well since it'll ease backports of other things/ decrease the potential of a conflict in the future?

mocobeta · 2022-05-18T06:57:20Z

Ok I'll backport it to the 9x branch.

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>

mocobeta · 2022-05-21T04:43:43Z

We increased the timeout to 120 seconds, but the test still occasionally fails on Windows. I opened #917 to disable it on windows for now.

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>

mocobeta added 2 commits May 16, 2022 18:15

mark TestScripts as Nightly test

99f7fc0

add Actions workflow for distribution tests

65ee6c4

mocobeta commented May 16, 2022

View reviewed changes

mocobeta added 2 commits May 16, 2022 18:39

run distribution tests on major OSs (linux,mac,windows)

cd53d59

exclude windows for now

2118fcf

dweiss reviewed May 16, 2022

View reviewed changes

.github/workflows/distribution.yml Outdated Show resolved Hide resolved

dweiss reviewed May 16, 2022

View reviewed changes

.github/workflows/distribution.yml Outdated Show resolved Hide resolved

dweiss reviewed May 16, 2022

View reviewed changes

.github/workflows/distribution.yml Outdated Show resolved Hide resolved

mocobeta added 2 commits May 16, 2022 19:17

fix restore-keys, comment out the chmod +x step

0176c98

fix cache restore-keys in hunspell workflow

4e4b817

mocobeta commented May 16, 2022

View reviewed changes

.github/workflows/hunspell.yml Outdated Show resolved Hide resolved

mocobeta added 5 commits May 16, 2022 19:37

remove grant (chmod +x) step from all workflows.

fe816fc

trigger distribution workflow for all paths.

8203868

test run on windows

83bc74a

retry on windows

9baa608

looks like double quotation is needed

6471c19

try to run on both unix-like and windows vm.

3ac88dc

mocobeta marked this pull request as ready for review May 16, 2022 15:21

mocobeta added 5 commits May 17, 2022 00:30

add comments

69afb57

rename @nightly to @RequiresGUI, add 'tests.gui' sysprop for testing.

95570e2

fix javadocs

1b94719

force to run gui tests in CI build

3be29fe

remove -Dtests.nightly from CI test

3a6e27b

mocobeta changed the title ~~LUCENE-10531: Run scripts test nightly only~~ LUCENE-10531: Run GUI tests on CI only May 16, 2022

mocobeta commented May 16, 2022

View reviewed changes

fix actions workflow for windows.

5dbd13b

mocobeta added 5 commits May 17, 2022 16:53

remove obsolete windows commands

d84f265

reuse gradle cache among workflows

7675ec5

move if clause to set 'tests.gui' sysprop to randomization.gradle

bbf15b4

Merge branch 'run-scripts-test-nightly' into run-scripts-test-nightly

3e9f95d

partially revert bbf15b4

0289e7d

mocobeta mentioned this pull request May 17, 2022

Run scripts test nightly (alternative) mocobeta/lucene#1

Merged

mocobeta added 2 commits May 17, 2022 17:31

Merge pull request #1 from dweiss/run-scripts-test-nightly

170b45f

enable gui tests in smoke tester

eeeedd1

mocobeta added 2 commits May 17, 2022 18:02

yarn-caches -> gradle-caches

32aedb6

increase timeout seconds to 120

2e834c6

dweiss approved these changes May 17, 2022

View reviewed changes

mocobeta merged commit b911d1d into apache:main May 18, 2022

mocobeta deleted the run-scripts-test-nightly branch May 18, 2022 00:26

mocobeta added a commit that referenced this pull request May 18, 2022

LUCENE-10531: Add @RequiresGUI test group for GUI tests (backport #893)

50e0b7f

msokolov pushed a commit to msokolov/lucene that referenced this pull request May 20, 2022

LUCENE-10531: Add @RequiresGUI test group for GUI tests (apache#893)

3984c73

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>

shaie pushed a commit to mdmarshmallow/lucene that referenced this pull request Jun 22, 2022

LUCENE-10531: Add @RequiresGUI test group for GUI tests (apache#893)

53513f6

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>

asfimport mentioned this pull request May 23, 2022

Mark testLukeCanBeLaunched @Nightly test and make a dedicated Github CI workflow for it [LUCENE-10531] #11567

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LUCENE-10531: Add @RequiresGUI test group for GUI tests #893

LUCENE-10531: Add @RequiresGUI test group for GUI tests #893

mocobeta commented May 16, 2022 •

edited

mocobeta May 16, 2022

dweiss May 16, 2022

mocobeta May 16, 2022

dweiss May 16, 2022

mocobeta May 16, 2022

mocobeta May 16, 2022 •

edited

mocobeta May 16, 2022

mocobeta May 16, 2022

mocobeta commented May 16, 2022

mocobeta commented May 16, 2022

mocobeta commented May 16, 2022

mocobeta May 16, 2022

mocobeta commented May 16, 2022 •

edited

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022

dweiss commented May 17, 2022

dweiss commented May 17, 2022 •

edited

mocobeta commented May 17, 2022

dweiss commented May 17, 2022

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022 •

edited

mocobeta commented May 18, 2022

dweiss commented May 18, 2022

mocobeta commented May 18, 2022

mocobeta commented May 21, 2022

LUCENE-10531: Add @RequiresGUI test group for GUI tests #893

LUCENE-10531: Add @RequiresGUI test group for GUI tests #893

Conversation

mocobeta commented May 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mocobeta May 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mocobeta commented May 16, 2022

mocobeta commented May 16, 2022

mocobeta commented May 16, 2022

Choose a reason for hiding this comment

mocobeta commented May 16, 2022 • edited

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022

dweiss commented May 17, 2022

dweiss commented May 17, 2022 • edited

mocobeta commented May 17, 2022

dweiss commented May 17, 2022

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022

mocobeta commented May 17, 2022 • edited

mocobeta commented May 18, 2022

dweiss commented May 18, 2022

mocobeta commented May 18, 2022

mocobeta commented May 21, 2022

mocobeta commented May 16, 2022 •

edited

mocobeta May 16, 2022 •

edited

mocobeta commented May 16, 2022 •

edited

dweiss commented May 17, 2022 •

edited

mocobeta commented May 17, 2022 •

edited