Support for @Tag in HapiTestEngine

[vtronkov]

Note: - #9510 was merged into this PR

Description:

• Adding support for @Tag annotation in HapiTestEngine

Related issue(s):

Fixes #9436

Notes for reviewer:
To filter by @Tag you need to:

• Annotate a test/tests with @Tag("TAG_NAME") annotation
• Open test-clients/build.gradle.kts file and add the following config:

tasks.register<Test>("hapiTestSlow") {
    testClassesDirs = sourceSets.main.get().output.classesDirs
    classpath = sourceSets.main.get().runtimeClasspath

    useJUnitPlatform {
        includeTags("slow")
        excludeTags("performance")
    }

    // Do not yet run things on the '--module-path'
    modularity.inferModulePath.set(false)
}

(change the includeTags and excludeTags according to your tests)

Run the hapiTestSlow gradle task from IntelliJ:

or execute ./gradlew hapiTestSlow in the console

Checklist

• Documented (Code comments, README, etc.)
• Tested (unit, integration, etc.)

[github-actions]

Node: HAPI Test Results

1 242 tests   772 ✔️  1h 11m 24s ⏱️
   165 suites  470 💤
   165 files        0 ❌

Results for commit 51cccc6.

♻️ This comment has been updated with latest results.

[github-actions]

Node: E2E Test Results

    1 files      1 suites   20m 1s ⏱️
310 tests 310 ✔️ 0 💤 0 ❌
332 runs  332 ✔️ 0 💤 0 ❌

Results for commit 51cccc6.

♻️ This comment has been updated with latest results.

[github-actions]

Node: Unit Test Results

    2 262 files      2 262 suites   1h 32m 28s ⏱️
118 254 tests 118 220 ✔️ 34 💤 0 ❌
126 513 runs  126 479 ✔️ 34 💤 0 ❌

Results for commit 51cccc6.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (develop@ea89f76). Click here to learn what that means.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #9490   +/-   ##
==========================================
  Coverage           ?   65.22%           
  Complexity         ?    29651           
==========================================
  Files              ?     3261           
  Lines              ?   124483           
  Branches           ?    12912           
==========================================
  Hits               ?    81194           
  Misses             ?    40215           
  Partials           ?     3074           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Node: Integration Test Results

280 tests   280 ✔️  32m 53s ⏱️
    5 suites      0 💤
    5 files        0 ❌

Results for commit 51cccc6.

♻️ This comment has been updated with latest results.

[vtronkov]

Okay, I played a bit with a randomly picked gradle task:

tasks.register<Test>("hammerTest") {
    testClassesDirs = sources.output.classesDirs
    classpath = sources.runtimeClasspath

    shouldRunAfter(tasks.test)

    useJUnitPlatform { includeTags("HAMMER") }
    maxHeapSize = "8g"
    jvmArgs("-XX:ActiveProcessorCount=16")
}

If I annotate EndToEndPackageRunner with @HapiTestSuite then create a test method in it:

@Tag("HAMMER")
@Tag("contract.traceability")
@TestFactory
@HapiTest
Collection<DynamicContainer> hammer() {
    return List.of(extractSpecsFromSuite(ContractCreateSuite::new));
}

I can see that out of the box, we are calling the HapiTestEngine and it looks like we have the tags support. However, immediately after that we call JupiterTestEngine and we execute the test the old way. This is because the Gradle task is using the jUnit instead of our custom implementation.

I think we should have a story to first have the Gradle task to use the new HapiTestEngine in place and then I can test our engine and see what we support out of the box and what we should add.

Maybe @jjohannes or @nathanklick work on that?(I see you guys worked on these gradle tasks)

What do you guys think? @netopyr

[jsync-swirlds]

The trial you ran is flawed.
Hammer tests already have specific logic for running them, and whole suite extract methods (like the hammer method) cannot be annotated HapiTest because they won't work correctly with that engine (the engine won't find any valid tests to run).

A better trial would involve adding tags to existing HapiTest methods and copying the hapiTest block in gradle to exclude/include tags.

Perhaps try this setup (after tagging a few individual tests with "slow" and "unstable"):

// Existing hapiTest task
tasks.register<Test>("hapiTest") {
    testClassesDirs = sourceSets.main.get().output.classesDirs
    classpath = sourceSets.main.get().runtimeClasspath

    useJUnitPlatform {
        excludeTags("slow", "unstable")
    }

    // Do not yet run things on the '--module-path'
    modularity.inferModulePath.set(false)
}

// New task for slow/unstable tests
tasks.register<Test>("hapiSlowTest") {
    testClassesDirs = sourceSets.main.get().output.classesDirs
    classpath = sourceSets.main.get().runtimeClasspath

    useJUnitPlatform {
        includeTags("slow", "unstable")
    }

    // Do not yet run things on the '--module-path'
    modularity.inferModulePath.set(false)
}

[vtronkov]

@jsync-swirlds I misled you a bit. I was not actually using the HAMMER task - I just copied it and created a new one(HAMMER2). Nonetheless thanks for pointing out the existing hapiTest task - it helped debugging the issue.
The main issue was that I didn't override the getTags method and the default one was called(which returns an empty set).

Now it should work 😄