SummaryGeneratingListener delivered in getSummary() the double result in sum by failed and successful tests to found tests.

[mourav2222]

>>>> testPlanExecutionFinished...
>>>> test1() : ???
>>>> userCanSearchKeyword2() : ???
>>>> Human-readable test name : ???

Test run finished after 10309 ms
[         2 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         0 tests skipped         ]
[         6 tests started         ]
[         0 tests aborted         ]
[         2 tests successful      ]
[         4 tests failed          ]


BUILD SUCCESSFUL in 16s
3 actionable tasks: 2 executed, 1 up-to-date
18:35:12: Execution finished ':Tester.main()'.

dependencies {
    testImplementation 'org.junit.platform:junit-platform-launcher:1.9.2'

    testImplementation platform('org.junit:junit-bom:5.9.1')
    testImplementation 'org.junit.jupiter:junit-jupiter:5.9.2'
    testImplementation 'org.selenide:selenide-selenoid:2.3.7'

    // https://mvnrepository.com/artifact/io.qameta.allure/allure-selenide
    testImplementation 'io.qameta.allure:allure-selenide:2.22.2'
    testImplementation 'io.qameta.allure:allure-junit5:2.22.2'

    annotationProcessor('info.picocli:picocli-codegen:4.7.1')
    testImplementation 'info.picocli:picocli:4.7.1'

    testImplementation('org.apache.logging.log4j:log4j-api:2.20.0')
    testImplementation('org.apache.logging.log4j:log4j-core:2.20.0')
    testImplementation('org.apache.logging.log4j:log4j-slf4j2-impl:2.20.0') 
}

---

Other project has a same problem:

>>>> administrationUserInternalRoleAdminDelete() : SUCCESS
>>>> testPlanExecutionFinished...
>>>> administrationUserInternalRoleAdminNew() : SUCCESS
>>>> administrationUserInternalRoleAdminEdit() : SUCCESS
>>>> administrationUserInternalRoleAdminDelete() : SUCCESS

Test run finished after 33758 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         8 containers started    ]
[         0 containers aborted    ]
[         8 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         0 tests skipped         ]
[         6 tests started         ]
[         0 tests aborted         ]
[         6 tests successful      ]
[         0 tests failed          ]

---

package ru.heisenbug.service;

import org.junit.platform.engine.TestExecutionResult;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.*;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import ru.heisenbug.config.TestLauncherConfig;

import java.io.PrintWriter;

import static org.junit.platform.engine.discovery.ClassNameFilter.includeClassNamePatterns;

public  class TestLauncherService {
    final private Launcher launcher;
    private final Logger logger = LoggerFactory.getLogger(TestLauncherService.class);

    public TestLauncherService() {


        launcher = LauncherFactory.create();
    }

    public void executeTestPlan(String browser) {

        logger.info("Browser (executeTestPlan): {}", browser);

        SummaryGeneratingListener summaryGeneratingListener = new SummaryGeneratingListener();
        launcher.registerTestExecutionListeners(summaryGeneratingListener, new TestListener());

        LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder
                .request()
                .configurationParameter(TestLauncherConfig.BROWSER.getParameter(), browser)
                .selectors(DiscoverySelectors.selectPackage("ru.heisenbug.tests"))
                .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\.Google.*"))
                // .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\.XXGoogle.*"))
                // .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\..*"))
                .build();
        launcher.execute(request, summaryGeneratingListener);

        try(PrintWriter printWriter = new PrintWriter(System.out)) {
            summaryGeneratingListener.getSummary().printTo(printWriter);
        }
    }


    static class TestListener implements TestExecutionListener {

        @Override
        public void testPlanExecutionFinished(final TestPlan testPlan) {

            System.out.println(">>>> testPlanExecutionFinished...");
            TestExecutionListener.super.testPlanExecutionFinished(testPlan);
            testPlan.getRoots().stream().forEach(testIdentifier -> {
                testPlan.getDescendants(testIdentifier).stream().filter(TestIdentifier::isTest).forEach(testIdentifier1 -> {
                    System.out.println(">>>> " + testIdentifier1.getDisplayName() + " : ???");
                });

            });
        }

        @Override
        public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {

            if(testIdentifier.isTest()) {
                if (testExecutionResult.getStatus() == TestExecutionResult.Status.FAILED)
                {
                    System.out.println(">>>> " + testIdentifier.getDisplayName() + " : FAILED");
                }
                else if (testExecutionResult.getStatus() == TestExecutionResult.Status.SUCCESSFUL)
                {
                    System.out.println(">>>> " + testIdentifier.getDisplayName() + " : SUCCESS");
                }
            }
        }
    }
}

[sormuras]

Are there any dynamic tests involved? https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests

If yes, then the number of non-dynamic tests - that is the number "static" tests as discovered by the engines before the actual test run - is expected to be lower (or equal) of the number of started tests.

[mourav2222]

@sormuras No, i have no dynamic tests, only regular @Test static tests.

The effect happens with double counters if i add this Listener:

launcher.registerTestExecutionListeners(summaryGeneratingListener, new TestListener());

If comment out this line, all counter are correct.

I think, something runs wrong in the summaryGeneratingListener.getSummary() if the TestExecutionListener is registered.

Result with launcher.registerTestExecutionListeners(summaryGeneratingListener, new TestListener());:

Result without launcher.registerTestExecutionListeners(summaryGeneratingListener, new TestListener());:

[sbrannen]

I've edited the issue's description and your comments to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

[sbrannen]

@mourav2222, please refrain from posting screenshots of text.

[sbrannen]

I cannot reproduce the claimed duplicate behavior.

If you would like for us to assist you, please provide a minimal sample application that reproduces the problem -- preferably a project that we can download and run (such as a public Git repository or a ZIP file).

[mourav2222]

no problem, this is my project where it happened.
https://github.com/mourav2222/tests-cmd.git. Please take a branch windows

Start configuration: 'Tester - smoketest -chrome'

My Result:

>>>> testPlanExecutionFinished...
Counted Tests: 3
>>>> 1. test1() : SUCCESS
>>>> 2. userCanSearchKeyword2() : SUCCESS
>>>> 3. Human-readable test name : FAILED
===============================================================
Found Tests: 3
Succeeded Tests: 4
Failed Tests: 2
Aborted Tests: 0

Test run finished after 16579 ms
[         2 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         0 tests skipped         ]
[         6 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         2 tests failed          ]

[sbrannen]

Thanks for sharing the project.

The main branch seems to be OK, but the windows branch incorrectly registers the summaryGeneratingListener twice:

• once via launcher.registerTestExecutionListeners(...)
• a second time via launcher.execute(...)

That's why you see all counters doubled.

   public void executeTestPlan(String browser) {

        logger.info("Browser (executeTestPlan): {}", browser);

        SummaryGeneratingListener summaryGeneratingListener = new SummaryGeneratingListener();
        launcher.registerTestExecutionListeners(summaryGeneratingListener, new TestListener());

        LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder
                .request()
                .configurationParameter(TestLauncherConfig.BROWSER.getParameter(), browser)
                .selectors(DiscoverySelectors.selectPackage("ru.heisenbug.tests"))
                .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\.Google.*"))
                // .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\.XXGoogle.*"))
                // .filters(includeClassNamePatterns("ru\\.heisenbug\\.tests\\..*"))
                .build();
        launcher.execute(request, summaryGeneratingListener);

I am therefore closing this issue.

[mourav2222]

thanks for tips, it works corrtectly.