Concurrency test failures with GlassFish 7

[arjantijms]

As of today there are 10 2 0 failures when running the Concurrency 3 TCK with GlassFish 7:

[ERROR] Failures:
[ERROR] SecurityTests>Arquillian.arquillianBeforeClass:99 » Deployment Could not deploy security.ear
[ERROR] SecurityTests>Arquillian.arquillianBeforeClass:99 » Deployment Could not deploy security.ear

[ERROR] ManagedExecutorDefinitionTests>Arquillian.run:138->testCompletedFuture:104->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]

[ERROR] ManagedScheduledExecutorDefinitionTests>Arquillian.run:138->testCompletedFutureMSE:101->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]

[ERROR] ManagedThreadFactoryDefinitionTests>Arquillian.run:138->testManagedThreadFactoryDefinitionAllAttributes:84->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]
[ERROR] ManagedThreadFactoryDefinitionTests>Arquillian.run:138->testManagedThreadFactoryDefinitionAllAttributesEJB:90->TestClient.runTest:70->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]
[ERROR] ManagedThreadFactoryDefinitionTests>Arquillian.run:138->testParallelStreamBackedByManagedThreadFactory:106->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]
[ERROR] ManagedThreadFactoryDefinitionTests>Arquillian.run:138->testParallelStreamBackedByManagedThreadFactoryEJB:112->TestClient.runTest:70->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]

[ERROR] DeploymentDescriptorTests>Arquillian.run:138->testDeploymentDescriptorDefinesManagedThreadFactory:92->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]
[ERROR] SignatureTests>Arquillian.run:138->testSignatures:54->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]

The total result is:

[ERROR] Tests run: 166, Failures: 10, Errors: 0, Skipped: 6

[arjantijms]

The TCK can be run after building GF locally by navigating into

[glassfish root]/appserver/tests/tck/concurrency

And then typing:

mvn clean install

A single test can be run like this:

mvn clean install -Dtest=SecurityTests

and with debugging:

mvn clean install -Dtest=SecurityTests -Dglassfish.suspend

[arjantijms]

Exception related to "SecurityTests":

javax.naming.NameAlreadyBoundException: Use rebind to override 

nameee.jakarta.tck.concurrent.spec.ManagedScheduledExecutorService.security.SecurityTestRemote#ee.jakarta.tck.concurrent.spec.Managed

ScheduledExecutorService.security.SecurityTestRemote
        at com.sun.enterprise.naming.impl.TransientContext.doBindOrRebind(TransientContext.java:317)
        at com.sun.enterprise.naming.impl.TransientContext.bind(TransientContext.java:252)
        at com.sun.enterprise.naming.impl.SerialContextProviderImpl.bind(SerialContextProviderImpl.java:61)
        at com.sun.enterprise.naming.impl.LocalSerialContextProviderImpl.bind(LocalSerialContextProviderImpl.java:70)
        at com.sun.enterprise.naming.impl.SerialContext.bind(SerialContext.java:616)
        at com.sun.enterprise.naming.impl.SerialContext.bind(SerialContext.java:633)
        at java.naming/javax.naming.InitialContext.bind(InitialContext.java:421)
        at java.naming/javax.naming.InitialContext.bind(InitialContext.java:421)
        at com.sun.enterprise.naming.impl.GlassfishNamingManagerImpl.publishObject(GlassfishNamingManagerImpl.java:188)
        at com.sun.enterprise.naming.impl.GlassfishNamingManagerImpl.publishObject(GlassfishNamingManagerImpl.java:172)
        at com.sun.ejb.containers.BaseContainer$JndiInfo.publish(BaseContainer.java:4360)
        at com.sun.ejb.containers.BaseContainer.initializeHome(BaseContainer.java:1348)
Caught: java.lang.RuntimeException: Error while binding JNDI name ee.jakarta.tck.concurrent.spec.ManagedScheduledExecutorService.security.SecurityTestRemote#ee.jakarta.tck.concurrent.spec.ManagedScheduled
ExecutorService.security.SecurityTestRemote for EJB SecurityTestEjb
        at com.sun.ejb.containers.BaseContainer.initializeHome(BaseContainer.java:1361)
        at com.sun.ejb.containers.StatelessSessionContainer.initializeHome(StatelessSessionContainer.java:159)
        at com.sun.ejb.containers.StatelessContainerFactory.createContainer(StatelessContainerFactory.java:39)
        at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:198)

[arjantijms]

Note that a couple of these tests have challenges:

jakartaee/concurrency#226

and

jakartaee/concurrency#212

@aubi Did your code and Payara pass these already?

[aubi]

Hi,
in our TCK runner I ignored the tests I challenged. For the ManagedThreadFactory I have a solution, but I think the way Payara is implemented is correct.

[arjantijms]

@aubi Do you have the full lists of tests you excluded?

Without exclusions we have:

Tests run: 166, Failures: 10, Errors: 0, Skipped: 6

[arjantijms]

@aubi I think it's this list, right?

https://github.com/payara/jakartaee-10-tck-runners/blob/main/concurrent-tck/src/test/resources/tck-suite.xml

[aubi]

@arjantijms
Yes, exactly.

[arjantijms]

With the exclusions added things look much better:

[ERROR] Failures: 
[ERROR]   SignatureTests>Arquillian.run:138->testSignatures:54->TestClient.runTest:52->TestClient.runTest:60->TestClient.assertSuccessfulURLResponse:143->ArquillianTests.assertTrue:108 Output did not contain successful message: SUCCESS expected [true] but found [false]

Just the signature test faling now.

[ERROR] Tests run: 150, Failures: 1, Errors: 0, Skipped: 0

[arjantijms]

@aubi As of PR #24016 all tests now pass, mostly thanks to your amazing hard work!

===============================================
jakarta-concurrency
Total tests run: 150, Passes: 150, Failures: 0, Skips: 0
===============================================

[INFO] Tests run: 150, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 178.632 s - in TestSuite

[aubi]

@breakponchito did half of the work! :-)

[arjantijms]

@breakponchito congrats and thanks to you too!