Create guidance on how to produce standalone TCKs

[starksm64]

Is your feature request related to a problem? Please describe.
We have a mix of testing technologies that don't mix well together to enable a collection of standalone TCKs that can be composed to produce various profile level TCKs. We need to provide guidance on how to achieve this.

Describe the solution you'd like
The specification project should own the TCK source and produce the TCK artifacts. This needs to be done in such a way that the artifacts can be run to validate implementations at the individual specification level, as well as integrated into platform level TCKs.

Describe alternatives you've considered
Jakarta Batch, Bean Validation and CDI make use of TestNG and Arquillian with maven artifacts to enable composition. Ondro Mihályi looked at this approach and also Junit 5 in the context of Jakarta Batch.

Additional context
An investigation of using TestNG/Arquillian and Junit 5 to update the Jakarta Batch TCK was described in this blog by Ondro Mihályi:
https://ondro.inginea.eu/index.php/possible-ways-to-use-arquillian-in-jakarta-ee-tcks/

[scottmarlow]

I volunteer to ping projects for feedback, especially those that have submitted ballots for EE 10 some of which already have standalone TCKs as noted in parens below. :

• CDI lite (priority for core profile)
• JSONB 2.1 (priority for core profile)
• JSONP 2.1 (priority for core profile)
• RESTful Web Services 4.0 (priority for core profile)
• Annotations (priority for core profile)
• Config (priority for core profile)
• Java Mail 2.1 (https://github.com/eclipse-ee4j/mail-tck)
• Persistence 3.1
• Server Faces 4.0
• Connectors 2.1
• Concurrency 3.0
• Servlet 6.0
• JSP 3.1
• Websocket 2.1
• JSTL 3.0
• Expression Language 5.0
• Security 3.0
• Authorization 3.0
• Authentication 3.0
• XML Web Services 4.0
• SOAP with Attachments 3.0
• XML Binding 4.0 (https://github.com/eclipse-ee4j/jaxb-tck)
• Faces 4.0
• Activation 2.1 (https://github.com/eclipse-ee4j/jaf-tck)
• MVC 2.1 (https://github.com/eclipse-ee4j/mvc-tck)

IMO we can start developing the Core Profile TCK tests as each Standalone SPEC team stages (TCK) test artifacts that can be consumed by the Core Profile TCK. IMO, the Core Profile TCK can cache the staged artifacts that it consumes such that it can control the frequency of syncing with the latest (breaking) test changes from the specification projects (to reduce the friction for the Profile TCK team and also avoid daily fires when staged specification TCK test classes change). Arquillian seems important for working with various Core Profile implementations.

Another requirement to work out is how TCK tests state the Specification requirement that they are testing such that users running the TCK tests can quickly identify the Specification text that describes the requirement (e.g. AssertionIDs are an example of test sources including a unique identifier that can be looked up in a separate table to identify the exact Specification section describes the requirement).

[starksm64]

I would order the priority to match those specifications that are currently being targeted for the core profile. I think there is some confusion/concern that there is going to be a big impact to the existing web and platform CTS projects. I'm envisioning a new core profile platform TCK that is a composition of the independent specification TCKs with additional profile specific tests. I need to create an EE10 issue to explicitly track the specifications that will be in the core profile, but it currently is CDI, JSON*, Rest, Annotations, Config,

[scottmarlow]

Proposal document is at https://docs.google.com/document/d/1yDXTUUULUrFrUi1DV_9OcBKIiZLVxrZkA38MMvYdP-U/edit#