Validate Platform requirements for spec providing standalone TCKs

[scottmarlow]

See jakartaee/jsonp-api#297 discussion about possible ways to validate Platform requirements for JSONP.

[lukasj]

this can be generalized to Validate Platform requirements for spec providing standalone TCKs to cover ie JSONB or JAXRS as well

[scottmarlow]

CC #574 #757 jakartaee/jsonp-api#297 jakartaee/jsonb-api#281

[m0mus]

Here is my understanding about how it should work.

Standalone TCK tests

• Each specification should contain standalone TCK tests in a separate repository or in API repository. Standalone tests are testing an implantation in Java SE container.
• It doesn't matter what technologies are used to write TCK tests. Important that TCK tests are released as a Maven artifact and can be run as part of an implementation project testing.
• In order to test an implementation for compliance, the implementation must pass all standalone TCK tests.

Jakarta Platform TCK tests

• A spec can define some additional requirements when it's used as part of Jakarta platform. In this case Jakarta Platform should contain TCK tests for this functionality. It should not include all standalone spec TCK tests. It's a separate set of tests and can be relatively small.
• It doesn't matter what technology Jakarta Platform is using for TCK tests. It's up to the platform or Jakarta TCK project to decide.
• In order to test a platform implementation, the implementation must use certified implementation for each spec (see 3-rd bullet in the section above) and pass all platform TCK tests (see a bullet above).

[starksm64]

I think this proposal is good. The details today are mixed in that ideally the current TCK level dependencies would match API dependencies. For example CDI is relatively early on the in the API dependency tree, but it has tests for all downstream containers. Ideally there would be a set of profile tests that could be integrated into the respective profile/platform TCK, but we don't have the common infrastructure to easily support this today.