Question about test cases distribution between FULL and non-FULL modes

[AlexeySachkov]

Hi folks,

We have been working on support for SYCL 2020 specialization constants in our SYCL implementation at intel/llvm and we have realized that unless you switch into FULL mode, you are missing a huge piece of data type coverage for test_specialization_constants. In particular, we do not test vec and marray data types in non-FULL mode at all in those tests.

Therefore, I was wondering if there are some guideline about how CTS should be written to ensure meaningful coverage in both FULL and non-FULL modes? Do we need to explicitly document the distribution in test plans?

[psalz]

We discussed this in the working group and the consensus seems to be that its likely not possible to come up with general guidelines for what types should and should not be included in the non-full conformance mode. The problem is that the sets of types that trigger different behavior vary between both SYCL features and SYCL implementations, so there's likely no one-size-fits-all answer here. Instead we need to rely on code review and implementer feedback to come up with a good tradeoff on a case-by-case basis.

That being said, the types that are to be tested in non-full conformance mode should probably be listed in test plans. We can definitely add a note for this to the test plans README.