Proposal for Multi-Library, Multi-Resource Tests

[evan-gordon]

This is based off of a discussion we had at the spring 2024 HL7 Connectathon.

The group discussed at length about adding the ability to run tests that support multiple libraries as well as accepting input resources. Since the current tests only support single statements we cannot currently test features like valuesets or the code in valueset feature. Other features like fluent functions would also be benefitted by this feature.

The team discussed a potential format for defining a test group that overcomes these and other existing gaps and came up with the following example json. The following is an example meta file for defining the inputs to a test runner which would be needed to run one of these tests and also like the associated file outputs to be asserted on.

testGroupName.json

{
  "description": "...",
  "cqlLibraries": [
    "testFile.cql"
  ],
  "resources": [
    "patient.json"
  ],
  "valuesets": [
    "valueset.json"
  ],
  "outputFile": "output.json"
}

For the output file we as a team discussed using the json definition defined here https://github.com/cqframework/clinical_quality_language/wiki/CQL-Serialization for the expected output format.

[suyashkumar]

After more discussion, we've pivoted to XML to remain more consistent with the existing test schema, allow readable multi-line tests, something along the lines of soemthing like what's below.

We haven't decided fully on references to data and libraries on disk.

<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://hl7.org/fhirpath/tests" xsi:schemaLocation="http://hl7.org/fhirpath/tests ../../testSchema/testSchema.xsd"
	name="OverloadMatching" reference="https://cql.hl7.org/09-b-cqlreference.html#aggregate-functions">
    <test name="SimpleOverloadMatching">
        <library>
        library test version '0.0.1'

        define function A(foo Integer): 1
        define function A(foo Decimal): 2

        define output: A(1.0)
        </library>
        <description>This is a test description.</description>
        <output name="output">1</output>
    </test>

    <test name="LibraryRef">
        <libraryRef identifier="mylib" path="tests/123.cql"/>
        <output name="output">1</output>
        <output name="output1">2</output>
    </test>

    <test name="DataRef">
        <libraryRef identifier="mylib" path="tests/123.cql"/>
        <dataRef path="../commondata/data.json"/>
        
        <output name="output">1</output>
        <output name="output1">2</output>
    </test>

</tests>