-
Notifications
You must be signed in to change notification settings - Fork 4
TestScript Imports
TestScript is a FHIR standard resource used for writing tests. A single TestScript resource is run by an interpreter resulting in a single TestReport resource containing the results. While this approach works it does not allow for elements of a TestScript to be reused. Reuse is important when building a large number of tests. Common elements should be coded separately and included in tests. This is an application of the DRY concept in software - Don't Repeat Yourself. FHIR resources have two extensibility features, extension and containment. Extensions are used in this proposal.
There is a second approach to reuse possible with TestScript. Tests could be coded and maintained in an external language, not part of the FHIR standard, and then translated into TestScript. This makes TestScript an interchange language. In this environment reuse is not an issue. This is not being pursued.
The following TestScript features are relevant.
Action - the primary building block of TestScript is the action. There are two types of actions: operation and assert. Operation is an interaction with a System Under Test (SUT) like POST or GET. Assert is an evaluation - the if part of an if-then statement. The then part of the statement is to emit an error or warning.
Test - a collection of actions executed in order. A script (instance of TestScript) contains zero or more tests.
Setup - a collection of actions to be run before tests to initialize the SUT for testing.
Teardown - a collection of actions run after the tests to cleanup the SUT. Teardown includes operations only (no asserts).
Fixture - a container for a resource so the resource can be passed between actions. Fixtures can be static or dynamic. Static fixtures are defined within the source of the test and are used to initialize the SUT. Dynamic fixtures hold the result (response) of an Operation. Dynamic fixtures include a header and body (resource) - the entire message returned from the SUT. Static fixtures contain only the resource (body). Operations that include a body (POST) point to a fixture to define the body contents.
Assigning ids to fixtures is handled differently for static and dynamic fixtures. Static fixtures are required to have an id attribute which holds the id of the fixture. Dynamic fixtures are created by operations and the id is assigned using the targetId attribute of the operation.
Variable - reference to part of a Fixture. Variables are used to extract content out of a Fixture header or body for inclusion in an operation or assert. Variables can also hold fixed data (a constant). If there exists a local variable defined in TestScript, an external variable or the variable that is passed into a module component with the same name overrides the local variable value.
Three of the above elements are the focus for reuse: Fixtures, Variables, and Tests.
Tests, operations and assertions, are the obvious focus of testing. They interact with the system under test (SUT) and evaluate it. Fixtures and Variables supply the data to be used in Tests and the data passed between tests.
The Component term refers to the internal TestScript extension URI. The term module is used interchangeably with component.
To achieve reuse, TestScripts are divided into two types: scripts and components. Scripts are the documented tests a test harness can run against an SUT. They might be visible in a testing user interface. A script is a standalone test.
Components are used to build scripts. A component can be either a standalone TestScript or an import of TestScript Action(s). It contains test instructions that can be bundled into a script. Components are stored in a Library. Libraries, as stored on the file system, are directories containing TestScript files (XML or JSON). A component is defined in a TestScript file. Scripts are authored with a combination of custom content and references to components.
For standalone test import, the parent TestScript must have a Test reserved for the import of another TestScript. The Test Action Operation is used for the import module call. The imported TestScript can have many Tests. The reason for a Test reservation is that it conveys the imported top-level TestScript TestReport status. In the UI, the display component is a recursive Test display, so when the Test is expanded, it shows in the Test details.
For a direct transfer of TestScript Action imports, importing a TestScript means that 1) Component or Module has only one Test, and 2) all of its actions are directly transferred to the parent TestScript Test. Many Modules can be imported within a parent TestScript Test Operation. When this pattern is used, there is a Test with a series of imported actions.
A script can use any number of components. It does so by referencing the Library TestScript and passing parameters to it.
References to components are coded using FHIR Extension elements:
Component reference:
"extension": [{
"url": "urn:component",
"valueString": "../Library/MyTestScriptComponent.json"
}]
Note: the url values used in Extension are specified by FHIR to be full URLs identifying StructureDefinitions. These examples use simple names for readability.
The TestScript resource specification documents the following ways of identifying these elements:
Static fixtures - id attribute
Dynamic fixtures - targetId attribute in an operation
Tests - TestScript.test.name
Variables - TestScript.variable.name
A component will be coded with sourceIds and variable names known only to that component. They are local names. A script needs to pass values to components: sourceIds and variables. The names assigned in the script are meaningful only to the script. To handle this interchange, components are defined with positional parameters which are passed by scripts.
A call to a component from a script includes a reference to the component (shown above) and also needs a specification of the positional parameters.
"modifierExtension": [{
"url": "https://github.com/usnistgov/asbestos/wiki/TestScript-Import", // import a component
"extension": [{
"url": "component", // the component
"valueString": "../Library/PdbTransaction.json"
}, {
"url": "urn:variable-in", // variable 1
"valueString": "PhoneNumber" // Variable that can be resolved in Script
}, {
"url": "urn:fixture-in", // fixture 1
"valueString": "pdbResponse" // resource identified by fixture with id pdbResponse
}]
}]
The nested extensions could have been coded as a Parameters resource if it was allowed in an extension.
This is a complete component import specification which includes the passing of positional parameters. First the extension is identified as an import (of a component). The nested extension identified by urn:component identifies the component to be imported. The next two extensions specify two positional parameters, a variable and a fixture. Variables and fixtures can be intermingled. In the translator they are interpreted as two separate list. The order within each list is important.
The above component import aligns with this module header declaration in the component:
{
"resourceType": "TestScript",
"modifierExtension": [{
"url": "urn:module"
"extension": [{
"url": "urn:fixture-in", // inbound fixture 1
"valueString": "response"
}, {
"url": "urn:variable-in", // inbound variable 1
"valueString": "telephone"
}]
}]
...
First variable in the call aligns with the first variable in the module header etc.
Values and data need to be returned from the component back to the script. This can be done through variables and fixtures. This is done by passing variable names and fixture names to the component through (additional) positional parameters. In the import:
{
"url": "urn:variable-out", // outbound variable 1
"valueString": "IPAddress" // variable name
}, {
"url": "urn:fixture-out", // outbound fixture 1
"valueString": "responseMessage" // fixture name
}
These additional parameters are specified in the script to call the component. At the completion of the component call, variable IPAddress is set to a string and fixture responseMessage is set to a response (header and body).
The matching parameter declaration in the component looks like:
{
"url": "urn:variable-out", // outbound variable 1
"valueString": "serverAddress" // local variable name
}, {
"url": "urn:fixture-out", // outbound fixture 1
"valueString": "serverMessage" // fixture name
}
To summarize, fixture-in, variable-in, fixture-out, and variable-out are organized into separate lists. They can be intermingled in the import and in the module header but once they are sorted out into their separate lists they are handled as ordered parameters. First fixture-in in the call aligns with the first fixture-in in the module.
When a module is to-be called, the calling script may pass an input fixture, which contains variable references. The variable references in the input fixture rely on variable-in-no-translation extension. The variable is interpolated and replaced in the fixture. The no-translation variable has no purpose by it self in the module. An example is passing a static fixture which is coded with a variable in its contents to a module which will POST it. The name of the variable is coded in two places, the caller script and the source of the static fixture being passed to the module. This name alignment must be maintained. In this case the module should be called with:
{
"url": "urn:variable-in-no-translation",
"valueString": "patientResourceId"
}
There is no corresponding module header declaration. A complete example can be found in asbestos-war/src/test/resources/delegation/libraryTest/TestScript.xml
When a module is called, the local variable map is searched for the variable name. If the internal TestScript variable map key does not exist, the parent TestScript variable map, if it exists, is used to search for the key. If this key is not found, the key is treated as a variable value can be evaluated for an expression against an anonymous fixture and the value is considered an output of an anonymous variable. Anonymous variables can be expressions like '1' or 'true', each beginning with a single quote. This eliminates the requirement to code simple variables in the parent TestScript. However, the module component definition still requires the parameter mapping as required for the variable-in parameter mapping in the TestScript header. If an anonymous variable evaluation error occurs, then an error is returned. If a normal variable (any value that does not begin with a single quote) is not found in the variable map, then a referenced but not declared error is returned. Anonymous variables are traceable in the TestReport, if the Test is Passes, and shown in the "Called Module:" report section.
Running a TestScript returns a TestReport. That is unchanged. The top level TestScript, the script as discussed above, carries the overall result - pass or fail. If a module/component fails, that result/status is reflected in the result/status of the test.action.operation that called it. The execution of the component produces a separate TestReport that contains the details of its operation.
This example is taken from ModuleIT, an integration test run during the build of Asbestos.
<TestScript xmlns="http://hl7.org/fhir">
<url value=""/>
<name value="test1"/>
<status value="draft"/>
Static fixture defined with the test. Patient/Alex_Alder.json is a relative path to this file. The reason for packaging this in a Bundle comes from the way the toolkit manages Patients for conformance testing.
<fixture id="patient-bundle">
<!--
This patient comes out of the test Patient cache
-->
<autocreate value="false"/>
<autodelete value="false"/>
<resource>
<reference value="Patient/Alex_Alder"/>
</resource>
</fixture>
Content for the ProvideDocumentBundle transaction packaged with the test.
<fixture id="pdb-bundle">
<autocreate value="false"/>
<autodelete value="false"/>
<resource>
<reference value="Bundle/pdb.xml"/>
</resource>
</fixture>
Variable holding the reference to the Patient resource taken from the Patient Bundle.
<variable>
<name value="patientResourceId"/>
<expression value="Bundle.entry.fullUrl"/>
<sourceId value="patient-bundle"/>
</variable>
<test>
<action>
<operation>
Import coded as part of an operation. The overall extension is identified as an import. The import contains a component reference (path is relative to the directory holding this TestScript). One fixture is passed to the component (fixture-in). Its local name is pdb-bundle. The patientResourceId variable is passed. Since this variable name is coded in Bundle/pdb.xml it is passed without translation (variable-in-no-translation). A single fixture is returned (fixture-out), the response message from the PDB transaction.
Each parameter is coded with the local name - the name coded in this script.
<modifierExtension url="https://github.com/usnistgov/asbestos/wiki/TestScript-Import">
<extension url="component">
<valueString value="module.xml"/>
</extension>
<extension url="urn:fixture-in">
<valueString value="pdb-bundle"/>
</extension>
<extension url="urn:variable-in-no-translation">
<valueString value="patientResourceId"/>
</extension>
<extension url="urn:fixture-out">
<valueString value="pdb-response"/>
</extension>
</modifierExtension>
</operation>
</action>
<action>
This assert duplicates what is found in the component. It is included to show that the fixture-out can be referenced/used. If the transaction had failed then the above operation would have failed.
<assert>
<description value="... transaction was successful"/>
<response value="okay"/>
<sourceId value="pdb-response"/>
<warningOnly value="false"/>
</assert>
</action>
</test>
</TestScript>
<TestScript xmlns="http://hl7.org/fhir">
This header must align with the caller's import. The header is defined by the modifierExtension with url urn:module. The caller and this module each has one fixture-in and one fixture-out so the alignment is good. The fixture-in is given a local name that is used in other parts of this module. The fixture-out reference a fixture produced by this module.
<modifierExtension url="urn:module">
<extension url="urn:fixture-in">
<valueString value="pdb-bundle-param"/>
</extension>
<extension url="urn:fixture-out">
<valueString value="pdb-response-module"/>
</extension>
</modifierExtension>
<url value=""/>
<name value="test1"/>
<status value="draft"/>
<test>
<action>
This operation uses the input fixture and produces content for the output fixture of the module.
<operation>
<type>
<system value="https://github.com/usnistgov/asbestos/wiki/Testscript-Operation-Codes"/>
<code value="mhd-pdb-transaction"/>
</type>
<responseId value="pdb-response-module"/>
<sourceId value="pdb-bundle-param"/>
</operation>
</action>
<action>
This assert validates the operation of the above operation. Normally this module would have many more asserts.
<assert>
<description value="... transaction was successful"/>
<response value="okay"/>
<warningOnly value="false"/>
</assert>
</action>
</test>
</TestScript>
When this module completes, it returns: the overall result (pass/fail) and the output fixture.
{
"resourceType": "TestReport",
"name": "callTest",
"status": "completed",
"testScript": {
"reference": "/home/bill/develop/asbestos/asbestos-war/target/test-classes/delegation/callTest/module.xml"
},
The component execution passed.
"result": "pass",
"issued": "2020-04-01T06:38:40-04:00",
"participant": [
{
"type": "server",
"uri": "http://localhost:8081/asbestos/proxy/default__fhirpass",
"display": "NIST Asbestos Proxy"
},
{
"type": "test-engine",
"uri": "https://github.com/usnistgov/asbestos",
"display": "NIST Asbestos TestEngine"
}
],
"test": [{ {
"action": [
{
"operation": {
"result": "pass",
"message": "### POST http://localhost:8081/asbestos/proxy/default__fhirpass\n### Fixtures\n**pdb-response-module**: <a href=\"http://localhost:8082/session/default/channel/fhirpass/lognav/2020_04_01_06_38_40_236\" target=\"_blank\">http://localhost:8082/session/default/channel/fhirpass/lognav/2020_04_01_06_38_40_236</a>\n**pdb-bundle-param**: Bundle/pdb.xml (static)\n### Variables\n",
"detail": "http://localhost:8081/asbestos/log/default/fhirpass/Bundle/2020_04_01_06_38_40_236"
}
},
{
"assert": {
"result": "pass",
"message": "Operator is equals\nexpected is okay\nfound is okay\nokay equals okay"
}
}
]
}]
}
{
"resourceType": "TestReport",
"name": "callTest",
"testScript": {
"reference": "/home/bill/develop/asbestos/asbestos-war/target/test-classes/delegation/callTest/TestScript.xml"
},
Script passed.
"result": "pass",
"issued": "2020-04-01T06:38:39-04:00",
"participant": [
{
"type": "server",
"uri": "http://localhost:8081/asbestos/proxy/default__fhirpass",
"display": "NIST Asbestos Proxy"
},
{
"type": "test-engine",
"uri": "https://github.com/usnistgov/asbestos",
"display": "NIST Asbestos TestEngine"
}
],
"test": [
{
"action": [
{
This operation corresponds to the component.
"operation": {
"result": "pass"
}
},
{
This assert used the returned fixture.
"assert": {
"result": "pass",
"message": "Operator is equals\nexpected is okay\nfound is okay\nokay equals okay"
}
}
]
}
]
}