Command line output says "reject", validation result document says "accept"

[fbuettner-hb]

The output on the command line and the validation result documents diverge when there are schematron rules with role="warning" or when the configuration file sets a custom error level "warning" to a schematron rule.

I have attached a minimal example that reproduces the problem.

example.xsd:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="test">
        <xs:complexType>
            <xs:attribute name="a" type="xs:integer" use="required"/>
            <xs:attribute name="b" type="xs:integer" use="required"/>
            <xs:attribute name="c" type="xs:integer" use="required"/>
        </xs:complexType>
    </xs:element>
</xs:schema>

example.sch (please notice role="warning" for SCH-TEST-2):

<sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron" queryBinding="xslt2">
    <sch:pattern>
        <sch:rule context="test" role="warn">
            <sch:assert id="SCH-TEST-1" test="xs:integer(@a) gt 10">Value of 'a' must be greater than 10.</sch:assert>
            <sch:assert id="SCH-TEST-2" test="xs:integer(@b) gt 10" role="warning">Value of 'b' should be greater than 10.</sch:assert>
            <sch:assert id="SCH-TEST-3" test="xs:integer(@c) gt 10">Value of 'c' must be greater than 10.</sch:assert>
        </sch:rule>
    </sch:pattern>
</sch:schema>

scenarios.xml:

<scenarios xmlns="http://www.xoev.de/de/validator/framework/1/scenarios" frameworkVersion="1.0.0">
    ...
    <scenario>
         ...
            <customLevel level="warning">SCH-TEST-3</customLevel>
         ...
    </scenario>
     ...
</scenarios>

example1.xml:

<test a="4" b="11" c="11"/>

example2.xml:

<test a="11" b="4" c="11"/>

example3.xml:

<test a="11" b="11" c="4"/>

Tabular command line output wrongly says "reject" for all three example documents:

Validation result documents correctly say "reject", "accept" and "accept"
(rep:report/rep:assessment/rep:reject, rep:report/rep:assessment/rep:accept, rep:report/rep:assessment/rep:accept in results/example*.xml).

Complete example attached
minimal-example.zip.

Reproduce as follows:

java -jar validationtool-1.5.0-java8-standalone.jar -r configuration/ -s configuration/scenarios.xml -o results -h examples/*.xml

[fbuettner-hb]

Problem solved. It is a configuration problem.

By default, the validator does not recognize the generated validation report (whose generation is configured as an XSLT skript). Hence, any XML schema violation and any failed schematron assertion (regardless of the severity) leads to "reject".
To make the validator recognize the actual assessment of the report (which considers severities and also custom level settings) one needs to add the following line to each scenario:

<acceptMatch>exists(/*:report/*:assessment/*:accept)</acceptMatch>

In full:

<scenario>
  <name>test scenario</name>
  <match>/test</match>
  <validateWithXmlSchema>
  <resource>
     <name>XML Schema</name>
     <location>resources/example.xsd</location>
   </resource>
</validateWithXmlSchema>
        <validateWithSchematron>
            <resource>
                <name>Schematron</name>
                <location>resources/example.xsl</location>
            </resource>
        </validateWithSchematron>
        <createReport>
            <resource>
                <name>Report</name>
                <location>resources/default-report.xsl</location>
            </resource>
            <customLevel level="warning">SCH-TEST-3</customLevel>
        </createReport>
        <acceptMatch>exists(/*:report/*:assessment/*:accept)</acceptMatch>
    </scenario>