Skip to content

How to Run the Test Suite Locally

Jump to bottom
Amanda Galtman edited this page May 14, 2024 · 67 revisions

XSpec is automatically tested via GitHub Actions, Azure Pipelines and AppVeyor every time a pull request is submitted or merged into the master branch. Contributors can also run the test suite locally before submitting a pull request; doing so usually avoids breaking tests and speeds up the review process.

Contents

Installing Prerequisites

Java

As always, Java Runtime Environment must be installed.

Details

Linux

sudo apt install default-jre

Windows

Download the installer from Java SE Runtime Environment 8 or OpenJDK 17. Run it to complete the installation.

Bats

For Linux/macOS, you need Bats v1.8.0+:

Details

Linux

sudo npm install -g bats

macOS

brew install bats-core

Windows

Bats is not necessary.

curl

curl command line tool is used for downloading the test dependencies.

Details

Linux

sudo apt install curl

Windows

Windows 10 1803 or later has curl built-in.

For older versions of Windows, installing curl and including it in PATH may work. (not tested)

tar

tar command line tool is used for extracting the test dependencies.

Details

Windows

Windows 10 1803 or later has tar built-in.

For older versions of Windows, install 7z command line tool instead and include it in PATH.

(Optional) Setting XSpec Test Environment

XSpec is tested against multiple sets of dependencies. If the default set is sufficient for your local testing, you can skip this step.

If you need to test locally against a particular set of dependencies (e.g., you are upgrading the Saxon version used in testing and, therefore, need to run that particular version), this step configures that set. Do this step whenever you open a new terminal (or command prompt).

  1. Look in the test/ci/env/ directory for the .env files other than global.env. These text files define the different test environments. In that directory, the README.md file has some notes about the syntax the .env files use.
  2. Find the .env file that corresponds to the set of dependencies you want to use for your local testing.
  3. Set the XSPEC_TEST_ENV environment variable to the filename stem of the .env file you want to use. For instance, enter export XSPEC_TEST_ENV=saxon-10 on Linux/macOS or set XSPEC_TEST_ENV=saxon-10 on Windows.

Installing Dependencies

After installing the prerequisites, you need to install the dependencies.

Installing the dependencies is automated. They are installed in a temporary directory and some volatile environment variables are set for the current terminal (or command prompt). Since they are volatile, you need to do this installation whenever you open a new terminal (or command prompt).

Linux/macOS

Run test/ci/install-deps.sh with source:

Details 
$ source test/ci/install-deps.sh
Setting up saxon-9-9
...

Check the installation state with test/ci/print-env.sh:

Details

(Some lines omitted for brevity.)

$ test/ci/print-env.sh

=== Print Java version
java version "1.8.0_151"

=== Print Ant version
Apache Ant(TM) version 1.10.7 compiled on September 1 2019

=== Print Saxon version
SAXON-HE 9.9.1.7J from Saxonica (build 030415)

=== Check XML Calabash
XML Calabash version 1.1.30, an XProc processor.

=== Print XML Resolver version
XmlResolver 1.2

=== Check BaseX
BaseX 9.3.1 [Standalone]

=== Check BaseX server start and stop
HTTP Server was started (port: 8984).
HTTP Server was stopped (port: 8984).

=== Print Bats version
Bats 0.4.0

=== Print environment variables
XMLCALABASH_JAR=/tmp/xspec/xmlcalabash/xmlcalabash-1.2.5-99/xmlcalabash-1.2.5-99.jar
XML_RESOLVER_JAR=/tmp/xspec/xml-resolver/resolver.jar
BASEX_JAR=/tmp/xspec/basex/basex/BaseX.jar
SAXON_JAR=/tmp/xspec/saxon/saxon9he.jar

Windows

Run test\ci\install-deps.cmd:

Details 
C:\xspec>test\ci\install-deps.cmd
Setting up saxon-9-9
...

Check the installation state with test\ci\print-env.cmd:

Details

(Some lines omitted for brevity.)

C:\xspec>test\ci\print-env.cmd

=== Print Java version
openjdk version "1.8.0_222"

=== Print Ant version
Apache Ant(TM) version 1.10.7 compiled on September 1 2019

=== Print Saxon version
SAXON-HE 9.9.1.7J from Saxonica (build 030415)

=== Check XML Calabash
XML Calabash version 1.1.30, an XProc processor.

=== Print XML Resolver version
XmlResolver 1.2

=== Check BaseX
BaseX 9.3.1 [Standalone]

=== Check BaseX server start and stop
HTTP Server was started (port: 8984).
HTTP Server was stopped (port: 8984).

=== Print environment variables
BASEX_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\basex\basex\BaseX.jar
SAXON_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\saxon\saxon9he.jar
XMLCALABASH_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\xmlcalabash\xmlcalabash-1.2.5-99\xmlcalabash-1.2.5-99.jar
XML_RESOLVER_JAR=C:\Users\VSSADM~1\AppData\Local\Temp\xspec\xml-resolver\resolver.jar

Running Test Suites

After installing the prerequisites and the dependencies, you should now be able to run the XSpec test suites on your machine. Go to the xspec directory on the branch you want to test and run the following scripts for executing the full test suite:

Bats tests

This test suite ensures that XSpec behaves as expected under specific conditions. If you are missing a dependency, the test suite should be able to tell you what is missing or skip some tests.

Details

Linux/macOS

$ test/run-bats.sh
...
75 tests, 0 failures

Windows

It should end with EXIT_CODE=0.

C:\xspec>test\run-bats.cmd
=== START TEST CASES ================================================
...
=== END TEST CASES ==================================================
EXIT_CODE=0

XSpec tests

This test suite ensures that test/*.xspec files produce no Failure result on Schematron/XQuery/XSLT, which ensures there is no regression in the core logic of XSpec (src/compiler in particular). See README for details.

It should end with BUILD SUCCESSFUL.

Details

Linux/macOS

$ test/run-xspec-tests-ant.sh
Buildfile: .../xspec/test/ant/build.xml
...
BUILD SUCCESSFUL
Total time: 3 minutes 55 seconds

Windows

C:\xspec>test\run-xspec-tests-ant.cmd
...
Buildfile: C:\xspec\test\ant\build.xml
...
BUILD SUCCESSFUL
Total time: 1 minute 3 seconds

oXygen

Instead of running the script, you can run the test suite on oXygen.

  1. Open test/ant/build.xml in oXygen.
  2. In the Transformation Scenarios pane, select ANT (with Saxon 9 EE XSLT support) and apply it. (You may want to duplicate it and set -silent in Additional arguments.)

End-to-end tests

These test suites ensure that test\end-to-end\cases\*.xspec and test\end-to-end\cases-coverage\*.xspec files produce the report files as expected on Schematron/XQuery/XSLT. See README for details.

Each one should end with BUILD SUCCESSFUL.

Details

Linux/macOS

$ test/end-to-end/run-e2e-tests.sh
Buildfile: .../xspec/test/end-to-end/ant/run-e2e-tests/build.xml
...
BUILD SUCCESSFUL
Total time: 2 minutes 31 seconds

$ test/end-to-end/run-e2e-coverage-tests.sh
Buildfile: .../xspec/test/end-to-end/ant/run-e2e-tests/build.xml
...
BUILD SUCCESSFUL
Total time: 23 seconds

Windows

C:\xspec>test\end-to-end\run-e2e-tests.cmd
...
Buildfile: C:\xspec\test\end-to-end\ant\run-e2e-tests\build.xml
...
BUILD SUCCESSFUL
Total time: 52 seconds

C:\xspec>test\end-to-end\run-e2e-coverage-tests.cmd
...
Buildfile: C:\xspec\test\end-to-end\ant\run-e2e-tests\build.xml
...
BUILD SUCCESSFUL
Total time: 23 seconds

oXygen

Instead of running the script, you can run the test suite on oXygen.

  1. Open test/end-to-end/ant/run-e2e-tests/build.xml in oXygen.
  2. In the Transformation Scenarios pane, select ANT (with Saxon 9 EE XSLT support) and apply it. (You may want to duplicate it and set -silent in Additional arguments.)

Tips

  • Windows virtual machines for various hypervisers including VirtualBox are available for testing. They are valid for 90 days and can be downloaded at the Microsoft website.

  • Do not try to run all the test suite commands in parallel in multiple shell windows, as the commands can interfere with each other.

  • Do not press Ctrl+c before the tests finish running. That could cause intermediate files to remain in your workspace, which could cause your next test run to fail.

  • If you have in-progress test files with file names ending with .xspec, they might get run as part of the test suite. If you expect them to fail and don't want those failures to interfere with the test suite results, you can mark failing scenarios as pending or rename the files to avoid ending with .xspec.

  • If you want to run some of the Bats test cases and skip all the other cases, you can set --filter parameter of run-bats.sh or run-bats.cmd:

    Linux/macOS

    $ test/run-bats.sh --filter "CUSTOM|HOME"
 ✓ XSPEC_HOME
 ✓ XSPEC_HOME is not a directory
 ✓ XSPEC_HOME seems to be corrupted
 ✓ SAXON_CP has precedence over SAXON_HOME
 ✓ invoking xspec using SAXON_HOME finds Saxon jar and XML Catalog Resolver jar
 ✓ invoking xspec for XSLT with SAXON_CUSTOM_OPTIONS
 ✓ invoking xspec for XQuery with SAXON_CUSTOM_OPTIONS

7 tests, 0 failures

    Windows

    C:\xspec>test\run-bats.cmd --filter "CUSTOM|HOME"
7 test case(s) (Filter: "CUSTOM|HOME")
=== START TEST CASES ================================================
CASE #1: XSPEC_HOME
...
...PASS
CASE #2: XSPEC_HOME is not a directory
...
...PASS
CASE #3: XSPEC_HOME seems to be corrupted
...
...PASS
CASE #4: SAXON_CP has precedence over SAXON_HOME
...
...PASS
CASE #5: invoking xspec using SAXON_HOME finds Saxon jar and XML Catalog Resolver jar
...
...PASS
CASE #6: invoking xspec for XSLT with SAXON_CUSTOM_OPTIONS
...
...PASS
CASE #7: invoking xspec for XQuery with SAXON_CUSTOM_OPTIONS
...
...PASS
=== END TEST CASES ==================================================
EXIT_CODE=0

  • If you want to inspect temporary files after running Bats, you can set --no-tempdir-cleanup command line option and check the temporary directory:

    $ test/run-bats.sh --no-tempdir-cleanup
...
BATS_RUN_TMPDIR: /tmp/bats-run-qqggFd

    This option is not available on Windows.

  • XSpec tests and End-to-end tests run in parallel, and interspersed log messages can make it difficult to tell which test caused an Ant failure. When investigating a failure in your local workspace, consider temporarily specifying thread.count=1 Ant property:

    Linux/macOS

    test/run-xspec-tests-ant.sh -Dthread.count=1
    test/end-to-end/run-e2e-tests.sh -Dthread.count=1
    test/end-to-end/run-e2e-coverage-tests.sh -Dthread.count=1

    Windows

    test\run-xspec-tests-ant.cmd -Dthread.count=1
    test\end-to-end\run-e2e-tests.cmd -Dthread.count=1
    test\end-to-end\run-e2e-coverage-tests.cmd -Dthread.count=1

    Oxygen

    In the Transformation Scenarios pane, duplicate ANT (with Saxon 9 EE XSLT support) and set thread.count to 1 in the Parameters tab.

    That way, XSpec tests will run sequentially.

  • If you have a log from XSpec tests with interspersed messages and have trouble finding out which specific XSpec test failed, look for the message about test/ant/worker/build-worker.xml in the stack of error messages. If it references a line number in this generated file, that line should provide the path to the failing test.

  • If you want to run only some of the test cases of XSpec tests or End-to-end tests, you can set xspecfiles.dir.url.query Ant property:

    Linux/macOS

    test/run-xspec-tests-ant.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"
    test/end-to-end/run-e2e-tests.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"
    test/end-to-end/run-e2e-coverage-tests.sh -Dxspecfiles.dir.url.query="select=issue-*.xspec"

    Windows

    test\run-xspec-tests-ant.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"
    test\end-to-end\run-e2e-tests.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"
    test\end-to-end\run-e2e-coverage-tests.cmd -Dxspecfiles.dir.url.query="select=issue-*.xspec"

    xspecfiles.dir.url.query Ant property is used when scanning the test case directory. It works as a query parameter of URI passed to Saxon's fn:collection() implementation.

  • Debugging XProc is not an easy task, as it lacks a live debugger. To inspect intermediate artifacts going through the pipeline, you can use x:log step implemented in src/harnesses/harness-lib.xpl:

    Linux/macOS

    java -cp xmlcalabash-1.5.4-100.jar:slf4j-simple-1.7.36.jar \
     com.xmlcalabash.drivers.Main \
     -i source=tutorial/escape-for-regex.xspec \
     -p xspec-home=file:/tmp/xspec/ \
     -p log-indent=/tmp/log.txt \
     src/harnesses/saxon/saxon-xslt-harness.xproc

    Windows

    java -cp "xmlcalabash-1.5.4-100.jar;slf4j-simple-1.7.36.jar" ^
     com.xmlcalabash.drivers.Main ^
     -i source=tutorial/escape-for-regex.xspec ^
     -p xspec-home="file:///C:/xspec/" ^
     -p log-indent=file:///C:/my/log.txt ^
     src\harnesses\saxon\saxon-xslt-harness.xproc

    In this example, you'll get /tmp/log.txt (Linux/macOS) or C:\my\log.txt (Windows) file containing the artifact at <x:log if-set="log-indent">.

If you have any questions about XSpec or want to discuss, contribute, and share information with the XSpec community, please go to our issue tracker.

For historical searches, the Google group might also be useful.

Intro

Installation

Getting Started

Writing Scenarios

Running Scenarios

Understanding Test Results

Advanced Usage

XQuery

Schematron

Editors Integration

Continuous Integration

Related Information

Under the Hood

Developing XSpec

Managing XSpec Project

Clone this wiki locally