The :source:`Bio-Formats testing framework <components/test-suite>` component contains most of the infrastructure to run automated tests against the data repository.
After checking out source code and building all the JAR files (see
:doc:`building-bioformats`), switch to the :file:`test-suite` component and run the tests using the :program:`ant` test-automated target:
$ cd components/test-suite $ ant -Dtestng.directory=$DATA/metamorph -Dtestng.configDirectory=$CONFIG/metamorph test-automated
where $DATA is the path to the full data repository and $CONFIG is the path to the configuration repository.
Multiple options can be passed to the :program:`ant` test-automated target
by setting the testng.${option} option via the command line. Useful options
are described below.
- testng.directory
Mandatory option. Specifies the root of the data directory to be tested:
$ ant -Dtestng.directory=$DATA/metamorph -Dtestng.configDirectory=$CONFIG/metamorph test-automated
On Windows, the arguments to the test command must be quoted:
> ant "-Dtestng.directory=$DATA\metamorph" "-Dtestng.configDirectory=$CONFIG\metamorph" test-automated
- testng.configDirectory
Mandatory option. Specifies the root of the directory containing the configuration files. This directory must have the same hierarchy as the one specified by
testng.directoryand contain :file:`.bioformats` configuration files:$ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config test-automated
- testng.configSuffix
Specifies an optional suffix for the configuration files:
$ ant -Dtestng.directory=/path/to/data -Dtestng.configSuffix=win test-automated
- testng.memory
Specifies the amount of memory to be allocated to the |JVM|:
$ ant -Dtestng.directory=$DATA -Dtestng.memory=4g test-automated
Default: 512m.
- testng.threadCount
Specifies the number of threads to use for testing:
$ ant -Dtestng.directory=$DATA -Dtestng.threadCount=4 test-automated
Default: 1.
You should now see output similar to this:
Buildfile: build.xml
init-title:
[echo] ----------=========== bio-formats-testing-framework ===========----------
...
test-automated:
[testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [${logback.configurationFile}]
[testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
[testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
[testng] 17:05:28,713 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/opt/ome/bioformats/components/test-suite/logback.xml]
[testng] 17:05:28,835 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
[testng] 17:05:28,837 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
[testng] 17:05:28,876 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.sift.SiftingAppender]
[testng] 17:05:28,878 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SIFT]
[testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [loci.tests.testng] to DEBUG
[testng] 17:05:28,891 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
[testng] 17:05:28,891 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SIFT] to Logger[ROOT]
[testng] 17:05:28,892 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[loci.tests.testng]
[testng] 17:05:28,892 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
[testng] 17:05:28,894 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@706a04ae - Registering current configuration as safe fallback point
[testng] [2015-08-18 17:05:28,904] [main] testng.directory = /ome/data_repo/test_per_commit/
[testng] 17:05:28,908 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [loci.tests.testng.TimestampedLogFileAppender]
[testng] 17:05:28,909 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [logfile-main]
[testng] 17:05:28,955 |-INFO in loci.tests.testng.TimestampedLogFileAppender[logfile-main] - File property is set to [target/bio-formats-test-main-2015-08-18_17-05-28.log]
[testng] [2015-08-18 17:05:28,963] [main] testng.multiplier = 1.0
[testng] [2015-08-18 17:05:28,964] [main] testng.in-memory = false
[testng] [2015-08-18 17:05:28,964] [main] user.language = en
[testng] [2015-08-18 17:05:28,964] [main] user.country = US
[testng] [2015-08-18 17:05:28,964] [main] Maximum heap size = 455 MB
[testng] Scanning for files...
[testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
[testng] [2015-08-18 17:05:32,258] [main] Total files: 480
[testng] [2015-08-18 17:05:32,258] [main] Scan time: 3.293 s (6 ms/file)
[testng] [2015-08-18 17:05:32,258] [main] ----------------------------------------
[testng] Building list of tests...
and then eventually:
[testng] =============================================== [testng] Bio-Formats software test suite [testng] Total tests run: 19110, Failures: 0, Skips: 0 [testng] =============================================== [testng] BUILD SUCCESSFUL Total time: 16 minutes 42 seconds
In most cases, test failures should be logged in the main console output as:
[testng] [2015-08-18 17:13:13,625] [pool-1-thread-1] SizeZ: FAILED (Series 0 (expected 2, actual 1))
To identify the file, look for the initialization line preceding the test failures under the same thread:
[testng] [2015-08-18 17:13:12,376] [pool-1-thread-1] Initializing /ome/data_repo/test_per_commit/ome-tiff/img_bk_20110701.ome.tif:
The console output is also recorded under :file:`components/test-suite/target` as :file:`bio-formats-software-test-main-$DATE.log` where "$DATE" is the date on which the tests started in "yyyy-MM-dd_hh-mm-ss" format. The detailed report of each thread is recorded under :file:`bio-formats-software-pool-$POOL-thread-$THREAD-main-$DATE.log`
Configuration files can be generated for files or directories using the
:program:`ant` gen-config target. This generation target supports the same options as :program:`ant` test-automated:
$ ant -Dtestng.directory=/path/to/data -Dtestng.configDirectory=/path/to/config -Dtestng.memory=4g -Dtestng.threadCount=6 gen-config
Tests for the Bio-Formats MATLAB toolbox are written using the xunit framework and are located under :sourcedir:`components/formats-gpl/test/matlab`.
To run these tests, you will need to download or clone matlab-xunit, a xUnit framework with JUnit-compatible XML output. Then add this package together with the Bio-Formats MATLAB to your MATLAB path:
% Add the matlab-xunit toolbox to the MATLAB path
addpath('/path/to/matlab-xunit');
% Add the Bio-Formats MATLAB source to the MATLAB path
% For developers working against the source code
addpath('/path/to/bioformats/components/formats-gpl/matlab');
addpath('/path/to/bioformats/artifacts');
% For developers working against a built artifact, e.g. a release
% addpath('/path/to/bfmatlab');You can run all the MATLAB tests using :command:`runxunit`:
cd /path/to/bioformats/components/formats-gpl/test/matlab
runxunitIndividual test classes can be run by passing the name of the class:
cd /path/to/bioformats/components/formats-gpl/test/matlab
runxunit TestBfsaveIndividual test methods can be run by passing the name of the class and the name of the method:
cd /path/to/bioformats/components/formats-gpl/test/matlab
runxunit TestBfsave:testLZWFinally, to output the test results under XML format, you can use the -xmlfile option:
cd /path/to/bioformats/components/formats-gpl/test/matlab
runxunit -xmlfile test-output.xml