type=page status=published title=Executing Tests next=debug.html prev=config.html ~~ attributes.conf Executing Tests
5 Executing Tests
The {TechnologyShortName} TCK uses the JavaTest harness to execute the
tests in the test suite. For detailed instructions that explain how to
run and use JavaTest, see the JavaTest User's Guide and Reference in
the documentation bundle.
This chapter includes the following topics:
* link:#GBFUZ[Starting JavaTest]
* link:#GBFWM[Running a Subset of the Tests]
* link:#GCLRR[Running the TCK Against your selected CI]
* link:#GCLRZ[Running the TCK Against a Vendor's Implementation]
* link:#GBFVK[Test Reports]
[NOTE]
=======================================================================
The instructions in this chapter assume that you have installed and
configured your test environment as described in
link:install.html#GBFTP[Chapter 3, "Installation,"] and
link:config.html#GBFVV[Chapter 4, "Setup and Configuration,"],
respectively.
=======================================================================
[[GBFUZ]][[starting-javatest]]
5.1 Starting JavaTest
~~~~~~~~~~~~~~~~~~~~~
There are two general ways to run the {TechnologyShortName} TCK using
the JavaTest harness software:
* Through the JavaTest GUI
* From the command line in your shell environment
[NOTE]
=======================================================================
The `ant` command referenced in the following
two procedures and elsewhere in this guide is the Apache Ant
build tool, which will need to be downloaded separately.
The `build.xml` file in `<TS_HOME>/bin` contains the various Ant
targets for the {TechnologyShortName} TCK test suite.
=======================================================================
[[GBFWH]][[to-start-javatest-in-gui-mode]]
5.1.1 To Start JavaTest in GUI Mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Execute the following commands:
[source,oac_no_warn]
----
cd <TS_HOME>/bin
ant gui
----
[[GBFVW]][[to-start-javatest-in-command-line-mode]]
5.1.2 To Start JavaTest in Command-Line Mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
link:using-examples.inc[role=include]
[[GBFWM]][[running-a-subset-of-the-tests]]
5.2 Running a Subset of the Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the following modes to run a subset of the tests:
* link:#GBFVT[Section 5.2.1, "To Run a Subset of Tests in GUI Mode"]
* link:#GBFWK[Section 5.2.2, "To Run a Subset of Tests in Command-Line Mode"]
* link:#GBFVL[Section 5.2.3, "To Run a Subset of Tests in Batch Mode
Based on Prior Result Status"]
[[GBFVT]][[to-run-a-subset-of-tests-in-gui-mode]]
5.2.1 To Run a Subset of Tests in GUI Mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. From the JavaTest main menu, click *Configure*, then click *Change
Configuration*, and then click *Tests to Run*. +
The tabbed Configuration Editor dialog box is displayed.
2. Click *Specify* from the option list on the left.
3. Select the tests you want to run from the displayed test tree, and
then click *Done*. +
You can select entire branches of the test tree, or use `Ctrl+Click` or
`Shift+Click` to select multiple tests or ranges of tests, respectively,
or select just a single test.
4. Click *Save File*.
5. Click *Run Tests*, and then click *Start* to run the tests you selected. +
Alternatively, you can `right-click` the test you want from the test tree
in the left section of the JavaTest main window, and choose *Execute
These Tests* from the menu.
6. Click *Report*, and then click *Create Report*.
7. Specify the directory in which the JavaTest test harness will write
the report, and then click *OK* +
A report is created, and you are asked whether you want to view it.
8. Click *Yes* to view the report.
[[GBFWK]][[to-run-a-subset-of-tests-in-command-line-mode]]
5.2.2 To Run a Subset of Tests in Command-Line Mode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Change to the directory containing the tests you want to run.
2. Start the test run by executing the following command: +
+
[source]
----
ant runclient
----
+
The tests in the directory and its subdirectories are run.
[[GBFVL]][[to-run-a-subset-of-tests-in-batch-mode-based-on-prior-result-status]]
5.2.3 To Run a Subset of Tests in Batch Mode Based on Prior Result Status
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can run certain tests in batch mode based on the test's prior run
status by specifying the `priorStatus` system property when invoking
`ant`
Invoke `ant` with the `priorStatus` property.
The accepted values for the `priorStatus` property are any combination
of the following:
* `fail`
* `pass`
* `error`
* `notRun`
For example, you could run all the {TechnologyShortName} tests with a
status of failed and error by invoking the following commands:
[source,oac_no_warn]
----
ant -DpriorStatus="fail,error" runclient
----
Note that multiple `priorStatus` values must be separated by commas.
[[GCLRR]][[running-the-tck-against-the-ri]]
5.3 Running the TCK Against another CI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some test scenarios are designed to ensure that the configuration and deployment of
all the prebuilt {TechnologyShortName} TCK tests against one Compatible
Implementation are successful operating with other compatible implementations, and that the TCK is ready for
compatibility testing against the Vendor and Compatible Implementations.
1. Verify that you have followed the configuration instructions in
link:config.html#GBFVU[Section 4.1, "Configuring Your Environment to Run
the TCK Against the Compatible Implementation."]
2. If required, verify that you have completed the steps in
link:config.html#GCLIW[Section 4.3.2, "Deploying the Prebuilt Archives."]
3. Run the tests, as described in link:#GBFUZ[Section 5.1, "Starting
JavaTest,"] and, if desired, link:#GBFWM[Section 5.2, "Running a Subset
of the Tests."]
[[GCLRZ]][[running-the-tck-against-a-vendors-implementation]]
5.4 Running the TCK Against a Vendor's Implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This test scenario is one of the compatibility test phases that all
Vendors must pass.
1. Verify that you have followed the configuration instructions in
link:config.html#GCLHU[Section 4.2, "Configuring Your Environment to
Repackage and Run the TCK Against the Vendor Implementation."]
2. If required, verify that you have completed the steps in
link:config.html#GCLIL[Section 4.3.3, "Deploying the
Test Applications Against the Vendor Implementation."]
3. Run the tests, as described in link:#GBFUZ[Section 5.1, "Starting
JavaTest,"] and, if desired, link:#GBFWM[Section 5.2, "Running a Subset
of the Tests."]
[[GBFVK]][[test-reports]]
5.5 Test Reports
~~~~~~~~~~~~~~~~
A set of report files is created for every test run. These report files
can be found in the report directory you specify. After a test run is
completed, the JavaTest harness writes HTML reports for the test run.
You can view these files in the JavaTest ReportBrowser when running in
GUI mode, or in the web browser of your choice outside the JavaTest
interface.
To see all of the HTML report files, enter the URL of the `report.html`
file. This file is the root file that links to all of the other HTML
reports.
The JavaTest harness also creates a `summary.txt` file in the report
directory that you can open in any text editor. The `summary.txt` file
contains a list of all tests that were run, their test results, and
their status messages.
[[GBFWD]][[creating-test-reports]]
5.5.1 Creating Test Reports
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use the following modes to create test reports:
* link:#GBFVH[Section 5.5.1.1, "To Create a Test Report in GUI Mode"]
* link:#GBFVC[Section 5.5.1.2, "To Create a Test Report in Command-Line Mode"]
[[GBFVH]][[to-create-a-test-report-in-gui-mode]]
5.5.1.1 To Create a Test Report in GUI Mode
+++++++++++++++++++++++++++++++++++++++++++
1. From the JavaTest main menu, click *Report*, then click *Create Report*. +
You are prompted to specify a directory to use for your test reports.
2. Specify the directory you want to use for your reports, and then
click *OK*. +
Use the Filter list to specify whether you want to generate reports for
the current configuration, all tests, or a custom set of tests. +
You are asked whether you want to view report now.
3. Click *Yes* to display the new report in the JavaTest ReportBrowser.
[[GBFVC]][[to-create-a-test-report-in-command-line-mode]]
5.5.1.2 To Create a Test Report in Command-Line Mode
++++++++++++++++++++++++++++++++++++++++++++++++++++
1. Specify where you want to create the test report.
a. To specify the report directory from the command line at runtime,
use:
+
[source]
----
ant -Dreport.dir="report_dir"
----
+
Reports are written for the last test run to the directory you specify.
2. To specify the default report directory, set the `report.dir`
property in +{jteFileName}+. +
For example:
+
[source]
----
report.dir="/home/josephine/reports"
----
+
3. To disable reporting, set the `report.dir` property to `"none"`,
either on the command line or in +{jteFileName}+. +
For example:
+
[source]
----
ant -Dreport.dir="none"
----
+
[[GBFVB]][[viewing-an-existing-test-report]]
5.5.2 Viewing an Existing Test Report
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use the following modes to view an existing test report:
* link:#GBFVO[Section 5.5.2.1, "To View an Existing Report in GUI Mode"]
* link:#GBFWB[Section 5.5.2.2, "To View an Existing Report in
Command-Line Mode"]
[[GBFVO]][[to-view-an-existing-report-in-gui-mode]]
5.5.2.1 To View an Existing Report in GUI Mode
++++++++++++++++++++++++++++++++++++++++++++++
1. From the JavaTest main menu, click *Report*, then click *Open Report*. +
You are prompted to specify the directory containing the report you want
to open.
2. Select the report directory you want to open, and then click *Open*. +
The selected report set is opened in the JavaTest ReportBrowser.
[[GBFWB]][[to-view-an-existing-report-in-command-line-mode]]
5.5.2.2 To View an Existing Report in Command-Line Mode
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Use the Web browser of your choice to view the `report.html` file in the
report directory you specified from the command line or in +{jteFileName}+.
link:using.inc[role=include]