Skip to content

Commit

Permalink
8295663: Rephrase introduction to testing.md
Browse files Browse the repository at this point in the history
Reviewed-by: ihse, erikj
  • Loading branch information
Ludvig Janiuk authored and magicus committed Nov 7, 2022
1 parent 7e85b41 commit b6738c1
Show file tree
Hide file tree
Showing 2 changed files with 84 additions and 56 deletions.
19 changes: 12 additions & 7 deletions doc/testing.html
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ <h1 class="title">Testing the JDK</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</a><ul>
<li><a href="#overview">Overview</a></li>
<li><a href="#running-tests-locally-with-make-test">Running tests locally with <code>make test</code></a><ul>
<li><a href="#configuration">Configuration</a></li>
</ul></li>
<li><a href="#test-selection">Test selection</a><ul>
Expand All @@ -49,10 +50,12 @@ <h1 class="title">Testing the JDK</h1>
<li><a href="#editing-this-document">Editing this document</a></li>
</ul>
</nav>
<h2 id="using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</h2>
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
<p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases.</p>
<h2 id="overview">Overview</h2>
<p>The bulk of JDK tests use <a href="https://openjdk.org/jtreg/">jtreg</a>, a regression test framework and test runner built for the JDK's specific needs. Other test frameworks are also used. The different test frameworks can be executed directly, but there is also a set of make targets intended to simplify the interface, and figure out how to run your tests for you.</p>
<h2 id="running-tests-locally-with-make-test">Running tests locally with <code>make test</code></h2>
<p>This is the easiest way to get started. Assuming you've built the JDK locally, execute:</p>
<pre><code>$ make test</code></pre>
<p>This will run a default set of tests against the JDK, and present you with the results. <code>make test</code> is part of a family of test-related make targets which simplify running tests, because they invoke the various test frameworks for you. The &quot;make test framework&quot; is simple to start with, but more complex ad-hoc combination of tests is also possible. You can always invoke the test frameworks directly if you want even more control.</p>
<p>Some example command-lines:</p>
<pre><code>$ make test-tier1
$ make test-jdk_lang JTREG=&quot;JOBS=8&quot;
Expand All @@ -62,6 +65,8 @@ <h2 id="using-make-test-the-run-test-framework">Using &quot;make test&quot; (the
$ make test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
$ make test TEST=&quot;micro:java.lang.reflect&quot; MICRO=&quot;FORK=1;WARMUP_ITER=2&quot;
$ make exploded-test TEST=tier2</code></pre>
<p>&quot;tier1&quot; and &quot;tier2&quot; refer to tiered testing, see further down. &quot;TEST&quot; is a test selection argument which the make test framework will use to try to find the tests you want. It iterates over the available test frameworks, and if the test isn't present in one, it tries the next one. The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code> are kept as aliases.</p>
<h3 id="configuration">Configuration</h3>
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
<p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=&lt;path to JMH jars&gt;</code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
Expand All @@ -81,8 +86,8 @@ <h3 id="common-test-groups">Common Test Groups</h3>
<p>A brief description of the tiered test groups:</p>
<ul>
<li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li>
<li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components (for example, XML).</p></li>
<li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests (<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li>
<li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components(for example, XML).</p></li>
<li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests(<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li>
<li><p><code>tier4</code>: This test group includes every other test not covered by previous tiers. It includes, for example, <code>vmTestbase</code> suites for Hotspot, which run for many hours even on large machines. It also runs GUI tests, so the same <code>TEST_JOBS</code> and <code>JTREG_KEYWORDS</code> caveats apply.</p></li>
</ul>
<h3 id="jtreg">JTReg</h3>
Expand Down
121 changes: 72 additions & 49 deletions doc/testing.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,26 @@
% Testing the JDK

## Using "make test" (the run-test framework)
## Overview

This new way of running tests is developer-centric. It assumes that you have
built a JDK locally and want to test it. Running common test targets is simple,
and more complex ad-hoc combination of tests is possible. The user interface is
forgiving, and clearly report errors it cannot resolve.
The bulk of JDK tests use [jtreg](https://openjdk.org/jtreg/), a regression
test framework and test runner built for the JDK's specific needs. Other test
frameworks are also used. The different test frameworks can be executed
directly, but there is also a set of make targets intended to simplify the
interface, and figure out how to run your tests for you.

The main target `test` uses the jdk-image as the tested product. There is
also an alternate target `exploded-test` that uses the exploded image
instead. Not all tests will run successfully on the exploded image, but using
this target can greatly improve rebuild times for certain workflows.
## Running tests locally with `make test`

Previously, `make test` was used to invoke an old system for running tests, and
`make run-test` was used for the new test framework. For backward compatibility
with scripts and muscle memory, `run-test` (and variants like
`exploded-run-test` or `run-test-tier1`) are kept as aliases.
This is the easiest way to get started. Assuming you've built the JDK locally,
execute:

$ make test

This will run a default set of tests against the JDK, and present you with the
results. `make test` is part of a family of test-related make targets which
simplify running tests, because they invoke the various test frameworks for
you. The "make test framework" is simple to start with, but more complex
ad-hoc combination of tests is also possible. You can always invoke the test
frameworks directly if you want even more control.

Some example command-lines:

Expand All @@ -28,6 +33,20 @@ Some example command-lines:
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
$ make exploded-test TEST=tier2

"tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a
test selection argument which the make test framework will use to try to
find the tests you want. It iterates over the available test frameworks, and
if the test isn't present in one, it tries the next one. The main target
`test` uses the jdk-image as the tested product. There is also an alternate
target `exploded-test` that uses the exploded image instead. Not all tests
will run successfully on the exploded image, but using this target can
greatly improve rebuild times for certain workflows.

Previously, `make test` was used to invoke an old system for running tests,
and `make run-test` was used for the new test framework. For backward
compatibility with scripts and muscle memory, `run-test` and variants like
`exploded-run-test` or `run-test-tier1` are kept as aliases.

### Configuration

To be able to run JTReg tests, `configure` needs to know where to find the
Expand Down Expand Up @@ -91,49 +110,53 @@ if you want to shortcut the parser.

### Common Test Groups

Ideally, all tests are run for every change but this may not be practical due to the limited
testing resources, the scope of the change, etc.
Ideally, all tests are run for every change but this may not be practical due
to the limited testing resources, the scope of the change, etc.

The source tree currently defines a few common test groups in the relevant `TEST.groups`
files. There are test groups that cover a specific component, for example `hotspot_gc`.
It is a good idea to look into `TEST.groups` files to get a sense what tests are relevant
to a particular JDK component.
The source tree currently defines a few common test groups in the relevant
`TEST.groups` files. There are test groups that cover a specific component,
for example `hotspot_gc`. It is a good idea to look into `TEST.groups` files
to get a sense what tests are relevant to a particular JDK component.

Component-specific tests may miss some unintended consequences of a change, so other
tests should also be run. Again, it might be impractical to run all tests, and therefore
_tiered_ test groups exist. Tiered test groups are not component-specific, but rather cover
the significant parts of the entire JDK.
Component-specific tests may miss some unintended consequences of a change, so
other tests should also be run. Again, it might be impractical to run all
tests, and therefore
_tiered_ test groups exist. Tiered test groups are not component-specific, but
rather cover the significant parts of the entire JDK.

Multiple tiers allow balancing test coverage and testing costs. Lower test tiers are supposed to
contain the simpler, quicker and more stable tests. Higher tiers are supposed to contain
progressively more thorough, slower, and sometimes less stable tests, or the tests that require
special configuration.
Multiple tiers allow balancing test coverage and testing costs. Lower test
tiers are supposed to contain the simpler, quicker and more stable tests.
Higher tiers are supposed to contain progressively more thorough, slower, and
sometimes less stable tests, or the tests that require special
configuration.

Contributors are expected to run the tests for the areas that are changed, and the first N tiers
they can afford to run, but at least tier1.
Contributors are expected to run the tests for the areas that are changed, and
the first N tiers they can afford to run, but at least tier1.

A brief description of the tiered test groups:

- `tier1`: This is the lowest test tier. Multiple developers run these tests every day.
Because of the widespread use, the tests in `tier1` are carefully selected and optimized to run
fast, and to run in the most stable manner. The test failures in `tier1` are usually followed up
on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows,
if enabled, run `tier1` tests.

- `tier2`: This test group covers even more ground. These contain, among other things,
tests that either run for too long to be at `tier1`, or may require special configuration,
or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components
(for example, XML).

- `tier3`: This test group includes more stressful tests, the tests for corner cases
not covered by previous tiers, plus the tests that require GUIs. As such, this suite
should either be run with low concurrency (`TEST_JOBS=1`), or without headful tests
(`JTREG_KEYWORDS=\!headful`), or both.

- `tier4`: This test group includes every other test not covered by previous tiers. It includes,
for example, `vmTestbase` suites for Hotspot, which run for many hours even on large
machines. It also runs GUI tests, so the same `TEST_JOBS` and `JTREG_KEYWORDS` caveats
apply.
- `tier1`: This is the lowest test tier. Multiple developers run these tests
every day. Because of the widespread use, the tests in `tier1` are
carefully selected and optimized to run fast, and to run in the most stable
manner. The test failures in `tier1` are usually followed up on quickly,
either with fixes, or adding relevant tests to problem list. GitHub Actions
workflows, if enabled, run `tier1` tests.

- `tier2`: This test group covers even more ground. These contain, among other
things, tests that either run for too long to be at `tier1`, or may require
special configuration, or tests that are less stable, or cover the broader
range of non-core JVM and JDK features/components(for example, XML).

- `tier3`: This test group includes more stressful tests, the tests for corner
cases not covered by previous tiers, plus the tests that require GUIs. As
such, this suite should either be run with low concurrency
(`TEST_JOBS=1`), or without headful tests(`JTREG_KEYWORDS=\!headful`), or
both.

- `tier4`: This test group includes every other test not covered by previous
tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run
for many hours even on large machines. It also runs GUI tests, so the same
`TEST_JOBS` and `JTREG_KEYWORDS` caveats apply.

### JTReg

Expand Down

1 comment on commit b6738c1

@openjdk-notifier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.