Skip to content

Commit

Permalink
Update test appeals to current TCK process guide.
Browse files Browse the repository at this point in the history 
Signed-off-by: Scott M Stark <starksm64@gmail.com>
  • Loading branch information
@starksm64 @bshannon
starksm64 authored and bshannon committed Jun 26, 2019
1 parent f664cee commit 2fe4f7e
Showing 1 changed file with 67 additions and 130 deletions.
197 changes: 67 additions & 130 deletions user_guides/Template/src/main/jbake/content/rules.adoc
View file
Expand Up @@ -293,142 +293,79 @@ include::rules.inc[]
2.3 Test Appeals Process
~~~~~~~~~~~~~~~~~~~~~~~~

Oracle has a well established process for managing challenges to its
Java technology Test Suites and plans to continue using a similar
process in the future. Oracle, as {TechnologyShortName} Maintenance Lead, will
authorize representatives from the Java Partner Engineering group to be
the point of contact for all test challenges. Typically this will be the
engineer assigned to a company as part of its {TechnologyShortName} TCK support.

If a test is determined to be invalid in function or if its basis in the
specification is suspect, the test may be challenged by any licensee of
the {TechnologyShortName} TCK. Each test validity issue must be covered
by a separate test challenge. Test validity or invalidity will be
determined based on its technical correctness such as:

* Test has bugs (i.e., program logic errors).
* Specification item covered by the test is ambiguous.
* Test does not match the specification.
* Test assumes unreasonable hardware and/or software requirements.
* Test is biased to a particular implementation.
Challenges based upon issues unrelated to technical correctness as
defined by the specification will normally be rejected.

Test challenges must be made in writing to Java Partner Engineering and
include all relevant information as described in link:#CJAFGAEE[Example
2-1, "Test Challenge Form"]. The process used to determine the validity
or invalidity of a test (or related group of tests) is described in
link:#CJAJEAEI[Section 2.3.1, "TCK Test Appeals Steps."]

All tests found to be invalid will either be placed on the Exclude List
for that version of the {TechnologyShortName} TCK or have an alternate
test made available.

* Tests that are placed on the Exclude List will be placed on the
Exclude List within one business day after the determination of test
validity. The new Exclude List will be made available to all
{TechnologyShortName} TCK licensees on the Java Licensee Engineering website.
* Oracle, as Maintenance Lead has the option of creating alternative
tests to address any challenge. Alternative tests (and criteria for
their use) will be made available on the Java Licensee Engineering
website. +
[NOTE]
=======================================================================
Passing an alternative test is deemed equivalent to passing the original
test.
=======================================================================
Jakarta has a well established process for managing challenges to its
TCKs. Any implementor may submit a challenge to one or more tests in the
{TechnologyShortName} TCK as it relates to their implementation. Implementor
means the entity as a whole in charge of producing the final certified release.
*Challenges filed should represent the consensus of that entity*.

2.3.1 Valid Challenges
^^^^^^^^^^^^^^^^^^^^^^
Any test case (e.g., test class, @Test method), test case configuration (e.g., deployment descriptor), test beans, annotations, and other resources considered part of the TCK may be challenged.

The following scenarios are considered in scope for test challenges:

* Claims that a test assertion conflicts with the specification.
* Claims that a test asserts requirements over and above that of the specification.
* Claims that an assertion of the specification is not sufficiently implementable.
* Claims that a test is not portable or depends on a particular implementation.
2.3.2 Invalid Challenges
^^^^^^^^^^^^^^^^^^^^^^^^
The following scenarios are considered out of scope for test challenges and will be immediately closed if filed:

* Challenging an implementation’s claim of passing a test. Certification is an honor system and these issues must be raised directly with the implementation.
* Challenging the usefulness of a specification requirement. The challenge process cannot be used to bypass the specification process and raise in question the need or relevance of a specification requirement.
* Claims the TCK is inadequate or missing assertions required by the specification. See the Improvement section, which is outside the scope of test challenges.
* Challenges that do not represent a consensus of the implementing community will be closed until such time that the community does agree or agreement cannot be made. The test challenge process is not the place for implementations to initiate their own internal discussions.
* Challenges to tests that are already excluded for any reason.
* Challenges that an excluded test should not have been excluded and should be re-added should be opened as a new enhancement request
Test challenges must be made in writing via the {TechnologyShortName} specification project issue tracker
as described in link:#CJAJEAEI[Section 2.3.3, "TCK Test Appeals Steps."]

All tests found to be invalid will be placed on the Exclude List
for that version of the {TechnologyShortName} TCK.


[[CJAJEAEI]][[tck-test-appeals-steps]]

2.3.1 TCK Test Appeals Steps
2.3.3 TCK Test Appeals Steps
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1. {TechnologyFullName} TCK licensee writes a test
challenge to Java Licensee Engineering contesting the validity of one or
a related set of {TechnologyShortName} tests. +
A detailed justification for why each test should be invalidated must be
included with the challenge as described in link:#CJAFGAEE[Example 2-1,
"Test Challenge Form"].
2. Java Licensee Engineering evaluates the challenge. +
If the appeal is incomplete or unclear, it is returned to the submitting
licensee for correction. If all is in order, Java Licensee Engineering
will check with the responsible test developers to review the purpose
and validity of the test before writing a response as described in
link:#CJAGGCIF[Example 2-2, "Test Challenge Response Form"]. Java
Licensee Engineering will attempt to complete the response within 5
business days. If the challenge is similar to a previously rejected test
challenge (i.e., same test and justification), Java Licensee Engineering
will send the previous response to the licensee.
3. The challenge and any supporting materials from test developers is
sent to the specification engineers for evaluation. +
A decision of test validity or invalidity is normally made within 15
working days of receipt of the challenge. All decisions will be
documented with an explanation of why test validity was maintained or
rejected.
4. The licensee is informed of the decision and proceeds accordingly. +
If the test challenge is approved and one or more tests are invalidated,
Oracle places the tests on the Exclude List for that version of the
{TechnologyFullName} TCK (effectively removing the test(s) from
the Test Suite). All tests placed on the Exclude List will have a bug
report written to document the decision and made available to all
licensees through the bug reporting database. If the test is valid but
difficult to pass due to hardware or operating system limitations,
Oracle may choose to provide an alternate test to use in place of the
original test (all alternate tests are made available to the licensee
community).
5. If the test challenge is rejected, the licensee may choose to
escalate the decision to the Executive Committee (EC), however, it is
expected that the licensee would continue to work with Oracle to resolve
the issue and only involve the EC as a last resort.
[[sthref8]][[test-challenge-and-response-forms]]

2.3.2 Test Challenge and Response Forms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

link:#CJAFGAEE[Example 2-1] shows the test challenge information you
must provide to Java Licensee Engineering to initiate a challenge, and
link:#CJAGGCIF[Example 2-2] shows the test challenge response format.

[[CJAFGAEE]]

Example 2-1 Test Challenge Form

[source,oac_no_warn]
----
Test Challenger Name and Company:
Specification Name(s) and Version(s):
Test Suite Name and Version:
Exclude List Version:
Test Name:
Complaint (argument for why test is invalid):
.jtr file of the failing test:
Console log of the JavaTest harness and device with all debugging flags turned on (if applicable):
.jti or .jte file for the test run:
Startup scripts for the JavaTest harness and agent (if applicable):
----

[[CJAGGCIF]]

Example 2-2 Test Challenge Response Form

[source,oac_no_warn]
----
Test Defender Name and Company:
Test Defender Role in Defense (e.g., test developer, Maintenance Lead, etc.):
Specification Name(s) and Version(s):
Test Suite Name and Version:
Test Name:
Defense (argument for why test is valid):
[Multiple challenges and corresponding responses may be listed here.]
Implications of test invalidity (e.g., other affected tests and test framework code, creation or exposure of ambiguities in spec (due to unspecified requirements), invalidation of the reference implementation, creation of serious holes in test suite):
Alternatives (e.g., are alternate test(s) appropriate?):
----
1. Challenges should be filed via the {TechnologyFullName} specification project’s issue tracker using the label `challenge` and include the following information:
* The relevant specification version and section number(s)
* The coordinates of the challenged test(s)
* The exact TCK and exclude list versions
* The implementation being tested, including name and company
* The full test name
* A full description of why the test is invalid and what the correct behavior is believed to be
* Any supporting material; debug logs, test output, test logs, run scripts, etc.
2. Specification project evaluates the challenge. +
Challenges can be resolved by a specification project lead, or a project challenge triage team, after a consensus of the specification project committers is reached or attempts to gain consensus fails.
Specification projects may exercise lazy consensus, voting or any practice that follows the principles of Eclipse Foundation Development Process.
The expected timeframe for a response is two weeks or less.
If consensus cannot be reached by the specification project for a prolonged period of time, the default recommendation is to exclude the tests and address the dispute in a future revision of the specification.
3. Accepted Challenges. +
A consensus that a test produces invalid results will result in the exclusion of that test from certification requirements, and an immediate update and release of an official distribution of the TCK including the new exclude list. The associated `challenge` issue must be closed with an `accepted` label to indicate it has been resolved.
4. Rejected Challenges and Remedy. +
When a`challenge` issue is rejected, it must be closed with a label of `invalid` to indicate it has been rejected.
There appeal process for challenges rejected on technical terms is outlined in Escalation Appeal.
If, however, an implementer feels the TCK challenge process was not followed, an appeal issue should be filed with specification project’s TCK issue tracker using the label `challenge-appeal`.
A project lead should escalate the issue with the Jakarta EE Specification Committee via email (jakarta.ee-spec.committee@eclipse.org).
The committee will evaluate the matter purely in terms of due process.
If the appeal is accepted, the original TCK challenge issue will be reopened and a label of `appealed-challenge` added, along with a discussion of the appeal decision, and the `challenge-appeal` issue with be closed.
If the appeal is rejected, the `challenge-appeal` issue should closed with a label of `invalid`.
5. Escalation Appeal. +
If there is a concern that a TCK process issue has not been resolved satisfactorily, the
https://www.eclipse.org/projects/dev_process/#6_5_Grievance_Handling[Eclipse Development Process Grievance Handling] procedure should be followed to escalate the resolution. Note that this is not a mechanism to attempt to handle implementation specific issues.
[[CJAJECIE]][[specifications-for-technology]]

Expand Down

0 comments on commit 2fe4f7e

Please sign in to comment.