This directory contains the common infrastructure for the following tests (also referred below as projects).
- referrer-policy/
- mixed-content/
- upgrade-insecure-requests/
Subdirectories:
resources
: Serves JavaScript test helpers.subresource
: Serves subresources, with support for redirects, stash, etc. The subresource paths are managed bysubresourceMap
and fetched inrequestVia*()
functions inresources/common.js
.scope
: Serves nested contexts, such as iframe documents or workers. Used frominvokeFrom*()
functions inresources/common.js
.tools
: Scripts that generate test HTML files. Not used while running tests./referrer-policy/generic/subresource-test
: Sanity checking tests for subresource invocation (This is still placed outside common/)
The test generator (common/security-features/tools
) generates test HTML files from templates and a seed (spec.src.json
) that defines all the test scenarios.
The project (i.e. a WPT subdirectory, for example referrer-policy/
) that uses the generator should define per-project data and invoke the common generator logic in common/security-features/tools
.
This is the overview of the project structure:
common/security-features/
└── tools/ - the common test generator logic
└── template/ - the test files templates
project-directory/ (e.g. referrer-policy/)
├── spec.src.json
├── generic/
│ ├── test-case.sub.js - Per-project test helper
│ └── tools/
│ └── generator.py - Per-project generator script
└── gen/ - generated tests
Invoking project-directory/generic/tools/generate.py
will parse the spec JSON and determine which tests to generate (or skip) while using templates.
The repository already contains generated tests, so if you're making changes, see the Removing all generated tests section below, on how to remove them before you start generating tests which include your changes.
# Chdir into the project directory.
cd ~/web-platform-tests/project-directory
# Generate the test files under gen/ (HTMLs and .headers files).
./generic/tools/generate.py
# Add all generated tests to the repo.
git add gen/ && git commit -m "Add generated tests"
During the generation, the spec is validated by common/security-features/tools/spec_validator.py
. This is specially important when you're making changes to spec.src.json
. Make sure it's a valid JSON (no comments or trailing commas). The validator reports specific errors (missing keys etc.), if any.
Simply remove all files under project-directory/gen/
.
# Chdir into the project directory.
cd ~/web-platform-tests/project-directory
# Remove all generated tests.
rm -r gen/
Note: this section is currently obsolete. Only the release template is working.
The generator script ./generic/tools/generate.py
has two targets: release
and debug
.
-
Using release for the target will produce tests using a template for optimizing size and performance. The release template is intended for the official web-platform-tests and possibly other test suites. No sanity checking is done in release mode. Use this option whenever you're checking into web-platform-tests.
-
When generating for
debug
, the produced tests will contain more verbosity and sanity checks. Use this target to identify problems with the test suites when making changes locally. Make sure you don't check in tests generated with the debug target.
Note that release is the default target when invoking generate.py
.
The main test logic lives in project-directory/generic/test-case.sub.js
with helper functions defined in /common/security-features/resources/common.js
so you should probably start there.
For updating the test suites you will most likely do a subset of the following:
-
Add a new subresource type:
- Add a new sub-resource python script to
/common/security-features/subresource/
. - Add a sanity check test for a sub-resource to
referrer-policy/generic/subresource-test/
. - Add a new entry to
subresourceMap
in/common/security-features/resources/common.js
. - Add a new entry to
valid_subresource_names
in/common/security-features/tools/spec_validator.py
. - Add a new entry to
subresource_schema
inspec.src.json
. - Update
source_context_schema
to specify in which source context the subresource can be used.
- Add a new sub-resource python script to
-
Add a new subresource redirection type
- TODO: to be documented. Example: web-platform-tests/wpt#18939
-
Add a new subresource origin type
- TODO: to be documented. Example: web-platform-tests/wpt#18940
-
Add a new source context (e.g. "module sharedworker global scope")
- TODO: to be documented. Example: web-platform-tests/wpt#18904
-
Add a new source context list (e.g. "subresource request from a dedicated worker in a
<iframe srcdoc>
")- TODO: to be documented.
-
Implement new or update existing assertions in
project-directory/generic/test-case.sub.js
. -
Exclude or add some tests by updating
spec.src.json
test expansions. -
Implement a new delivery method.
- TODO: to be documented. Currently the support for delivery methods are implemented in many places across
common/security-features/
.
- TODO: to be documented. Currently the support for delivery methods are implemented in many places across
-
Regenerate the tests and MANIFEST.json
For examples of spec JSON files, see referrer-policy/spec.src.json or mixed-content/spec.src.json.
-
specification
Top level requirements with description fields and a
test_expansion
rule. This is closely mimicking the Referrer Policy specification structure. -
excluded_tests
List of
test_expansion
patterns expanding into selections which get skipped when generating the tests (aka. blacklisting/suppressing) -
test_expansion_schema
Provides valid values for each field. Each test expansion can only contain fields and values defined by this schema (or
"*"
values that indicate all the valid values defined this schema). -
subresource_schema
Provides metadata of subresources, e.g. supported delivery types for each subresource.
-
source_context_schema
Provides metadata of each single source context, e.g. supported delivery types and subresources that can be sent from the context.
-
source_context_list_schema
Provides possible nested combinations of source contexts. See Source Contexts section below for details.
Each field in a test expansion can be in one of the following formats:
-
Single match:
"value"
-
Match any of:
["value1", "value2", ...]
-
Match all:
"*"
NOTE: An expansion is always constructive (inclusive), there isn't a negation operator for explicit exclusion. Be aware that using an empty list []
matches (expands into) exactly nothing. Tests which are to be excluded should be defined in the excluded_tests
section instead.
A single test expansion pattern, be it a requirement or a suppressed pattern, gets expanded into a list of selections as follows:
-
Expand each field's pattern (single, any of, or all) to list of allowed values (defined by the
test_expansion_schema
) -
Permute - Recursively enumerate all selections across all fields
Be aware that if there is more than one pattern expanding into a same selection, the pattern appearing later in the spec JSON will overwrite a previously generated selection. To make sure this is not undetected when generating, set the value of the expansion
field to default
for an expansion appearing earlier and override
for the one appearing later.
A selection is a single test instance (scenario) with explicit values that defines a single test. The scenario is then evaluated by the TestCase
in JS. For the rest of the arranging part, examine /common/security-features/tools/generate.py
to see how the values for the templates are produced.
Taking the spec JSON, the generator follows this algorithm:
-
Expand all
excluded_tests
to create a blacklist of selections -
For each specification requirement: Expand the
test_expansion
pattern into selections and check each against the blacklist, if not marked as suppresed, generate the test resources for the selection
In source_context_list_schema
, we can specify
-
source contexts from where subresource requests are sent, and
-
how policies are delivered, by source contexts and/or subresource requests.
-
sourceContextList
: an array ofSourceContext
objects, and -
subresourcePolicyDeliveries
: an array ofPolicyDelivery
objects.
They have the same object format as described in
common/security-features/resources/common.js
comments, and are directly
serialized to generated HTML files and passed to JavaScript test code,
except that:
- The first entry of
sourceContextList
'ssourceContextType
should be alwaystop
, which represents the top-level generated test HTML. (This entry is omitted in the JSON passed to JavaScript, but the policy deliveries specified here are written as e.g.<meta>
elements in the generated test HTML or HTTP headers) - Instead of
PolicyDelivery
object (insourceContextList
orsubresourcePolicyDeliveries
), following placeholder strings can be used.
The keys of source_context_list_schema
can be used as the values of
source_context_list
fields, to indicate which source context configuration
to be used.
Each test contains
delivery_key
(derived from the top-leveldelivery_key
) anddelivery_value
,delivery_type
(derived fromtest_expansion
),
which represents the target policy delivery, the policy delivery to be tested.
The following placeholder strings in source_context_list_schema
can be used:
"policy"
:- Replaced with the target policy delivery.
- Can be used to specify where the target policy delivery should be delivered.
"policyIfNonNull"
:- Replaced with the target policy delivery, only if it has non-null value. If the value is null, then the test file is not generated.
"anotherPolicy"
:- Replaced with a
PolicyDelivery
object that has a different value from the target policy delivery. - Can be used to specify e.g. a policy that should be overridden by the target policy delivery.
- Replaced with a
For example, when the target policy delivery is
{deliveryType: "http-rp", key: "referrerPolicy", value: "no-referrer"}
,
"sourceContextList": [
{
"sourceContextType": "top",
"policyDeliveries": [
"anotherPolicy"
]
},
{
"sourceContextType": "classic-worker",
"policyDeliveries": [
"policy"
]
}
]
is replaced with
"sourceContextList": [
{
"sourceContextType": "top",
"policyDeliveries": [
{
"deliveryType": "meta",
"key": "referrerPolicy",
"value": "unsafe-url"
}
]
},
{
"sourceContextType": "classic-worker",
"policyDeliveries": [
{
"deliveryType": "http-rp",
"key": "referrerPolicy",
"value": "no-referrer"
}
]
}
]
which indicates
- The top-level Document has
<meta name="referrer" content="unsafe-url">
. - The classic worker is created with
Referrer-Policy: no-referrer
HTTP response headers.
These represent supported delivery types and subresources for each source context or subresource type. These are used
- To filter out test files for unsupported combinations of delivery types, source contexts and subresources.
- To determine what delivery types should be used for
anotherPolicy
placeholder.