Skip to content
Test suite for the Web Media API Snapshot Specification
Branch: wmas2018
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.well-known Origin Policy: Add support for Feature Policy. Nov 19, 2018
common Make Server-Timing tests more resilient (#13789) Nov 2, 2018
docs testharness.md: Remove errant spaces in file paths Dec 5, 2018
fonts Rename of OWNER files to META.yml Jun 12, 2018
images Update README.md May 9, 2019
interfaces Update interfaces/event-timing.idl (#14387) Dec 6, 2018
media Ship CreateImageBitmap resize options Aug 13, 2018
resources Merge branch 'feat/manual-timeout' into wmats2018 May 8, 2019
tools
.azure-pipelines.yml Rename "safari_webdriver" to "safari", get rid of "safari" (#14213) Nov 23, 2018
.codecov.yml Add codecov configutation. (#5623) Apr 24, 2017
.gitignore redo deleted .gitignore Mar 11, 2019
.gitmodules Fix #7200: Update vendored html5lib to 1.0.1 Jun 6, 2018
.pyup.yml Add requirements_flake8.txt to .pyup.yml Nov 5, 2018
.taskcluster.yml
.travis.yml
CODEOWNERS
CONTRIBUTING.md
LICENSE.md Fix the first header in the license file (#5244) Mar 28, 2017
README.md
check_stability.ini
lint.whitelist [streams] Refactor to multi-global test format (#14172) Nov 22, 2018
serve.py Add a warning when running ./serve when tools.serve cannot be importe… Oct 30, 2016
server-side.md Make tests self-hosting Jan 6, 2014
testharness_runner.html Add testharness_runner file here for now. Apr 17, 2015
update-built-tests.sh Delete unbuilt css3-selectors tests; move built tests up Aug 24, 2018
wave implemented statistics script; Jun 17, 2019
wmats2018-subset.sh update wmats2018-subset.sh to copy license files of external test suites Jun 5, 2019
wpt Add a wpt command as a frontend to all cli interactions. Jul 26, 2017
wpt.py Allow |python wpt <command>| to work on Windows. Aug 16, 2017

README.md

The Web Media API Test Suite 2018

The Web Media API Snapshot Test Suite 2018 (WMATS2018) is a test suite for the Web Media API Snapshot 2018 specification. The test suite and specification are being developed as part of the CTA WAVE Project.

This project is forked from W3C Web Platform Tests and is customized to automate test runs on web browsers for embedded devices and appliances suchs as TV sets, set-top boxes, consoles, etc.

A hosted version is available at: https://webapitests2018.ctawave.org

Test server

Architecture

architecure

WMATS2018 extends the WPT Runner in a way that only a single window is required to run the tests. The Test Runner itself is migrated to a Node.js Server. The embedded device runs individual tests and sends the results to the Node.js Test Runner via XHR, which update the internal state of the running test session and pushes the new updates to a companion page which shows the progress of a test session (The Companion Page offers also other helpful features like generating test reports, exporting test results, compare test sessions, etc. through an easy to use interface). Once a Test is completed, the Node.js Test Runner sends the URL of the next test which will be opened in the same browser window and the steps will be repeated until all tests are completed. The WPT test files are included without any changes.

Setup

Requirements:

  • Python 2.7+ (but not Python 3.x)
  • Node.js 8.x.x (LTS recommended)
  • Git 2.4+ (should support sparse checkout)
  • Bash script support (on windows we recommend Git BASH)

Generate hosts file:

$ ./wpt make-hosts-file | sudo tee -a /etc/hosts

on Windows:

$ python wpt make-hosts-file | Out-File %SystemRoot%\System32\drivers\etc\hosts -Encoding ascii -Append

Initialize WAVE Server:

$ ./wave init

Download test files according to WMAS2018 specification, call from WPT root directory:

$ ./wmats2018-subset.sh

To download the test results of the four reference browsers (Chromium, Firefox, Safari, Edge):

$ ./wave download-reference-results

Start WAVE Server:

$ ./wave start

Start Web Platform Test runner:

$ ./wpt serve

Open in Web browser (on the same host):

http://web-platform.test:8050

Remote Server

If you are planning to host the server on a different device than the DUT, make sure to replace 120.0.0.1 with your server IP and web-platform.test with your domain name in /etc/hosts.

Configuration

The default configuration is loaded from the config.default.json in the root directory. Configurations from the config.json in the same format as the config.default.jsonoverride those.

Provide a different location with --config < path_to_config > as a start parameter.

Test Run Parameters

It is possible to parameterize a test run with various query parameters provided with the query in the url of the initial request.

Example:

web-platform.test:8050/?path=/2dcontext&types=testharness

Query Parameters

Parameter Description Example
path Specify tests to run. Can be directory or file. Multiple paths can be chained by using , . Paths with leading '/' are interpreted as absolute paths, paths without as regular expressions. (Default: /) web-platform.test:8050/?path=/2dcontext
types What types of tests to run. Possible types: testharness, manual and reftest (Default: testharness. Please note that including manual or reftest might lead to unexpected behaviour on embedded devices. It is recommended to only use testharness for automated test runs) web-platform.test:8050/?types=testharness,manual
timeout Specify a server side timeout in ms after which a test with no result is timed out web-platform.test:8050/?timeout=65000
token and resume Providing a token of an unfinished session will resume it. web-platform.test:8050/?token=2fb0fb80-63db-4425-8a76-2ea3e6f8269d&resume=1

Certificates

By default pregenerated certificates for the web-platform.test domain are provided in tools/certs. If you wish to generate new certificates for any reason it's possible to use OpenSSL when starting the server, or starting a test run, by providing the --ssl-type=openssl argument to the wpt serve or wpt run commands.

If you installed OpenSSL in such a way that running openssl at a command line doesn't work, you also need to adjust the path to the OpenSSL binary. This can be done by adding a section to config.json like:

"ssl": {"openssl": {"binary": "/path/to/openssl"}}

Trusting Root CA

To prevent browser SSL warnings when running HTTPS tests locally, the web-platform-tests Root CA file cacert.pem in tools/certs must be added as a trusted certificate in your OS/browser.

You can’t perform that action at this time.