web-platform-tests · jugglinmike · Aug 2, 2019 · Jul 10, 2019 · Jul 13, 2019 · Jul 13, 2019
diff --git a/docs/assets/reftest-tutorial-test-screenshot.png b/docs/assets/reftest-tutorial-test-screenshot.png
diff --git a/docs/writing-tests/index.md b/docs/writing-tests/index.md
@@ -18,6 +18,7 @@ There's also a load of [general guidelines](general-guidelines) that apply to al
    h2tests
    lint-tool
    manual
+   reftest-tutorial
    reftests
    rendering
    server-features

diff --git a/docs/writing-tests/reftest-tutorial.md b/docs/writing-tests/reftest-tutorial.md
@@ -0,0 +1,276 @@
+# Writing a reftest
+
+<!--
+Note to maintainers:
+
+This tutorial is designed to be an authentic depiction of the WPT contribution
+experience. It is not intended to be comprehensive; its scope is intentionally
+limited in order to demonstrate authoring a complete test without overwhelming
+the reader with features. Because typical WPT usage patterns change over time,
+this should be updated periodically; please weigh extensions against the
+demotivating effect that a lengthy guide can have on new contributors.
+-->
+
+Let's say you've discovered that WPT doesn't have any tests for the `dir`
+attribute of [the `<bdo>`
+element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/bdo). This
+tutorial will guide you through the process of writing and submitting a test.
+You'll need to [configure your system to use WPT's
+tools](../running-tests/from-local-system), but you won't need them until
+towards the end of this tutorial. Although it includes some very brief
+instructions on using git, you can find more guidance in [the tutorial for git
+and GitHub](../appendix/github-intro).
+
+WPT's reftests are great for testing web-platform features that have some
+visual effect. [The reftests reference page](reftests) describes them in the
+abstract, but for the purposes of this guide, we'll only consider the features
+we need to test the `<bdo>` element.
+
+```eval_rst
+.. contents::
+   :local:
+```
+
+## Setting up your workspace
+
+To make sure you have the latest code, first type the following into a terminal
+located in the root of the WPT git repository:
+
+    $ git fetch git@github.com:web-platform-tests/wpt.git
+
+Next, we need a place to store the change set we're about to author. Here's how
+to create a new git branch named `reftest-for-bdo` from the revision of WPT we
+just downloaded:
+
+    $ git checkout -b reftest-for-bdo FETCH_HEAD
+
+Now you're ready to create your patch.
+
+## Writing the test file
+
+First, we'll create a file that demonstrates the "feature under test." That is:
+we'll write an HTML document that displays some text using a `<bdo>` element.
+
+WPT has thousands of tests, so it can be daunting to decide where to put a new
+one. Generally speaking, [test files should be placed in directories
+corresponding to the specification text they are
+verifying](../test-suite-design). `<bdo>` is defined in [the "text-level
+semantics" chapter of the HTML
+specification](https://html.spec.whatwg.org/multipage/text-level-semantics.html),
+so we'll want to create our new test in the directory
+`html/semantics/text-level-semantics/the-bdo-element/`. Create a file named
+`rtl.html` and open it in your text editor.
+
+Here's one way to demonstrate the feature:
+
+```html
+<!DOCTYPE html>
+<meta charset="utf-8">
+<title>BDO element dir=rtl</title>
+<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
+<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
+
+<p>Test passes if WAS is displayed below.</p>
+<bdo dir="rtl">SAW</bdo>
+```
+
+That's pretty dense! Let's break it down:
+
+- ```html
+  <!DOCTYPE html>
+  <meta charset="utf-8">
+  ```
+
+  We explicitly set the DOCTYPE and character set to be sure that browsers
+  don't infer them to be something we aren't expecting. We're omitting the
+  `<html>` and `<head>` tags. That's a common practice in WPT, preferred
+  because it makes tests more concise.
+
+- ```html
+  <title>BDO element dir=rtl</title>
+  ```
+  The document's title should succinctly describe the feature under test.
+
+- ```html
+  <link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
+  ```
+
+  The "help" metadata should reference the specification under test so that
+  everyone understands the motivation. This is so helpful that [the CSS Working
+  Group requires it for CSS tests](css-metadata)! If you're writing a reftest
+  for a feature outside of CSS, feel free to omit this tag.
+
+- ```html
+  <meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
+  ```
+
+  The "assert" metadata is a structured way for you to describe exactly what
+  you want your reftest to verify. For a direct test like the one we're writing
+  here, it might seem a little superfluous. It's much more helpful for
+  more-involved tests where reviewers might need some help understanding your
+  intentions.
+
+  This tag is optional, so you can skip it if you think it's unnecessary. We
+  recommend using it for your first few tests since it may let reviewers give
+  you more helpful feedback. As you get more familiar with WPT and the
+  specifications, you'll get a sense for when and where it's better to leave it
+  out.
+
+- ```html
+  <p>Test passes if WAS is displayed below.</p>
+  ```
+
+  We're communicating the "pass" condition in plain English to make the test
+  self-describing.
+
+- ```html
+  <bdo dir="rtl">SAW</bdo>
+  ```
+
+  This is the real focus of the test. We're including some text inside a
+  `<bdo>` element in order to demonstrate the feature under test.
+
+Since this page doesn't rely on any [special WPT server
+features](server-features), we can view it by loading the HTML file directly.
+There are a bunch of ways to do this; one is to navigate to the
+`html/semantics/text-level-semantics/the-bdo-element/` directory in a file
+browser and drag the new `rtl.html` file into an open web browser window.
+
+![](/assets/reftest-tutorial-test-screenshot.png "screen shot of the new test")
+
+Sighted people can open that document and verify whether or not the stated
+expectation is satisfied. If we were writing a [manual test](manual), we'd be
+done. However, it's time-consuming for a human to run tests, so we should
+prefer making tests automatic whenever possible. Remember that we set out to
+write a "reference test." Now it's time to write the reference file.
+
+## Writing a "match" reference
+
+The "match" reference file describes what the test file is supposed to look
+like. Critically, it *must not* use the technology that we are testing. The
+reference file is what allows the test to be run by a computer--the computer
+can verify that each pixel in the test document exactly matches the
+corresponding pixel in the reference document.
+
+Make a new file in the same
+`html/semantics/text-level-semantics/the-bdo-element/` directory named
+`rtl-ref.html`, and save the following markup into it:
+
+```html
+<!DOCTYPE html>
+<meta charset="utf-8">
+<title>BDO element dir=rtl reference</title>
+
+<p>Test passes if WAS is displayed below.</p>
+<p>WAS</p>
+```
+
+This is like a stripped-down version of the test file. In order to produce a
+visual rendering which is the same as the expected rendering, it uses a `<p>`
+element whose contents is the characters in right-to-left order. That way, if
+the browser doesn't support the `<bdo>` element, this file will still show text
+in the correct sequence.
+
+This file is also completely functional without the WPT server, so you can open
+it in a browser directly from your hard drive.
+
+Currently, there's no way for a human operator or an automated script to know
+that the two files we've created are supposed to match visually. We'll need to
+add one more piece of metadata to the test file we created earlier. Open
+`html/semantics/text-level-semantics/the-bdo-element/rtl.html` in your text
+editor and add another `<link>` tag as described by the following change
+summary:
+
+```diff
+ <!DOCTYPE html>
+ <meta charset="utf-8">
+ <title>BDO element dir=rtl</title>
+ <link rel="author" title="Sam Smith" href="mailto:sam@example.com">
+ <link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
++<link rel="match" href="rtl-ref.html">
+ <meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
+
+ <p>Test passes if WAS is displayed below.</p>
+ <bdo dir="rtl">SAW</bdo>
+```
+
+Now, anyone (human or computer) reviewing the test file will know where to find
+the associated reference file.
+
+## Verifying our work
+
+We're done writing the test, but we should make sure it fits in with the rest
+of WPT before we submit it. This involves using some of the project's tools, so
+this is the point you'll need to [configure your system to run
+WPT](../running-tests/from-local-system).
+
+[The lint tool](lint-tool) can detect some of the common mistakes people make
+when contributing to WPT. To run it, open a command-line terminal, navigate to
+the root of the WPT repository, and enter the following command:
+
+    python ./wpt lint html/semantics/text-level-semantics/the-bdo-element
+
+If this recognizes any of those common mistakes in the new files, it will tell
+you where they are and how to fix them. If you do have changes to make, you can
+run the command again to make sure you got them right.
+
+Now, we'll run the test using the automated pixel-by-pixel comparison approach
+mentioned earlier. This is important for reftests because the test and the
+reference may differ in very subtle ways that are hard to catch with the naked
+eye. That's not to say your test has to pass in all browsers (or even in *any*
+browser). But if we expect the test to pass, then running it this way will help
+us catch other kinds of mistakes.
+
+The tools support running the tests in many different browsers. We'll use
+Firefox this time:
+
+    python ./wpt run firefox html/semantics/text-level-semantics/the-bdo-element/rtl.html
+
+We expect this test to pass, so if it does, we're ready to submit it. If we
+were testing a web platform feature that Firefox didn't support, we would
+expect the test to fail instead.
+
+There are a few problems to look out for in addition to passing/failing status.
+The report will describe fewer tests than we expect if the test isn't run at
+all. That's usually a sign of a formatting mistake, so you'll want to make sure
+you've used the right file names and metadata. Separately, the web browser
+might crash. That's often a sign of a browser bug, so you should consider
+[reporting it to the browser's
+maintainers](https://rachelandrew.co.uk/archives/2017/01/30/reporting-browser-bugs/)!
+
+## Submitting the test
+
+First, let's stage the new files for committing:
+
+    $ git add html/semantics/text-level-semantics/the-bdo-element/rtl.html
+    $ git add html/semantics/text-level-semantics/the-bdo-element/rtl-ref.html
+
+We can make sure the commit has everything we want to submit (and nothing we
+don't) by using `git diff`:
+
+    $ git diff --staged
+
+On most systems, you can use the arrow keys to navigate through the changes,
+and you can press the `q` key when you're done reviewing.
+
+Next, we'll create a commit with the staged changes:
+
+    $ git commit -m '[html] Add test for the `<bdo>` element'
+
+And now we can push the commit to our fork of WPT:
+
+    $ git push origin reftest-for-bdo
+
+The last step is to submit the test for review. WPT doesn't actually need the
+test we wrote in this tutorial, but if we wanted to submit it for inclusion in
+the repository, we would create a pull request on GitHub. [The guide on git and
+GitHub](../appendix/github-intro) has all the details on how to do that.
+
+## More practice
+
+Here are some ways you can keep experimenting with WPT using this test:
+
+- Improve coverage by adding more tests for related behaviors (e.g. nested
+  `<bdo>` elements)
+- Add another reference document which describes what the test should *not*
+  look like using [`rel=mismatch`](reftests)
diff --git a/docs/writing-tests/reftests.md b/docs/writing-tests/reftests.md
@@ -3,7 +3,9 @@
 Reftests are one of the primary tools for testing things relating to
 rendering; they are made up of the test and one or more other pages
 ("references") with assertions as to whether they render identically
-or not.
+or not. This page describes their aspects exhaustively; [the tutorial
+on writing a reftest](reftest-tutorial) offers a more limited but
+grounded guide to the process.
 
 ## How to Run Reftests
 
@@ -180,46 +182,5 @@ element works, it's possible to construct a reftest for underlining
 a block element, by constructing a reference using underlines on a
 ```<span>``` that wraps all the content inside the block.
 
-## Example Reftests
-
-This example follows the recommended approach in being a
-self-describing test as it has a simple statement on the page
-describing how it should render to pass the tests.
-
-### Test File
-
-This test verifies that a right-to-left rendering of **SAW** within a
-```<bdo>``` element displays as **WAS**.
-
-```html
-<!DOCTYPE html>
-<meta charset="utf-8">
-<title>BDO element dir=rtl</title>
-<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
-<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
-<link rel="match" href="test-bdo-001.html">
-<p>Pass if you see WAS displayed below.</p>
-<bdo dir="rtl">SAW</bdo>
-```
-
-### Reference File
-
-The reference file must look exactly like the test file,
-except that the code behind it is different.
-
-* All metadata is removed.
-* The ```title``` need not match.
-* The markup that created the actual test data is
-  different: here, the same effect is created with
-  very mundane, dependable technology.
-
-```html
-<!DOCTYPE html>
-<meta charset="utf-8">
-<title>HTML Reference File</title>
-<p>Pass if you see WAS displayed below.</p>
-<p>WAS</p>
-```
-
 [general guidelines]: general-guidelines
 [rendering]: rendering