-
Notifications
You must be signed in to change notification settings - Fork 15
Creating tests for the i18n test suite
The Internationalization Working Group has created a large number of tests, accessed via a test framework, and provides results for those tests. To find tests and results, see the test index page. We port tests to the Web Platform Tests repository, but some of our tests are exploratory or test features not described in W3C specifications.
Basically, the test format we use conforms to that of the WPT documentation, and you should be familiar with those guidelines. You should also look at existing tests in the test suite to get some ideas about how to create similar tests. There are also some differences between our approach to test development and that of WPT, because our audience is different. Here we provide some essential guidelines for i18n tests.
Here is an example of a typical, short test file. See the live test in the test harness.
<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="utf-8">
<title>arabic text-shadow</title>
<meta name="assert" id="assert" content="[exploratory] text-shadow on cursive Arabic text doesn't break the joins">
<link rel="help" href="https://drafts.csswg.org/css-text-decor-3/#text-shadow-property">
<link rel="prev" href="../../results/text-shadow.html">
<link rel="top" href="../../index.html#css3-text-decor">
<link rel="author" title="r12a" href="mailto:r12a@w3.org">
<script src="../../fonts/metadata_code.js"></script>
<link rel="stylesheet" href="../../fonts/metadata_styles.css"/>
<style>/* not part of the test */
@font-face {
font-family: 'webfont';
src: url('../../fonts/noto/NotoNaskhArabic-regular.woff2') format('woff2');
font-weight: normal;
font-style: normal;
}
.test { font-family: webfont; margin: .5em 5%; font-size: 8em; }
</style>
<style>/* part of the test */
#test { text-shadow: -6px 4px 2px pink; }
</style>
</head>
<body>
<div>
<em>Skip if <span style="text-shadow: 2px 2px 2px grey;" id="control">this text</span> has no shadow.</em><br>
Test passes if there are no pink lines between joined characters.
</div>
<div class="test" id="test" lang="ar">نشاط التدويل</div>
<div id="info">
<details>
<summary>Show notes</summary>
<p id="assertion"></p>
<ul class="notes">
<li><code>text-shadow</code> is set to <code>-6px 4px 2px pink</code>.</li>
<li>The 'skip' directive checks that <code>text-shadow</code> is supported. If it is not supported, the test is invalid.</li>
</ul>
<p id="links"></p>
</details>
</div>
<script>addToInfo()</script>
</body>
</html>Every file should have lang="en" in the HTML tag (unless it is testing language), and should have <meta charset="utf-8"/> (unless testing character encodings).
No line in the file should end with superfluous white space. If you are testing spaces at the end of a line, use  to represent the space. (This rule exists because WPT tests require it.)
Every file should contain the following, worded as concisely as possible (without being obscure!):
- a title element
- a
meta name="assert" id="assert"element. If this is an exploratory test (ie. not testing a documented spec feature) the assertion should begin with[exploratory]. - a
link rel="help"element - a
link rel="author"element -
<script src="../../fonts/metadata_code.js"></script>, to copy metadata to the explanatory section (removed during port to WPT repo) -
<link rel="stylesheet" href="../../fonts/metadata_styles.css"/>, to set styles for incidental information (removed during port to WPT repo) - a
divat the start of the body that gives instructions on how to tell if the test passes/fails - the test itself
- an
infosection that grabs the assertion and describes how the test works. This is collapsed by default.
All test files should test a single, specific thing. If you want to test multiple aspects of an issue, create multiple test files.
Let's look at these in a little more detail.
Although WPT tests are expected to be minimal in terms of displayed information, tests in the i18n test suite are designed to be informative, so that they can be readily understood when linked to from educational resources or gap-analysis documents, etc. (When a test is ported to the WPT repository, the superfluous information is removed, so it is good to keep that information on separate lines, to make it easy for the person converting the test.)
The example test above shows a 'skip directive'. These are control tests. In the example above, it's important to establish whether text-shadow works at all, before looking at its effect on cursive scripts. If it doesn't work, the test result is invalid. There may be more than one skip directive. (While porting to WPT format, these control tests may be converted to associated ref tests.)
At the bottom of the test is an info div. The first paragraph captures the assertion and displays it. There should be no need to change that paragraph.
After the paragraph is a list. The first item in the list should briefly describe how the test works. If there is a skip directive, there should be a list item describing its relevance. Other list items may be needed to indicate other relevant information.
The element with id 'helpLinks' is a placeholder for the help metadata to be automatically added to the explanatory information. There is no need to edit this element.
The title element is used in the test suite to help people spot a particular test at a glance. Ideally, every title should be unique to its test, and give an idea of what the test is about and how it differs from others. Conventionally, i18n tests use all lowercase.
In the i18n test results pages, if you include a colon in the title, only the text following the colon is displayed. For example, <title>CSS3 Text, text transform: capitalize, Dutch IJ</title> will be displayed as capitalize, Dutch IJ in the left column of a results table.
You must always include an assertion. The assertion tells you what the test sets out to check, and is one of the most important parts of the test. Without an assertion, reviewers and people looking at the results page cannot be sure what the test is about.
Writing an assertion is also good for you as a test writer, because it ensures that you have a clear idea what specifically you are setting out to test, and helps you ensure that you are testing a single thing. The assertion should be as concise as possible.
If this is an exploratory test, rather than one that tests a specific part of a specification, you should start the assertion with "[exploratory] ". Be sure to spell it correctly, as the test framework looks for this.
Use the href attribute to point to the section in the spec that describes the feature you are testing. This is picked up by the test framework.
If this is an exploratory test (so there is no actual spec text), try to point to a section of a spec that describes something similar.
This is useful if we need to change the test later, since it allows us to notify or consult the original author, where needed.
If you modify a test, you should add another such meta element.
When testing CSS features, it's usually helpful to add the relevant declarations at the start of the style element, and indicate which style declarations are not actually relevant to the test itself (see the example above).
When a test is displayed, people will want to know whether it passes or fails. It is essential that you indicate, as concisely as possible, what conditions indicate a pass.
The instructions will be displayed at the top of the page, and should always contain "Test passes if ...". You may precede or follow that with brief instructions, such as "Double-click on the middle word."
If you want to add further details, or provide controls for the use of the person looking at the test, include these in a separate div immediately below. Further details may be statements such as "Skip this test is vertical text is not supported."
The markup for a test is typically included in a div with an id and class called test.
The tests we develop for i18n are different from WPT because they are used educationally and target authors, they are not just for machine testing. This creates differences such as the following:
- We are most interested in tests that show whether the basic functionality works, and less interested in edge-cases.
- We prefer tests that allow you to see the feature in action, especially if it involves writing systems, rather than just producing a green box via a script. Therefore many of our tests use actual text in a given writing system, and we sometimes create the same test using a variety of writing systems for the text.
- Many of the tests we run cannot be automated.
- We prefer overlaying tests and references, where feasible, because it makes it easier for humans to grok the differences. This doesn't work as well for WPT variants of the test, since there tends to be some subpixel leaking of colour, so we need to change those tests before porting them to WPT. Hopefully, it's just a question of taking out the CSS that does the overlay.
- We need to be able to group and arrange test results in a way that ‘tells a story’ or illustrates how a feature works.
- We are less interested in nightly results than in results for production versions of browsers – the latter are the main focus for content authors. If we do record results for nightlies, we also record them for the latest released version of a browser.
- As previously described, WPT tests are minimal in terms of displayed information, whereas i18n test suite tests provide explanations about how the tests work for the reader.
This section provides some bits of code that may be useful.
Some special fonts are available for tests, including monospaced fonts. If you need to use them, check with the W3C staff.
It is often useful to use a webfont, either to avoid spatial variation across devices, or sometimes just to ensure that the person running a test can see all the characters. We typically use webfonts derived from Google's Noto fonts. Here is an example of some code you would put at the start of your style element for that:
@font-face {
font-family: 'webfont';
src: url('../../../fonts/noto/NotoSerifDevanagari-regular.woff2') format('woff');
font-weight: normal;
font-style: normal;
}
.test, .ref { font-family: webfont, serif; ... }In some cases, it is useful to change the font used in the test, especially for exploratory tests. Here is some code you can use for that.
<p>Change the font to <input id="fontFamily" type="text" onChange="setFont(this.value)"/></p>If you think it's useful to provide a list of fonts, you could use the following code:
<div class="control">Change the font:
<select onchange="document.getElementById('test').style.fontFamily = this.value; return false;">
<option value="0">Select...</option>
<option value="Al Bayan">Al Bayan</option>
<option value="Baghdad">Baghdad</option>
<option value="Decotype Naskh">Decotype Naskh</option>
<option value="Geeza Pro">Geeza Pro</option>
<option value="KufiStandardGK">KufiStandardGK</option>
<option value="Nadeem">Nadeem</option>
<option value="Damascus">Damascus</option>
...
</select>
</div>To allow the person viewing the test to increase/decrease the font size, you could use the following code. Note that you should only do this where there is some need (typically in exploratory tests only).
<p>Set font size <input id="fontSize" min="10" max="120" step="1" value="36" oninput="document.getElementById('sizeOfFont').textContent = this.value; tests = document.querySelectorAll('.test'); for (let i=0;i<tests.length;i++) { tests[i].style.fontSize = this.value+'px';}" style="width:40%; margin-left: 5%;" type="range"/> <span id="sizeOfFont" style="font-size:80%;">36</span>px</p>This is an example of a test that writes the text of the test over the reference text (relevant lines are marked with an initial *). The latter is coloured red, so that any differences show through. Particularly where there are several things to check (such as in text transform tests), this makes it much easier for people to see the differences, however, as noted above, these tests will need to be changed before porting to the WPT repository.
<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="utf-8"/>
<title>text transform capitalize, Dutch IJ</title>
<meta name="assert" content="[exploratory] The brower tailors text-transform: capitalize to put both I and J in titlecase at the start of a word when the language is Dutch.">
<link rel='author' title='r12a' href='mailto:ishida@w3.org'>
<link rel='help' href='http://dev.w3.org/csswg/css-text-3/#text-transform'>
<link rel="match" href="reference/text-transform-tailoring-001-ref.html">
<style type='text/css'>
@font-face {
font-family: 'webfont';
src: url('../../fonts/sil/GentiumPlus-regular.woff2') format('woff');
font-weight: normal;
font-style: normal;
}
.test, .ref { font-size: 200%; line-height: 2.5em; font-family: 'Gentium Plus', 'Noto Serif', 'Noto Sans', webfont, sans-serif; }
.test span, .ref span { margin-right: 0; white-space: nowrap; }
span { display: inline-block; min-width: 2em; }
*.ref span { color: red; } .ref { z-index: -100; }
*.ref, .test { position:absolute; top:0; }
/* the CSS above is not part of the test */
.test { text-transform: capitalize; }
</style>
</head>
<body>
<p class="instructions">Test passes _if there is no red_.</p>
*<div style="position:relative">
<div class="test" lang="nl">ijsland</div>
<div class="ref">IJsland</div>
*</div>
</body>
</html>In some cases (eg. when exploring line breaking behaviour) it can be useful to allow the person reading the page to expand or shrink the width of the box containing the test data. Here is some code you could use for that.
<div id="advice"> Use the slider to adjust the width of the box.<br/>
<input id="fontSizeSlider" min="10" max="90" step=".25" value="90" oninput="document.getElementById('sizeIndicator').textContent = Math.floor(window.innerWidth/100*this.value); tests = document.querySelectorAll('.test'); for (let i=0;i<tests.length;i++) { tests[i].style.width = this.value+'%';}" style="width:80%; margin-left: 5%;" type="range"> <span id="sizeIndicator" style="font-size:80%;">90</span>px</div>
<script>document.getElementById('sizeIndicator').textContent = Math.floor(window.innerWidth/100*90)</script>That code will allow the user to simultaneously alter the width of all divs with the class name test.
This is done by the W3C staff person. They will bundle up your tests, or add them to other bundles, add them to the test framework, and create a results page.
Anyone can submit results for a test, or set of tests. Follow the online instructions while looking at a test, and then send the results to the W3C staff person.