Permalink
Switch branches/tags
MultipartContent annevk-navigate-to-weird-schemes annevk/EventSource-U+0000 annevk/autofocus annevk/bc-closing annevk/content-length annevk/document-domain annevk/document-open-load-event annevk/document-open-steps-readiness annevk/document-write-reloading annevk/dom-listener-order annevk/encodeInto annevk/fetch-fragments annevk/h1-message-parsing annevk/html-ancestororigins annevk/http-versioning annevk/legacy-pre-activation-behavior-phase annevk/meta-global-annevk annevk/navigate-redirect annevk/navigate-source annevk/opaque-redirect-handling annevk/origin-header annevk/simplify-license annevk/sw-redirect-fragment annevk/template-content-node-document annevk/url-host-parser annevk/window.opener annevk/www-authenticate-parsing annevk/x-frame-options annevk/xhr-h2-statusText annevk/xhr-no-headers-received annevk/xhr-responseurl-with-fragment appveyor ayg/document-named-item bburg-unidentified-keys billing-address-redactions canvas/getContext-second-argument chrome_experimental chromium-export-cl-532595 chromium-export-cl-558917 chromium-export-cl-660842 chromium-export-cl-670844 chromium-export-cl-676883 chromium-export-cl-728365 chromium-export-cl-738353 chromium-export-cl-741361 chromium-export-cl-766373 chromium-export-cl-773321 chromium-export-cl-775179 chromium-export-cl-778160 chromium-export-cl-854092 chromium-export-cl-881121 chromium-export-cl-882906 chromium-export-cl-899411 chromium-export-cl-930542 chromium-export-cl-961719 chromium-export-cl-985959 chromium-export-cl-986593 chromium-export-cl-993812 chromium-export-cl-1013322 chromium-export-cl-1032554 chromium-export-cl-1118750 chromium-export-cl-1128385 chromium-export-cl-1136543 chromium-export-cl-1144133 chromium-export-cl-1146644 chromium-export-cl-1154225 chromium-export-cl-1188018 chromium-export-cl-1188643 chromium-export-cl-1199810 chromium-export-cl-1200863 chromium-export-cl-1212270 chromium-export-cl-1221696 chromium-export-cl-1222913 chromium-export-cl-1249391 chromium-export-cl-1264165 chromium-export-cl-1264520 chromium-export-cl-1270699 chromium-export-cl-1272339 chromium-export-cl-1277666 chromium-export-cl-1288573 chromium-export-cl-1291229 chromium-export-cl-1293050 chromium-export-cl-1293256 chromium-export-cl-1294874 chromium-export-cl-1295618 chromium-export-cl-1306337 chromium-export-cl-1306967 chromium-export-cl-1319616 chromium-export-cl-1322260 chromium-export-cl-1334659 chromium-export-cl-1338461 chromium-export-cl-1344261 chromium-export-cl-1355923 chromium-export-cl-1356966 chromium-export-cl-1363178 chromium-export-cl-1363971 chromium-export-cl-1365235 chromium-export-cl-1365250 chromium-export-cl-1366318 chromium-export-cl-1367725 chromium-export-cl-1367910 chromium-export-cl-1367935 chromium-export-cl-1368826 chromium-export-cl-1371392 chromium-export-cl-1372359 chromium-export-cl-1372556 chromium-export-cl-1373819 chromium-export-cl-1373848 chromium-export-cl-1374113 chromium-export-cl-1374352 chromium-export-cl-1374742 chromium-export-cl-1374988 chromium-export-cl-1375714 chromium-export-cl-I03bf57a24da3a7479199d04a05c4487342af20ca chromium-export-cl-I53e5b576adebf65ede8ebd1dabe084713ebdf875 chromium-export-cl-I865bb06f8b879e180d4aa6de57010e2609afb8b1 chromium-export-cl-I944ea2ea69447c612c01b9e6f723f110fa28a1f5 chromium-export-cl-I44767f3f65065e9fe0d84a051d4a6b503b439b89 chromium-export-cl-I05157521c98b50fc3816932045a13f6b0eefc9f0 chromium-export-cl-I9846168be36b1de26cc2a508e134c3dd555dafa7 chromium-export-cl-Ib8e35215c5b58721a043087130c9951a56fba3c1 chromium-export-cl-Ib629d1b7e8b86faffc0ddd51888f721df9da4d22 chromium-export-cl-Id7060b7d1292654fc8f23dd72eef629f6a874ef0 chromium-export-cl-Ie2ca10581a3e28974a1eff4e589ea8b62f4cfa62 chromium-export-cl-Ieaa430d9303ab7277440d96b61327b4ec997cd04 chromium-export-cl-Ied400facf474ad18f632c650b636bbbcf725b116 code-of-conduct.md codecov copy_pasta createElement-cereactions csswg-test-pr-561 csswg-test-pr-567 csswg-test-pr-1021 currencysys document-promises drawimage_svg_image_1 dup_shipping_ops epochs/daily epochs/weekly errorfields_warn event-handler-attribute-realm eventtarget-group executorwebdriver ffs-descriptor file_is_error fix-stp-version flaky-test-history foolip/audio-output-securecontext foolip/azure-win10 foolip/fullscreen-catch-throw foolip/getComputedStyle-border foolip/manifest-download-by-tag from-to-element fullscreen-request-vs-exit fullscreen-unloading-steps gecko/sync/upstream/open/1169290/1 gecko/sync/upstream/open/1436763/1 gecko/sync/upstream/open/1437255/1 gecko/1352355 gecko/1383454 gecko/1436763 gecko/1477117 gecko/1487295 gecko/1495601 gecko/1498383 gecko/1500001 gecko/1512026 gwhit-flex-changes-2 gwhit-flex-changes hallvors/clipboard-api-2 hallvors/window-opener idlharness-iterable idlharness-member-tostring initial-selection initial-value-selectionDirection jdm-branch-3 jdm-patch-1 jdm-patch-2 jdm-patch-7 jdm-patch-9 jdm-patch-10 jdm-patch-11 js-urls jugglinmike-master-demo jugglinmike-master-revert-switch jugglinmike-mimic-results-collection-2 jugglinmike-mimic-results-collection jugglinmike-wptrunner_tlbc-demo liamquin-patch-1 link-text-css-quirk lint-meta mach_wpt_serve manifest_optimise manifest_performance marcoscaceres-patch-1 marcoscaceres-patch-3-1 marcoscaceres-patch-3 master matchall-is-slow meta_global mkwst/navigation-domain nikos-svg2-marker nonce-hiding payment-request-abort-method-undefined pendingoperation py3-lint py3-make_hosts_file py3-test-handlers pyup-update-requests-2.20.1-to-2.21.0 readme reftest_fuzzy reland_tlbc requestBillingAddress revert-6614-support_firefox_prefixed_implementation_in_mediacapture-fromelementtests revert-9258-host_ip revert-13200-chrome-webdriver-default scripts-between-documents scroll-behavior scrolling-element show_global show_payment sideshowbarker/assert_object_equals speech-api-historical-onmark sync/upstream/open/1420672 sync/upstream/open/1436659 testfiles_space_path tokenlist updateWith-upon-fulfillment-manual user_activation_iframe valid-pmi wdspec_fixup webdriver-update-existing webdriver_script_timeout webidl-legacy-platform-object webmessaging-user-gesture window-open-optional-features woff2-binary-fonts wptreport_surrogates wptrunner-cdp-less-bad-9-demo wptrunner_testdriver wptserve_list_manifest zcorpan/allowed-to-use-snapshot zcorpan/assert-throws-no-name zcorpan/content-before-legend zcorpan/css-transitions-disabled zcorpan/fieldset-accessibility zcorpan/fieldset-border-radius-hittest zcorpan/fieldset-float-and-flex-or-grid zcorpan/fieldset-replaced-element zcorpan/fieldset-translatez-fixup zcorpan/fieldset-webkit-appearance zcorpan/heading-pseudo zcorpan/joint-session-history-timeout zcorpan/single-opt-in zcorpan/update-mediasource-duration zcorpan/webcrypto-any-js zcorpan/window-open-top-vs-screeny
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
96 lines (69 sloc) 3.68 KB
layout title order
page
Rendering Test Guidelines
10

There are a number of techniques typically used when writing rendering tests; these are especially using for [visual][] tests which need to be manually judged and following common patterns makes it easier to correctly tell if a given test passed or not.

Make tests self-describing

Tests should make it obvious when they pass and when they fail. It shouldn't be necessary to consult the specification to figure out whether a test has passed of failed.

Indicating success

Success is largely indicated by the color green; typically in one of two ways:

  • The green paragraph: arguably the simplest form of test, this typically consists of single line of text with a pass condition of, "This text should be green". A variant of this is using the background instead, with a pass condition of, "This should have a green background".

  • The green square: applicable to many block layout tests, the test renders a green square when it passes; these can mostly be written to match this reference. This green square is often rendered over a red square, such that when the test fails there is red visible on the page; this can even be done using text by using the [Ahem][] font.

More occasionally, the entire canvas is rendered green, typically when testing parts of CSS that affect the entire page. Care has to be taken when writing tests like this that the test will not result in a single green paragraph if it fails. This is usually done by forcing the short descriptive paragraph to have a neutral color (e.g., white).

Sometimes instead of a green square, a white square is used to ensure any red is obvious. To ensure the stylesheet has loaded, it is recommended to make the pass condition paragraph green and require that in addition to there being no red on the page.

Indicating failure

In addition to having clearly defined characteristics when they pass, well designed tests should have some clear signs when they fail. It can sometimes be hard to make a test do something only when the test fails, because it is very hard to predict how user agents will fail! Furthermore, in a rather ironic twist, the best tests are those that catch the most unpredictable failures!

Having said that, here are the best ways to indicate failures:

  • Using the color red is probably the best way of highlighting failures. Tests should be designed so that if the rendering is a few pixels off some red is uncovered or otherwise rendered on the page.

  • Tests of the line-height, font-size and similar properties can sometimes be devised in such a way that a failure will result in the text overlapping.

  • Some properties lend themselves well to making "FAIL" render in the case of something going wrong, for example quotes and content.

Other Colors

Aside from green and red, other colors are generally used in specific ways:

  • Black is typically used for descriptive text,

  • Blue is frequently used as an obvious color for tests with complex pass conditions,

  • Fuchsia, yellow, teal, and orange are typically used when multiple colors are needed,

  • Dark gray is often used for descriptive lines, and

  • Silver or light gray is often used for irrelevant content, such as filler text.

Obviously, none of this can be taken as absolute rules, as testing of color-related functionality will necessitate using some of these colors!

[ahem]: {{ site.baseurl }}{% link _writing-tests/ahem.md %} [visual]: {{ site.baseurl }}{% link _writing-tests/visual.md %}