Automated Test Framework
Note: this page was initially written to design an automated test framework for the MathJax project.
Mozilla has many automated test frameworks. One which is particularly interesting is the concept of reftests. As a basic example, suppose we want to check that <mstyle/> supports the attributes of the <mfenced/> element. Thus we have the following MathML fragment
<mstyle open="[" close="]" separators=";">
<mfenced>
<mn>1</mn>
<mn>2</mn>
</mfenced>
</mstyle>
and expect that the attributes on <mstyle/> are taken into account by the <mfenced/>. A naive way to do that is to use a reference image and compare with MathJax's actual rendering of the fragment. However even in that simple case the rendering depends on too many parameters such as the operating system & browser, text size, type of fonts used, default zoom factor etc. Thus it is not really possible to automate the tests unless we restrict ourselves to a very specific configuration. The idea of Mozilla's reftests is to use a web page both for the test and the reference. Hence in the example above, the reference fragment can simply be:
<mstyle>
<mfenced open="[" close="]" separators=";">
<mn>1</mn>
<mn>2</mn>
</mfenced>
and MathJax passes the test iff it renders the test and the reference identically! Although this approach may seem limited (for example a renderer which does not implement MathML at all would pass all tests) it turns out that it is a powerful tool. Generally, combining several == tests (where we require the test and the reference to be equal) and != tests (when we require them not to be equal) allows to check carefully the implementation of any feature. Other organizations such as the CSS WG or WebKit developers plan to use it. Experiments on a MathJax's branch have demonstrated the feasibility of this approach for MathJax.
Obviously, reftests enable to test MathJax's MathML support. However, it is not to difficult to imagine that they can also be useful in other part of MathJax's rendering engine. In general, we do not have a canonical conversion from LaTeX to MathML or from MathML to HTML+CSS. However, if we agree on such a conversion we could made reftests which compare the resulting XML trees instead of the visual rendering. Again, to give a basic example the two
's below should contain isomorphic trees after MathJax's processing:<div id="LaTeX">$$x$$</div>
<div id="MathML"><math display="block"><mi>x</mi></math></div>
and similarly with MathML and HTML+CSS trees. This has been used for a Mozilla add-on implementing itex2MML in order to write unit tests for the LaTeX to MathML conversion. Finally, it has even be imagined to use reftests for testing performance.
To conclude this section, let's mention Mozilla's crashtests. A crashtest is simply made of a single web page and the test is passed if the browser is able to load the page without crashing. This could also be used for MathJax, where crash can either mean a crash of the browser or of the javascript engine, and the test is passed after MathJax fires its final "processing complete" event.
There are many tools for testing Web applications, the most popular are listed here. Because we want MathJax to work in "all modern browsers", an important aspect is to be able to check the actual behavior of the browsers on each operating system. This means that tools which simulate the javascript execution and do not provide the real rendering should to be excluded. Also, it means that a top criterium for choosing a tool is the range of platforms supported. We first give the following table based on the information from the official website:
| Tool | OS |
Browsers |
|
HtmlUnit |
/ | Simulate a browser |
| iMacros | Windows |
Internet Explorer |
| Watir | Windows, Mac, Linux |
Internet Explorer, Firefox. (unofficial drivers exist for Chrome, Opera and Safari) |
| Sahi | Any OS? |
Any browser? |
| Selenium | Windows, Mac, Linux (possibly more) |
Firefox, IE, Safari, Opera, Chrome (possibly more) |
|
TestComplete |
Windows |
Internet Explorer |
| WebTest | / | Simulate a browser |
Hence we see that Watir, Sahi and Selenium are the more attractive according to the remark above. One issue with Watir is that it is using different libraries for each platforms and so its use may not be really convenient. Selenium and Sahi use "injection of javascript code" to control the browsers and so can in theory work in any platform. In practice, they can be limited by browser security policy (for example, I haven't be able to run the tests in Opera until now). To workaround that issue, Google has developed Webdriver which tries to use "whichever mechanism is most appropriate to control the browser". However, Selenium and Webdriver projects are now merging. A Watir interface to Webdriver is also being developed. The possibility to use WebDriver has also been investigated by Sahi developers.
In the remaining sections, we turn our attention to the three tools that provide the wider support of platforms.
Selenium seems to be the most mature product, it is by far the most well-known and other competitors always take it as a reference. The three tools are open source and seems to have an active community with blogs, documentation etc. Selenium (stable release: November 5, 2010 ; 51 contributors on SVN) will merge with Google's Webdriver project for the next version. Watir (stable release: October 26, 2010 ; 63 forks on GitHub) also has a large community of developers which provide versions for other programming languages and platforms. It looks like Sahi (stable release: December 3, 2010 ; 10 contributors on SourceForge) has the smallest community.
For MathJax unit tests, it is important to have a good script language as an interface. However, for some complicated tests which involve a very specific scenario it may be more convenient to record the actions directly with a graphical interface and generate test scripts. Sahi uses "Sahi Script" (Javascript-like) and has drivers in Java and Ruby. Selenium tests can be written in a variety of languages: Java, C#, Python, Ruby, Perl or PHP. Watir works only with Ruby, although some forks exist for .Net and Java languages. Sahi's recorder works in all platforms while Selenium has only an extension for Firefox. Watir does not have official recorders, but some exist outside the community.
As previously said, we want to run in-browser tests for MathJax in order to check the real behavior on each platform. The drawback of this method is that it can be slow to test each platform version sequentially. To solve that issue, Selenium has a separate tool called "Selenium Grid" which allows to run several instances of browsers on different processes. Sahi seems to have the best support, with an in-built multithreaded or parallel playback of tests. Watir's documentation only mention the possibility to run multiple instances of Internet Explorer. Finally, note that there is undergoing work to improve concurrency support in Webdriver that could benefit to Selenium and maybe also to Watir.
The major issue I have found with Selenium is timing issues: for example, you have to add "sleep" actions after taking a screenshot or loading a page. Sahi and Watir Websites say they deal better with this (although I haven't tested them). Looking at their respective APIs, the three tools seem to have a good set of functions to access elements in a page or to ask the browser to do an action. Nevertheless, to implement the reftests or other visual verifications, one important feature is the possibility to take screenshots. Selenium can do that and it has successfully be used to experiment reftests as indicated above. Watir has a function Watir::ScreenCapture but it seems to be only available with Windows. Sahi API does contain any function of this kind.
In view of the previous analysis, Selenium seems to be the most appropriate web testing tool to use for MathJax. It may still have some imperfections but the fact that it has merged with Google's Webdriver enables us to think that improvements will come in a near future.