Theory
A full paper on the inception of Watai is available in French. If you can't read it or want a shorter version, read below!
Software quality literature classically distinguishes three levels, or layers, of testing: “unit”, “functional”, and “integration”.
The first level, the one that is closest to the code, is unit testing. Unit testing is concerned with the smallest piece of code that contains business logic. Usually, that means methods or functions. In the context of web applications, the tested “units” may reside either on the server or the client-side. Both parts should be unit-tested if you're to be serious about quality.
The second level, “functional testing”, is about testing parts of the program that provide independent features. The exact size of the “part” depends on the architecture, the environment, the size of the system… It may be classes, modules, or whole parts in distributed architectures. In the context of web applications, this could be for example a JSON endpoint or a frontend component. The main issue is usually defining what is big enough to be worth a set of tests, yet small enough to be tested independently from the rest of the system.
Finally, “integration” testing is the level in which a combination of individual software modules are tested as a group. It also means testing through the same access means as the final users will. In the context of web applications, it means testing the result of combining client- and server-side code, and accessing it through a browser.
Watai is a tool for integration testing, and focuses on making it easy for developers to write user-oriented tests that consider the system under test as a black box, yet are maintainable enough to be worth investing in.
Based on the above definition, you might have already heard about what I call here “integration testing” under a different term.
You might have read about “customer”, “end-to-end” (also abbreviated as “E2E”), “system”, “acceptance”, “validation” or “acceptance” testing. All of these are many synonyms for the same reality.
The most problematic, as it goes against the definitions above, is “functional” testing. Some authors indeed name only “unit testing” and “functional testing”, considering the blurriness of the latter can go as large as testing the whole system.
In the end, there is no right or wrong way to call this activity. Based on the usual connotations and existing literature, I will only refer to it in this document as “integration testing”.
First, ensuring that a functionality, a feature that adds customer value, is present.
Second, proving that this functionality behaves properly. That is, given a set of inputs and a navigation path, the expected information is presented to the user.
Finally, in the context of web applications, we're fighting against platform discrepancies, usually referred to as “cross-browser issues”. So, we're not only concerned about functionalities being present and behaving properly, but about them doing so in all the browsers our target users could use to access our application.
Watai can detect the kind of bugs that appear only when all moving parts are put together to form the final software with minimal effort. However, no integration-level testing tool can be considered a proper testing strategy alone.
You should have unit tests for your application, both on the server- and on the client-side. If your application is complex, adding functional-level tests is a good thing.
Integration-level testing validates scenarios, expected behavior, but cannot help in pinpointing a failure origin as much as unit testing can. Moreover, even though Watai does its best to ease writing and maintaining such high-level tests, since they run across so many components, they will always require more effort than the kind of direct code access unit tests provide. They also can also catch bugs you wouldn't have been able to find otherwise, though.
Even though integration testing cannot replace other types of tests, it is quite surprising that so little of them are automated across the industry. It sounds pretty obvious one shouldn’t, except maybe on very early prototyping stages, manually run a function to unit test it. It is nowadays rather commonplace to use a unit-testing framework, and “xUnit” variants are available for all languages. However, it is also still very common to manually load a web page in all target browsers and reproduce the same actions in every single one.
This lack of automation has a very high cost. Of course, directly through time spent on manual testing, especially considering it is often developer time. But also through delayed releases and late detection of cross-browser issues that require reworking an otherwise ready feature because it misbehaves on some target browser, invalidating all previous acceptance.