About
Kyua is a quality assurance toolchain for software packages or, in other words, a collection of libraries and tools to implement and run automated tests against a piece of software.
Kyua's main strengths are:
- Ability to run test programs and test cases that are written in different languages and/or use various testing libraries. At the moment, Kyua is able to run test programs that do not use any framework (i.e. test programs that report their result via their exit status) and test programs that use the ATF libraries.
- Focus on the end user. Developers are encouraged to install their test suite along their production code so that end users can run these tests at will.
- Recording of historical data. The results of any executed test are stored in a database for further analysis. The recorded data includes the output of the test, a stacktrace if the test failed, the environment in which the test was run, and the specific error that the test yielded among other details.
- Generation of reports in various formats. Based on the data pre-recorded in the historical database, Kyua can currently generate plain-text and HTML reports.
The Kyua name was originally intended to be read just as the Q.A. acronym, from which its name was derived. However, myself and pretty much everybody else that reads the name Kyua says it as kyu-ah, so this is the correct pronunciation today.
Regarding the difference between Kyua and kyua, the former refers to the project as a whole whereas kyua refers explicitly to the command-line tool of Kyua.
Kyua is the successor to the Automated Testing Framework (ATF) project. ATF started as a Summer of Code project for NetBSD in 2007. The goal of the ATF project was to provide a set of libraries and a set of tools to implement an automated test suite for the NetBSD operating system. However, ATF was born as an independent project and has remained as such since then, to ensure that it can trivially benefit other operating systems and/or projects.
ATF was originally composed of two clearly-delineated components: the ATF libraries, which aided in writing test cases; and the ATF tools, which provided the runtime environment for those test cases. It is important to keep this separation in mind because as of ATF 0.20 the tools have been removed.
Soon after the first release of ATF, both the libraries and the tools were imported into NetBSD and the NetBSD test suite started growing, which in turn started turning it into a real useful tool for development and, particularly, release engineering.
Unfortunately, the original codebase of the ATF tools and the development process around them were quite convoluted. On the first hand, ATF was built on top of a prototype and, as is usually the case in these circumstances, it has not grown well over the years. In particular, some of the core design decisions and implementation details have routinely hindered the addition of extremely-desired features. On the other hand, the ATF project was being developed with a set of non-standard tools (Monotone for VCS, a custom Trac installation not well maintained, etc.) which pushed away some potential contributors.
Kyua was born in 2010 to fix the major issues in the ATF tools and, in that sense, Kyua could be thought of as "ATF v2". Kyua does not intend to replace the libraries.
After a couple of years of on-and-off development, Kyua was imported into NetBSD in February 2013. The deprecation and removal of the ATF tools in this context is still ongoing.
Kyua's main strength is that it encourages and allows developers to install the test suites of their projects along with other standard build products (binaries, libraries, documentation). Tests written with this methodology can run without access to the source code and usually without access to any development tools.
Consider the following use cases for this rationale:
- New system installation: Consider a production server, for a more realistic example. When setting up a new server with brand-new hardware, it's very desirable to ensure a minimum stability of the operating system that runs on top and how it interacts with the hardware. In this case, being able to run a test suite either for the full operating system or for specific components that are critical to the server, right out of the box, is invaluable.
- Continuous integration testing: Even though a machine may behave correctly upon a first installation, continuous maintenance of such machine may introduce inconsistencies that are difficult to catch. For example: given three packages A, B and C, where C depends on B and B depends on A, if A ever gets updated to a newer version both B and C could break. By having tests installed and available to be run at any time for these three components, it becomes possible to perform continuous testing of these as they are upgraded, configured or removed.
- Peace of mind: Every advanced user or developer wants to know that his machine is stable and that the software works as advertised. By allowing these users to run the tests themselves, read what the test are doing and witness them passing, the users can become more confident of the software and their systems.
In an ideal world, the above test scenarios could be covered by your software vendor during their Q.A. and release engineering processes. Unfortunately, in the open source world, it's unfeasible for a provider to test every possibly package combination under every possible hardware build with every possible operating system that the users might have. The idea is that, by pushing the ability to perform testing on the users' own systems, the users will become more confident of their setups and the developers will be able to catch more integration bugs than they usually would.
Kyua includes a generic test engine. What this means is that the runtime system of Kyua can execute tests implemented using various "interfaces"; i.e. it can run tests that use different libraries and/or frameworks.
The currently supported interfaces are:
- ATF: Test programs written with the ATF libraries.
- Plain: These are very simple test programs that report their success/failure status as an exit code and have no other special features. In particular, these programs are usually not linked to any special testing library. However, by virtue of attaching metadata to these programs at the Kyua level, Kyua can automatically enforce some preconditions on the execution of these tests.
- TAP: Test programs that output the Test Anything Protocol.