Test Case Management System Research
Kiwi is a Test Case Management System with user logins. It was built to manage manual tests for a product. It has user interfaces for writing of test plans and associated test cases (which including review cycles of the test plan within the software) and then executing the test plan. Each test case can "pass" or "fail".
See an example instance of Kiwi TCMS. You can log in with your GitHub account, make and save and run test plans and cases
The CSSWG Test Harness is a test harness written for testing CSS during the development of the CSS specification. Tests are written in a structure format and the Test Harness will present the tests to a tester. All tests are associate with a part of the CSS specification. They can be run by section of the CSS specification or in other arbitrary groupings. They record the user agent of the runner (but you can also override it), save results of "pass", "fail", "can't tell", and "skip".
See the test harness here. You can run tests, see the result of tests, etc, without a log in.
TestLink is a PHP web application.
Kiwi TCMS is an open source project with an active GitHub. It is a Django (python) web app.
The CSSWG's test harness is open source but maintained mostly by one person (Peter Linss) who currently plans to actively work on features this fall in order to better integrate the software with WPT. It is a web application that renders webpages on the server side with little to no javascript, written in object-oriented PHP. We would have support mirroring the mercurial repository and GitHub and doing further development there.
What is the expected format of a test and how to upload tests into the system?
Kiwi TCMS has a concept of a "Test Plan" and "Test Case". Kiwi TCMS has an API to upload and edit these test plans and test cases. We can write a python script to import tests from whatever structure format we design.
Test Plans have little structure. By default there are required fields "Name", "Product" and "Version". Otherwise there is a "Reference Link" and an editable "Test Plan Document" that can be written in Markdown.
All Test Cases must be associated with a Test Plan. Test Cases can be associate with multiple Test Plans (So, test plan could be used just to group Test Cases).
Test Cases also have a markdown text field and a handful of extra fields. You can structure the writing of the test how ever you would like in the markdown text field. This mark down test field must include both the instructions for the test and the expected result of the instructions.
CSSWG tests are written as an HTML document. The HTML document is presented in the test harness in an iframe, optionally along side a reference file. If there is a reference file, the test is that the test file should be visually identical to the reference file. Otherwise, the test file contains in it the description of the test (for example: "There should be a yellow box centered in a red box".
CSSWG tests can have hidden fields such author, links to specifications, title. These fields are not show to test but used to organize tests or effect the test harness.
CSSWG tests are grouped by specifications, and can be extended to group by other groupings with little work.
Can we record structure results (more than yes/no)? Can we customize all the fields and the order in which they are asked? Can we do branching based on answers?
Test Cases have little to know structure in the result recording.
You can run tests in groups according test cases in a "Test Run". Testers must navigate through the Tests (by expanding tests, which will show the markdown'd test text) and then they can record "Pass", "Fail" or a few other options such as "Error" or "Blocked") and fill in a plain text "Comment" box. Then the tester will mark the test plan as "Finished".
Tests can be marked as "Pass", "Fail", "Cannot Tell" or "Skip".
When selecting a group of test to run (based on section of the CSS Specifcation), the harness will:
- Show a test in an iframe with some structured data from the test (like the section of the specification).
- The tester can record the result of the test.
- The harness will open the next test.
Can we assign tests to specific testers.
Yes.
No. Individuals in the CSSWG can log in with the W3C credentials but the only extra option is the ability to delete test results.
Audit needed of: Example instance of Kiwi TCMS
Audit needed of: CSSWG Test Harness