-
Notifications
You must be signed in to change notification settings - Fork 43
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add test output tutorial * update test-output * update tutorial as Max's suggestions * update tutorial * check char<80 & spelling * update tutorial with ref link --------- Co-authored-by: Pierre Equoy <pierre.equoy@canonical.com>
- Loading branch information
Showing
2 changed files
with
122 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,121 @@ | ||
.. _test_output: | ||
|
||
============================ | ||
Test output | ||
============================ | ||
|
||
When Checkbox runs the tests, it generates a series of outputs, which include | ||
the ``ID`` and ``Category`` of the tests, along with the results of each of | ||
them (if a test ID fails, the cause for the failure would be displayed). | ||
|
||
Now we will introduce each section of the test output by performing a test | ||
for audio device detection in practice. | ||
|
||
.. code-block:: none | ||
=========[ Running job 1 / 3. Estimated time left (at least): 0:00:01 ]========= | ||
**Progress Indicator** shows the current status of the ongoing | ||
job, including the total number of jobs and the number completed. It also | ||
provides an estimated time left for completion, helping users track progress | ||
and estimate when the test will finish. | ||
|
||
|
||
.. code-block:: none | ||
-----------------------------[ Hardware Manifest ]------------------------------ | ||
**Summary** provides a brief sentence to help users understand the purpose of | ||
this job. | ||
|
||
In this case, it's `Hardware Manifest | ||
<https://checkbox.readthedocs.io/en/latest/reference/launcher.html | ||
#manifest-section>`_, which provides information | ||
about various hardware devices used in the tests. When certain tests involve | ||
specific hardware devices, Checkbox will prompt users to inform in advance | ||
whether the machine includes specific hardware devices required for testing | ||
purposes. | ||
|
||
|
||
.. code-block:: none | ||
ID: com.canonical.plainbox::manifest | ||
Category: com.canonical.plainbox::uncategorised | ||
**ID** indicates the job IDs run during the test. and it's organized into | ||
different **Categories** based on their functionalities or characteristics. | ||
|
||
|
||
.. code-block:: none | ||
... 8< ------------------------------------------------------------------------- | ||
ns: com.canonical.certification | ||
name: checkbox-provider-base | ||
has_audio_playback: False | ||
has_audio_capture: True | ||
------------------------------------------------------------------------- >8 --- | ||
This is the job stdout from ``com.canonical.plainbox::manifest``. If a job | ||
fails or is skipped, the message will also be shown in this section. | ||
|
||
In our case, we are testing audio device detection, and the manifest was | ||
prompted earlier to ask for user-provided information, this section displays | ||
the collected information from the user. | ||
|
||
|
||
.. code-block:: none | ||
Outcome: job passed | ||
At the end of each test, the **Outcome** will be displayed to indicate whether | ||
the test has passed. | ||
|
||
|
||
.. code-block:: none | ||
------------[ Check that at least one audio playback device exits ]------------- | ||
ID: com.canonical.certification::audio/detect-playback-devices | ||
Category: com.canonical.plainbox::audio | ||
Job cannot be started because: | ||
- resource expression "manifest.has_audio_playback == 'True'" evaluates to false | ||
Outcome: job cannot be started | ||
Oops! Apparently this job was skipped.Based on the stdout, it seems that this | ||
issue occurred because when Checkbox asking hardware manifest, we mistakenly | ||
set ``has_audio_playback`` to ``False``. Consequently, Checkbox determined that | ||
the machine lacks the necessary audio devices to support the test. | ||
|
||
|
||
.. code-block:: none | ||
☑ : Hardware Manifest | ||
☐ : Check that at least one audio playback device exits | ||
☑ : Check that at least one audio capture device exists | ||
After all the jobs are completed, a checklist will summarize the results of | ||
each test. | ||
|
||
|
||
.. code-block:: none | ||
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.html | ||
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.junit.xml | ||
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.tar.xz | ||
Checkbox collect all data related to the test and generates a **Test Reports** | ||
locally. | ||
|
||
|
||
.. code-block:: none | ||
Do you want to submit 'upload to certification' report? | ||
y => yes | ||
n => no | ||
Users would be asked whether they want to upload the results to the Canonical | ||
:term:`Certification website`. | ||
|
||
Congrats! Now you have a basic understanding | ||
of the test output. | ||
|