Skip to content

Commit

Permalink
Add test output tutorial (#639)
Browse files Browse the repository at this point in the history
* add test output tutorial

* update test-output

* update tutorial as Max's suggestions

* update tutorial

* check char<80 & spelling

* update tutorial with ref link

---------

Co-authored-by: Pierre Equoy <pierre.equoy@canonical.com>
  • Loading branch information
diohe0311 and pieqq committed Jul 31, 2023
1 parent c71db97 commit 048124e
Show file tree
Hide file tree
Showing 2 changed files with 122 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/tutorial/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ journey through the product.
:maxdepth: 1

using-checkbox/basic-launcher
using-checkbox/test-output

Extended tutorial
-----------------
Expand Down
121 changes: 121 additions & 0 deletions docs/tutorial/using-checkbox/test-output.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
.. _test_output:

============================
Test output
============================

When Checkbox runs the tests, it generates a series of outputs, which include
the ``ID`` and ``Category`` of the tests, along with the results of each of
them (if a test ID fails, the cause for the failure would be displayed).

Now we will introduce each section of the test output by performing a test
for audio device detection in practice.

.. code-block:: none
=========[ Running job 1 / 3. Estimated time left (at least): 0:00:01 ]=========
**Progress Indicator** shows the current status of the ongoing
job, including the total number of jobs and the number completed. It also
provides an estimated time left for completion, helping users track progress
and estimate when the test will finish.


.. code-block:: none
-----------------------------[ Hardware Manifest ]------------------------------
**Summary** provides a brief sentence to help users understand the purpose of
this job.

In this case, it's `Hardware Manifest
<https://checkbox.readthedocs.io/en/latest/reference/launcher.html
#manifest-section>`_, which provides information
about various hardware devices used in the tests. When certain tests involve
specific hardware devices, Checkbox will prompt users to inform in advance
whether the machine includes specific hardware devices required for testing
purposes.


.. code-block:: none
ID: com.canonical.plainbox::manifest
Category: com.canonical.plainbox::uncategorised
**ID** indicates the job IDs run during the test. and it's organized into
different **Categories** based on their functionalities or characteristics.


.. code-block:: none
... 8< -------------------------------------------------------------------------
ns: com.canonical.certification
name: checkbox-provider-base
has_audio_playback: False
has_audio_capture: True
------------------------------------------------------------------------- >8 ---
This is the job stdout from ``com.canonical.plainbox::manifest``. If a job
fails or is skipped, the message will also be shown in this section.

In our case, we are testing audio device detection, and the manifest was
prompted earlier to ask for user-provided information, this section displays
the collected information from the user.


.. code-block:: none
Outcome: job passed
At the end of each test, the **Outcome** will be displayed to indicate whether
the test has passed.


.. code-block:: none
------------[ Check that at least one audio playback device exits ]-------------
ID: com.canonical.certification::audio/detect-playback-devices
Category: com.canonical.plainbox::audio
Job cannot be started because:
- resource expression "manifest.has_audio_playback == 'True'" evaluates to false
Outcome: job cannot be started
Oops! Apparently this job was skipped.Based on the stdout, it seems that this
issue occurred because when Checkbox asking hardware manifest, we mistakenly
set ``has_audio_playback`` to ``False``. Consequently, Checkbox determined that
the machine lacks the necessary audio devices to support the test.


.. code-block:: none
☑ : Hardware Manifest
☐ : Check that at least one audio playback device exits
☑ : Check that at least one audio capture device exists
After all the jobs are completed, a checklist will summarize the results of
each test.


.. code-block:: none
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.html
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.junit.xml
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.tar.xz
Checkbox collect all data related to the test and generates a **Test Reports**
locally.


.. code-block:: none
Do you want to submit 'upload to certification' report?
y => yes
n => no
Users would be asked whether they want to upload the results to the Canonical
:term:`Certification website`.

Congrats! Now you have a basic understanding
of the test output.

0 comments on commit 048124e

Please sign in to comment.