Add Tutorial provider (#645)

* add empty tutorial provider * add tutorial jobs that end in 3 different outcomes * add additional units for the tutorial provider * Add tutorial job that uses an environment variable * Add readme for the tutorial provider * Split tutorial provider name and namespace * Amend existing tutorial pages to point to the newly created tutorial test plan * Add tutorial provider to the checkbox22 snap The base tutorial uses jobs and test plan from this provider as part of the introduction to Checkbox, so it needs to be present in the snap. * Add a Tutorial category * Tutorial provider: update existing jobs and add manual and semi-auto jobs * Tutorial provider: add interactive jobs to the test plan --------- Co-authored-by: Pierre Equoy <pierre.equoy@canonical.com>
canonical · Sep 8, 2023 · d47079a · d47079a
1 parent 4e1d282
commit d47079a
Show file tree

Hide file tree

Showing 10 changed files with 224 additions and 33 deletions.
diff --git a/checkbox-core-snap/series22/snap/snapcraft.yaml b/checkbox-core-snap/series22/snap/snapcraft.yaml
@@ -51,6 +51,10 @@ slots:
     interface: content
     read:
       - $SNAP/providers/checkbox-provider-certification-server
+  provider-tutorial:
+    interface: content
+    read:
+      - $SNAP/providers/checkbox-provider-tutorial
   checkbox-runtime:
     interface: content
     read:
@@ -438,6 +442,18 @@ parts:
       python3 manage.py build
       python3 manage.py install --layout=relocatable --prefix=/providers/checkbox-provider-certification-server --root="$SNAPCRAFT_PART_INSTALL"
     after: [checkbox-provider-certification-client]
+################################################################################
+  checkbox-provider-tutorial:
+    plugin: dump
+    source: providers/tutorial
+    source-type: local
+    override-build: |
+      export PYTHONPATH=$SNAPCRAFT_STAGE/lib/python3.10/site-packages:$SNAPCRAFT_STAGE/usr/lib/python3/dist-packages
+      for path in $(find "$SNAPCRAFT_STAGE/providers/" -mindepth 1 -maxdepth 1 -type d); do export PROVIDERPATH=$path${PROVIDERPATH:+:$PROVIDERPATH}; done
+      python3 manage.py validate
+      python3 manage.py build
+      python3 manage.py install --layout=relocatable --prefix=/providers/checkbox-provider-tutorial --root="$SNAPCRAFT_PART_INSTALL"
+    after: [checkbox-provider-base]
 ################################################################################
   gnome-randr:
     source: https://github.com/maxwellainatchi/gnome-randr-rust.git

diff --git a/docs/tutorial/using-checkbox/basic-launcher.rst b/docs/tutorial/using-checkbox/basic-launcher.rst
@@ -30,7 +30,7 @@ Create a file named ``mylauncher`` and add the following information in it:
     launcher_version = 1
 
     [test plan]
-    filter = *TODO*
+    filter = *tutorial*
 
 Save it, then launch Checkbox using this launcher:
 
@@ -42,23 +42,23 @@ The test plan selection screen should be much less intimidating now!
 
 Let's review the content of this launcher.
 
-The ``[launcher]`` section contains meta-data related to the launcher itself
+The ``[launcher]`` section contains meta-data related to the launcher itself.
 ``launcher_version`` is set to ``1`` as it is the version of the launcher
 syntax currently in use.
 
-In the ``[test plan]`` section, we set the ``filter`` to ``*TODO*``. By
+In the ``[test plan]`` section, we set the ``filter`` to ``*tutorial*``. By
 doing so, the only test plans that will be displayed are the one whose
-``id`` contain the word ``TODO``. Note the use of the ``*`` `glob pattern
+``id`` contain the word ``tutorial``. Note the use of the ``*`` `glob pattern
 <https://en.wikipedia.org/wiki/Glob_(programming)>`_, similar to Bash for
 instance.
 
-Try replacing ``*TODO*`` with ``*wireless*`` and see how it affects the
+Try replacing ``*tutorial*`` with ``*wireless*`` and see how it affects the
 list of test plans displayed in Checkbox.
 
 Select a test plan by default
 =============================
 
-Set the filter back to ``*TODO*`` in the ``[test plan]`` section and add
+Set the filter back to ``*tutorial*`` in the ``[test plan]`` section and add
 the following line:
 
 .. code-block:: none
@@ -71,12 +71,12 @@ the following line:
     app_id = com.canonical.certification:tutorial
 
     [test plan]
-    filter = *TODO*
-    unit = com.canonical.certification::smoke
+    filter = *tutorial*
+    unit = com.canonical.certification::tutorial-base
 
 Start Checkbox using the launcher. In addition to filtering the list of test
-plans, the TODO test plan is now selected by default. You just need to press
-``Enter`` to go to the test selection screen.
+plans, the "Checkbox Base Tutorial" test plan is now selected by default.
+You just need to press ``Enter`` to go to the test selection screen.
 
 Bypass test plan and test selection screens
 ===========================================
@@ -96,7 +96,7 @@ screen. Modify your launcher so it looks like that:
     app_id = com.canonical.certification:tutorial
 
     [test plan]
-    unit = com.canonical.certification::TODO
+    unit = com.canonical.certification::tutorial-base
     forced = yes
 
     [test selection]
@@ -109,11 +109,11 @@ Run Checkbox with this modified version of the launcher:
     $ checkbox.checkbox-cli launcher mylauncher
 
 Notice how none of the initial screens are shown and Checkbox immediately
-runs the TODO test plan. This is because:
+runs the "Checkbox Base Tutorial" test plan. This is because:
 
 - in the ``[test plan]`` section, we selected a test plan with ``unit =
-  TODO`` and we forced its use with ``forced = yes``, bypassing the test plan
-  selection screen;
+  com.canonical.certification::tutorial-base`` and we forced its use with
+  ``forced = yes``, bypassing the test plan selection screen;
 - in the ``[test selection]`` section, we forced the selection of all the
   tests, bypassing the test selection screen.
 
@@ -133,17 +133,18 @@ environment variable. Add the following lines in the launcher:
     app_id = com.canonical.certification:tutorial
 
     [test plan]
-    unit = com.canonical.certification::TODO
+    unit = com.canonical.certification::tutorial-base
     forced = yes
 
     [test selection]
     forced = yes
 
     [environment]
-    TUTO = tutorial
+    TUTORIAL = Value from my launcher!
 
-Run Checkbox using your launcher, and observe the output of the TODO test
-case. The output now shows ``tutorial``.
+Run Checkbox using your launcher, and observe the output of the
+``tutorial/environment_variable`` test case. The output now shows ``Value
+from my launcher!``.
 
 The ``[environment]`` section is often used to provide customized values to
 test cases. For instance, you may have a generic test case to connect to a
@@ -173,14 +174,14 @@ Edit the launcher file:
     stock_reports = text, submission_files
 
     [test plan]
-    unit = com.canonical.certification::TODO
+    unit = com.canonical.certification::tutorial-base
     forced = yes
 
     [test selection]
     forced = yes
 
     [environment]
-    TUTO = tutorial
+    TUTORIAL = Value from my launcher!
 
 Run Checkbox using this launcher and observe that once the test plan is
 finished running, Checkbox generates a summary on the screen and provides
@@ -307,14 +308,14 @@ At the top of the launcher file, add this line:
     stock_reports = text, submission_files
 
     [test plan]
-    unit = com.canonical.certification::TODO
+    unit = com.canonical.certification::tutorial-base
     forced = yes
 
     [test selection]
     forced = yes
 
     [environment]
-    TUTO = tutorial
+    TUTORIAL = Value from my launcher!
 
 Make the launcher executable:
 

diff --git a/docs/tutorial/using-checkbox/basic-remote.rst b/docs/tutorial/using-checkbox/basic-remote.rst
@@ -69,10 +69,10 @@ On the agent terminal, you should see something like  ``Using `$USER` user``
 where ``$USER`` is your local user name.
 
 On the controller terminal, you should get the list of test plans available. Go
-ahead and select the *TODO* test plan, keep all the tests selected and start
-the test session by pressing ``T``. All the test cases are being executed,
-then Checkbox generates the usual text summary as well as the submission
-files. Notice how the output is slightly different:
+ahead and select the "Checkbox Base Tutorial" test plan, keep all the tests
+selected and start the test session by pressing ``T``. All the test cases
+are being executed, then Checkbox generates the usual text summary as well
+as the submission files. Notice how the output is slightly different:
 
 .. code-block:: none
 
@@ -117,9 +117,9 @@ other device.
 Launchers in remote mode
 ========================
 
-In remote mode, you can use launchers the same way you did in local mode. If
-you still have the launcher file you created in the Launchers section*TODO*,
-run the following command:
+In remote mode, you can use launchers the same way you did in
+local mode. If you still have the launcher file you created in the
+:ref:`base_tutorial_launcher` section, run the following command:
 
 .. code-block:: none
 
@@ -138,8 +138,8 @@ of them is the interrupt screen. Run Checkbox remote:
 
     checkbox.checkbox-cli remote 127.0.0.1
 
-Select the *TODO* test plan, leave all the jobs selected, and press ``T``
-to start the testing session.
+Select the "Checkbox Base Tutorial" test plan, leave all the jobs selected,
+and press ``T`` to start the testing session.
 
 Now, while the tests are being executed by the agent, press ``Ctrl+C``
 on the controller. You should see a screen like this:

diff --git a/docs/tutorial/using-checkbox/checkbox-commands.rst b/docs/tutorial/using-checkbox/checkbox-commands.rst
@@ -288,9 +288,10 @@ You can also run a whole test plan using the ``run`` command:
 
 .. code-block:: none
 
-    checkbox.checkbox-cli run TODO
+    checkbox.checkbox-cli run com.canonical.certification::tutorial-base
 
-    ...
+This will run the Checkbox Base Tutorial test plan, executing all the jobs in
+it and providing a text summary of the test run.
 
 Wrapping up
 ===========

diff --git a/providers/tutorial/README.md b/providers/tutorial/README.md
@@ -0,0 +1,7 @@
+Summary
+=======
+
+This provider contains jobs and test plans used in the tutorial available in
+the official Checkbox documentation:
+
+https://checkbox.readthedocs.io/en/latest/tutorial/
diff --git a/providers/tutorial/manage.py b/providers/tutorial/manage.py
@@ -0,0 +1,10 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup, N_
+
+setup(
+    name="tutorial",
+    namespace="com.canonical.certification",
+    version="1.0",
+    description=N_("The Checkbox Tutorial provider"),
+    gettext_domain="com_canonical_certification_tutorial",
+)
diff --git a/providers/tutorial/units/category.pxu b/providers/tutorial/units/category.pxu
@@ -0,0 +1,3 @@
+unit: category
+id: tutorial
+_name: Tutorial
diff --git a/providers/tutorial/units/jobs.pxu b/providers/tutorial/units/jobs.pxu
@@ -0,0 +1,125 @@
+id: tutorial/passing
+category_id: tutorial
+plugin: shell
+_summary: A job that always passes
+command:
+ echo This job passes!
+ true
+
+id: tutorial/failing
+category_id: tutorial
+plugin: shell
+_summary: A job that always fails
+command:
+ echo This job fails!
+ false
+
+id: tutorial/crashing
+category_id: tutorial
+plugin: shell
+_summary: A job that always crashes
+command:
+ # Start the sleep 5 command, but if it doesn't finish in 1 second, send it the
+ # KILL signal to stop it immediately." Since sleep 5 is meant to run for 5
+ # seconds and the timeout is set to 1 second, the sleep 5 command will be
+ # killed by the KILL signal after just 1 second.
+ echo "This job crashes because we run a command to kill it before it's finished."
+ timeout -s KILL 0.1 sleep 5
+
+id: tutorial/has_resource
+category_id: tutorial
+plugin: shell
+requires:
+ (tutorial_resources.name == 'first-resource' and tutorial_resources.has_required_thing == 'true')
+_summary: A job that requires a resource and it is available
+command:
+ echo This job runs because its requirements are fulfilled.
+ true
+
+id: tutorial/no_resource
+category_id: tutorial
+plugin: shell
+requires:
+ (tutorial_resources.name == 'second-resource' and tutorial_resources.has_required_thing == 'true')
+_summary: A job that requires a resource but it's not available
+command:
+ true  # doesn't matter - will not run
+
+id: tutorial/passing_dependency
+category_id: tutorial
+plugin: shell
+depends:
+ tutorial/passing
+_summary: A job that depends on other job that passes
+command:
+ true  # this will run!
+
+id: tutorial/failing_dependency
+category_id: tutorial
+plugin: shell
+depends:
+ tutorial/failing
+_summary: A job that is skipped because it depends on a job that fails
+command:
+ true  # doesn't matter - will not run
+
+id: tutorial/environment_variable
+category_id: tutorial
+plugin: shell
+_summary: A job that displays an environment variable, if set
+environ: TUTORIAL
+command:
+ echo This job prints the content of a given environment variable if set, or a default value otherwise
+ echo ${TUTORIAL:-"Default Value"}
+
+id: tutorial/manual
+category_id: tutorial
+plugin: manual
+_summary: A manual job
+estimated_duration: 1m
+_purpose:
+ This is a manual job. User needs to select an outcome.
+_steps:
+ 1. Read the content of this job.
+ 2. Check that there are 3 sections (Purpose, Steps, Verification).
+_verification:
+ Did Checkbox display all 3 sections of the manual job?
+
+id: tutorial/user-interact
+category_id: tutorial
+plugin: user-interact
+_summary: A semi-automated job where the outcome is set automatically
+estimated_duration: 1m
+command: true
+_purpose:
+ This is a "user-interact" semi-automated job. It requires the user to perform
+ an interaction, after which the outcome is automatically set and Checkbox moves
+ on to the next job.
+ .
+ This test will run the command `true`, which always returns 0.
+_steps:
+ 1. Read the content of this job.
+ 2. Press Enter to start the test. The outcome will be set automatically to
+ "pass" based on the return value from the command, and Checkbox will then
+ move on to the next job.
+
+id: tutorial/user-interact-verify
+category_id: tutorial
+plugin: user-interact-verify
+_summary: A semi-automated job where the user manually sets the outcome
+estimated_duration: 1m
+command: true
+_purpose:
+ This is a "user-interact-verify" semi-automated job. It requires the user
+ to perform an interaction, then Checkbox executes a command and suggests
+ an outcome based its return code. However, in the end it is up to the user
+ to manually choose the right outcome.
+ .
+ This test will run the command `true`, which always returns 0.
+_steps:
+ 1. Read the content of this job.
+ 2. Press Enter to start the test. The outcome will be automatically set to
+ "pass" but you will have a chance to manually select another outcome.
+_verification:
+ Make sure that Checkbox suggested the outcome to be "pass", yet you can
+ still manually select another outcome.
diff --git a/providers/tutorial/units/resource.pxu b/providers/tutorial/units/resource.pxu
@@ -0,0 +1,11 @@
+id: tutorial_resources
+category_id: tutorial
+plugin: resource
+_summary: A job that generates different resources for tutorial purposes
+command:
+ echo name: first-resource
+ echo has_required_thing: true
+ echo
+ echo name: second-resource
+ echo has_required_thing: false
+ echo
diff --git a/providers/tutorial/units/test-plan.pxu b/providers/tutorial/units/test-plan.pxu
@@ -0,0 +1,17 @@
+unit: test plan
+id: tutorial-base
+_name: Checkbox Base Tutorial Test Plan
+bootstrap_include:
+ tutorial_resources
+include:
+ tutorial/passing
+ tutorial/failing
+ tutorial/crashing
+ tutorial/passing_dependency
+ tutorial/failing_dependency
+ tutorial/has_resource
+ tutorial/no_resource
+ tutorial/environment_variable
+ tutorial/manual
+ tutorial/user-interact
+ tutorial/user-interact-verify