Test framework

.github/workflows/test-lp.yml

@@ -44,4 +44,4 @@ jobs:
             Update test result file with recent run
             Auto-generated by create-pull-request: https://github.com/peter-evans/create-pull-request
           branch: update-stats
-          base: main
+          base: main

.github/workflows/test_lp.sh

@@ -0,0 +1,23 @@
+ALL_CHANGED_FILES=$@
+
+# Install dependencies and run tests,
+# if we found tests to run
+echo "All changed content paths:       ${ALL_CHANGED_FILES}"
+# Keep full paths for install guides,
+# get parent directory for learning paths
+CONTENT_PATHS=()
+for file in ${ALL_CHANGED_FILES[*]}; do
+   parentdir="$(dirname "$file")"
+   if echo "$parentdir" | grep -q "install-guides"; then
+      CONTENT_PATHS+=("$file")
+   else
+      CONTENT_PATHS+=("$parentdir")
+   fi
+done
+# Make sure each learning path is only tested once
+CONTENT_PATHS_UNIQUE=($(printf "%s\n" "${CONTENT_PATHS[@]}" | sort -u))
+echo "Unique content paths:         ${CONTENT_PATHS_UNIQUE[*]}"
+# Run the tests
+for file in ${CONTENT_PATHS_UNIQUE[*]}; do
+   tools/maintenance.py -i ${file}
+done

.gitignore

@@ -5,6 +5,7 @@ resources/
 node_modules/
 package-lock.json
 .hugo_build.lock
+.vscode
 
 # macOS files
 *.DS_Store

content/learning-paths/cross-platform/_example-learning-path/appendix-3-test.md

@@ -62,7 +62,8 @@ The framework will check the return code. If not 0, an error will be reported.
 If a specific return code is expected, it can be specified as follows:
 
 ```markdown
-    The file myfile.txt doesn't exist yet and this command returns 1:
+    The file myfile.txt doesn't exist yet
+    and this command should return 1:
 
     ```bash { ret_code="1" }
     test -f myfile.txt
@@ -71,16 +72,21 @@ If a specific return code is expected, it can be specified as follows:
 
 #### Command output
 
-When a command output is displayed in the instructions:
+You can display the shell which the command should run in by specifying the `command_line` option. You can also test for expected output in the instructions by specifying the line(s) where the expected output should be displayed. This is done by adding the pipe symbol (`|`). Since the first line should contain the command itself, the indexing of the expected output lines starts at 2. You can specify a span (if the expected output is more than one line) with the dash symbol, for example `"| 2-10"`.
 
 ```markdown
     Let's check is this command return the expected output:
 
-    ```bash { command_line="root@localhost | 2 }
+    ```bash { command_line="root@localhost | 2" }
     echo "hello world"
     hello world
     ```
 ```
+The code above renders to display the shell identity to the reader:
+```bash { command_line="root@localhost | 2" }
+echo "hello world"
+hello world
+```
 
 The framework will check if the command returns the same output and report an error otherwise.
 
@@ -115,7 +121,7 @@ It is important to note that the framework does run each code block as a separat
 
     This command will fail:
 
-    ```bash { ret_code="1" }
+    ```bash
     test -f myfile.txt
     ```
 
@@ -210,7 +216,7 @@ test_images:
 
 The `test_maintenance` field is a boolean that enables the framework.
 
-The `test_images` field is a list of Docker container images the framework can pull to test the Learning Path instructions. Check [Docker Hub](https://hub.docker.com/) to explore available images. 
+The `test_images` field is a list of Docker container images the framework can pull to test the Learning Path instructions. Check [Docker Hub](https://hub.docker.com/) to explore available images.
 
 ## Run the framework
 
@@ -228,48 +234,41 @@ Specify the `.md` file directly for single file tool install articles.
 ./tools/maintenance.py -i content/install-guides/mytool.md
 ```
 
-## Result summary
-
-The framework patches the metadata in the Learning Path's `_index.md` file or the .md file of the tool install to add a summary of the test status.
-
-```yaml
-test_maintenance: true
-test_images:
-- ubuntu:latest
-- fedora:latest
-test_status:
-- passed
-- failed
-```
+If the tests are successful, that will be communicated through the console.
 
-The field `test_status` is a list that indicated whether all tests passed for a corresponding Docker container image or if at least one test failed. 
+### Investigating failures
 
-In the example above, the summary indicates that for this Learning Path all tests passed for the image `ubuntu:latest` but at least one test failed for the image `fedora:latest`. More information about the failures are stored in Junit XML files.
+The framework will print information about any errors to the console. For a more refined analysis, add the `--debug` flag.
 
-## Visualize results
+```bash
+./tools/maintenance.py -i content/install-guides/mytool.md --debug
+```
 
-Test results are stored in XML Junit files. One XML file is created by Learning Path sub-article.
 
-It is possible to visualize the results in a web browser. The XML files can be converted with [xunit-viewer](https://www.npmjs.com/package/xunit-viewer). 
+### Saving the results
 
-If not already installed, install [Node.js](https://nodejs.org/en/) and run:
+If you want the results to be saved, add the `--stats-report` flag to the command. This will update a statistics file which publishes the result to the website.
 
+```bash
+./tools/maintenance.py -i content/install-guides/mytool.md --stats-report
 ```
-npm i -g xunit-viewer
+
+```yaml
+tests_and_status:
+  - ubuntu:latest: passed
+  - fedora:latest: failed
 ```
 
-Then, launch the web server (e.g. on port 5050) on the folder where the XML Junit files have been created:
+The field `tests_and_status` is a list that indicated whether all tests passed for a corresponding Docker container image or if at least one test failed.
 
-```
-xunit-viewer -r content/learning-paths/servers-and-cloud-computing/mynewlearningpath/ -s -p 5050
-```
+In the example above, the summary indicates that for this Learning Path all tests passed for the image `ubuntu:latest` but at least one test failed for the image `fedora:latest`. More information about the failures can be found in the console.
 
 ## Advanced usage for embedded development
-#### Using the Corstone-300 FVP
+### Using the Corstone-300 FVP
 
-By default, the framework runs instructions on the Docker images specified by the [metadata](#edit-metadata). For embedded development, it is possible to build software in a container instance and then check its behaviour on the Corstone-300 FVP. 
+By default, the framework runs instructions on the Docker images specified by the [metadata](#edit-metadata). For embedded development, it is possible to build software in a container instance and then check its behaviour on the Corstone-300 FVP.
 
-For this, all container instances used by the test framework mount a volume in `/shared`. This is where software for the target FVP can be stored. To check the execution, the FVP commands just need to be identified as a `fvp` section for the framework. 
+For this, all container instances used by the test framework mount a volume in `/shared`. This is where software for the target FVP can be stored. To check the execution, the FVP commands just need to be identified as a `fvp` section for the framework.
 
 For example: