Fix condition for long-runtime system tests and add user doc section (#…

…294) * Fix condition by splitting it * Add user doc description * Remove unnecessary debug log Co-authored-by: Kaloyan <36224699+krucod3@users.noreply.github.com> * Fix debug logs in ci/cd --------- Co-authored-by: Kaloyan <36224699+krucod3@users.noreply.github.com>
eclipse-ankaios · Jun 13, 2024 · a4719ff · a4719ff
1 parent c8aca0f
commit a4719ff
Show file tree

Hide file tree

Showing 2 changed files with 30 additions and 8 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -25,18 +25,21 @@ jobs:
       - uses: actions/checkout@v4.1.1
       - name: Build debug
         run: cargo build
-      - name: Run system tests
+      - name: Prepare system tests
         run: |
           mkdir -p target/robot_tests_result
           chown vscode target/robot_tests_result
-          if [ ${{ github.event_name == 'pull_request' }} ]; then
-            SYSTEM_TEST_EXCLUDE_TAG="non_execution_during_pull_request_verification"
-            echo "Run is a PR verification, system tests tagged with '$SYSTEM_TEST_EXCLUDE_TAG' are NOT executed and skipped."
-            su vscode -c "tools/run_robot_tests.sh --exclude $SYSTEM_TEST_EXCLUDE_TAG tests"
-          else
-            echo "Run is not a PR verification, all system tests are executed."
+      - name: Run system tests in pull request verification
+        if: ${{ github.event_name == 'pull_request' }}
+        run: |
+          SYSTEM_TEST_EXCLUDE_TAG="non_execution_during_pull_request_verification"
+          echo "Run is a PR verification, system tests tagged with '$SYSTEM_TEST_EXCLUDE_TAG' are NOT executed and skipped."
+          su vscode -c "tools/run_robot_tests.sh --exclude $SYSTEM_TEST_EXCLUDE_TAG tests"
+      - name: Run all system tests
+        if: ${{ github.event_name != 'pull_request' }}
+        run: |
+            echo "Event name is '${{ github.event_name }}', all system tests are executed."
             su vscode -c "tools/run_robot_tests.sh tests"
-          fi
       - uses: actions/upload-artifact@v4.3.3
         if: success() || failure()
         with:

diff --git a/doc/docs/development/system-tests.md b/doc/docs/development/system-tests.md
@@ -112,6 +112,25 @@ Test Ankaios CLI get workloads
     [Teardown]    Clean up Ankaios
 ```
 
+### Run long-runtime system tests upon merge into main
+
+To keep the pull request status check runtime short, system tests with a longer runtime (> 30-40 seconds) shall be excluded from the pull request CI/CD verification by assigning the tag "non_execution_during_pull_request_verification" directly to the test case. When the pull request is merged into the main branch, the system test is executed. A contributor shall check the test results of those system tests afterwards.
+
+Example system test that runs only on merge into main:
+
+```robot hl_lines="7"
+...
+
+*** Test Cases ***
+...
+
+Test Ankaios Podman stops retries after reaching the retry attempt limit
+    [Tags]    non_execution_during_pull_request_verification
+    [Setup]    Run Keywords    Setup Ankaios
+
+...
+```
+
 ## System test execution
 
 !!! warning