Suite setup and teardown are executed even if all tests are skipped

[gorern]

When all test cases in a suite are skipped because of the --skip command line option and a matching tag, the Suite Setup is executed, even when no test cases in the suite are executed. If the setup fails, all test cases will get status FAIL, even tough they shall be skipped.

It doesn't matter if each individual test case is tagged using [TAGS] or if the Force Tags setting is used.

A workaround is to use both the --skip and --skiponfailure options with the same tag. The status for all the keywords will then be SKIP. However, the Suite Setup will still be executed, which is a problem for us because the failure may leave our system in an undefined state.

We have previously used the --exclude option. Then the Suite Setup was not executed in the case described above. We switched to the --skip option because we want the not executed tests to be visible in the logs.

We are using Robot Framework 6.0.1.

Example:

*** Settings ***
Suite Setup    My Setup


*** Test Cases ***
Try skipMe
    [Tags]    skipMe
    Log    This should not be executed


*** Keywords ***
My Setup
    Log    This Setup will fail
    Fail

Executing the example:

$ robot --skip skipMe example.robot 
==============================================================================
Example                                                                       
==============================================================================
Try skipMe                                                            | FAIL |
Parent suite setup failed:
AssertionError
------------------------------------------------------------------------------
Example                                                               | FAIL |
Suite setup failed:
AssertionError

1 test, 0 passed, 1 failed
==============================================================================
Output:  /home/mbs/rf/output.xml
Log:     /home/mbs/rf/log.html
Report:  /home/mbs/rf/report.html
 

[mluyun]

Also encountered this in our test executions. This is probably not in line with the documented Skip feature behavior. If all test cases are skipped, the test suite should be skipped in the execution

The SKIP status also affects the statuses of the executed suites. Their statuses are set based on test statuses using these rules:

If there are failed tests, suite status is FAIL.
If there are no failures but there are passed tests, suite status is PASS.
If there are only skipped tests, or no tests at all, suite status is SKIP.

The return code to the system is the number of failed tests, skipped tests do not affect it.

[pekkaklarck]

Yeah, it's pretty stupid to run suite setup and/or teardown if all tests are unconditionally skipped. It ought to be pretty easy to check that based on tags before running the suite setup/teardown. We cannot detect would tests be skipped dynamically during execution, though. I consider this a bug and assign this to RF 6.1 milestone. I cannot promise we get it fixed already in that release, so if you consider this important take a look at creating a pull request.

The reason excluding tests behaves differently is that those tests are removed from the suite altogether and suites that become empty are removed as well. When skipping tests, suites aren't empty and suite setup/teardown is run unconditionally without looking are tests skipped or not.

[pekkaklarck]

Same thing happens if tests are excluded using robot:exclude tag. Normal --exclude is handled before tests are run, but this special tag is only looked at before individual tests are run.

[mluyun]

Thank you @pekkaklarck for the fix! 💯