Make --test and --include non-cumulative again due to backwards incompatibility problems

[kjkaizens]

robot --rerunfailed output.xml until robot version 6 picks up tests that had been failed in previous run. However, robot 7 just reruns all suite, both failed and passed tests

[pekkaklarck]

Could you please clarify?

[siongwai]

Hi,

I have encounter same problem as well, but I don't have idea to collect stack trace within robot framework, so I am sharing the screenshot about the command and console outputs.

Hope it help to elaborate clearer.

RF6:

RF7:

[pekkaklarck]

Could someone explain what happens and what you expected to happen? An example is fine, but please use text, not screenshots.

[kjkaizens]

Example

What was expected:(and what used to work with robot 6)

robot test.robot

Runs 20 tests. 17 passed 3 failed

robot --rerunfailed test.robot
Runs failed 3 failed tests

What happens? (with robot 7)

robot test.robot

Runs 20 tests. 17 passed 3 failed

robot --rerunfailed test.robot
Runs all 20 tests again

[pekkaklarck]

I tried reproducing this issue with the following data:

*** Test Cases ***
T1
   log  1
T2
   log  2
T3
   log  3
T4
   log  4
T5
   fail  5
T6
   log  6
T7
   log  7
T8
   log  8
T9
   fail  9
T10
   log  10

When I executed it with

robot tests.robot

two tests fail as expected. When I then use

robot --rerunfailed output.xml tests.robot

only the two failed tests are run again as expected.

There can be something different in your case, but the basic stuff certainly works. If you want us to investigate this, you need to provide a concrete example that we can run. Related to that, robot --rerunfailed test.robot that you used in your example isn't even a valid.

[rarajabs]

Hi,

I am able to reproduce when I filter tests with tag in robot command
robot --rerunfailed output.xml --include failed ".\tests.robot"

[rarajabs]

Addition: I have added a tag in Settings section of tests.robot, so i can run tests by tag with robot command

[pekkaklarck]

I can reproduce the change in behavior when using --rerunfailed and --include together. Is that what others are seeing or do you have some other problem?

The behavior change with --include is a side-effect of changing how --test and --include work together (#4721). In general these options for selecting which tests are run are nowadays cumulative when earlier tests needed to match all options. The behavior with --test and --include certainly won't be changed back, the change was done on purpose, but we can discuss how --rerunfailed and --include used together should behave. I personally don't see problems with them being cumulative similarly as other options. If you just want the failed tests to be run, you simply need to use only --rerunfailed.

[dverweij-dev]

I ran into the same issue when migrating from RF 6.1.1 to RF 7.0. If the --include (and in my case also the--exclude ) is specified, all the test run again and not only the ones that failed. I always thought that--rerunfailed means: select all tests that failed and only run the ones which match the --include tags (and also exclude the ones matching the --exclude tags). RF 6.1.1 shows this behavior. I don’t have a problem with the new behavior, in most cases I just want to rerun all the failed tests, but it is not downwards compatible anymore, so we have to change some scripts/pipelines. This not a big deal of course, but maybe it is a good idea to bring back the old behavior with respect to --rerunfailed, to avoid confusion and make it downwards compatible again.

[pekkaklarck]

I agree that the old behavior when using --rerunfailed and --include/--exclude was fine, but I also consider the current behavior ok, and I'm not certain does it make the change it back. The problem with changing it is that --rerunfailed is internally implemented so that it uses the same functionality as --test, and I pretty strongly believe that we should preserve the current behavior when using --test and --include/--exclude together. We certainly can change --rerunfailed implementation, but it makes this a bit bigger task.

I'll ask from other users on our Slack how do they feel about this.

[kjkaizens]

developing new features and ideas are always welcomed, but not considering legacy support is to be considered too.

[pekkaklarck]

The change with --test and --include was done one purpose and issue #4721 was marked backwards incompatible. The change with --rerunfailed and --include was an unexpected side-effect. If we agree that the old behavior in that context was better, we can change the --rerunfailed/--include behavior to same as earlier.

[dverweij-dev]

Thanks for your quick response. I agree with your proposal.

[pekkaklarck]

We need to make a decision about this. Adding to RF 7.0.1 scope so we don't forget about it.

[Wolfe1]

Just chiming in that we are experiencing this issue as well and would like to see the previous behavior reinstated. Regardless of --include/--exclude the tests are not failed and should not be rerun.

[pekkaklarck]

The old behavior will be restored. There weren't that much benefits with #4721, so it's better to revert that than try to fix --rerunfailed otherwise. This fixes similar issues reported, for example, with Pabot as well.

[pekkaklarck]

I hopefully get other open issues in RF 7.0.1 done soon and will then create a release candidate with all the fixes. I hope you can test it when it's available to make sure the behavior with --test, --include and --rerunfailed is as expected also in your environment.

[pekkaklarck]

RF 7.0.1 rc 1 with this fix has just been released. It would be great if you could test does the fix work properly before the final release that's targeted for Monday June 10.

[dverweij-dev]

Tested scenario with --rerunfailed and --include (and also --exclude and --test ) with RF 7.0.1 rc1 and works fine for me. Thanks for fixing this issue!

[neiljhowell]

@pekkaklarck our test with RF 7.0.1 rc1 was successful as well.