[bugfix] Do not wipe the output dir of a test if run with --dry-run

[vkarak]

Closes #3551.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.22%. Comparing base (ff41341) to head (a332dec).
⚠️ Report is 759 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3553      +/-   ##
==========================================
+ Coverage   86.66%   91.22%   +4.55%     
==========================================
  Files          61       62       +1     
  Lines       12140    13465    +1325     
==========================================
+ Hits        10521    12283    +1762     
+ Misses       1619     1182     -437     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ekouts]

For me it's not clear what * should * happen with the old output directory. People may be confused by looking at the output of old runs when they are testing --dry-run. On the other hand I understand that removing the old results with a "dry run" option can also be unwanted behaviour, so lgtm!

[vkarak]

For me it's not clear what * should * happen with the old output directory. People may be confused by looking at the output of old runs when they are testing --dry-run. On the other hand I understand that removing the old results with a "dry run" option can also be unwanted behaviour, so lgtm!

Removing the output directory is a totally undesired side effect, as this is supposed to hold scripts and data from what it has been actually executed. In the docs we state that --dry-run generates the scripts in the stage directory and, also, in dry-run mode we do not copy back to the output directory anything from the test, because what is in keep_files is likely not produced. So, essentially, the --dry-run mode was literally just wiping out the output directory.