[feat] Introduce a --dry-run option

[ekouts]

Adding a "dry-run" attribute instead of a cli option gives to the user more flexibility, in case they want to run a fixture/dependency for some reason. The user can always run with -Sdry_run_mode=1 so that they don't actually change the test in most cases.

I also changed the color of the "OK" and "RUN" messages to yellow for tests that have dry_run_mode=True but we can easily change it back if you don't like it. I just thought that this feature can easily lead to mistakes if someone is setting the test in dry_run_mode without noticing.

Closes #2242.

[codecov-commenter]

Codecov Report

Base: 86.61% // Head: 86.62% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (ceb5615) compared to base (db171e7).
Patch coverage: 95.12% of modified lines in pull request are covered.

❗ Current head ceb5615 differs from pull request most recent head cc40905. Consider uploading reports for the commit cc40905 to get more accurate results

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #2746   +/-   ##
========================================
  Coverage    86.61%   86.62%           
========================================
  Files           60       60           
  Lines        11215    11259   +44     
========================================
+ Hits          9714     9753   +39     
- Misses        1501     1506    +5     

Impacted Files	Coverage Δ
reframe/frontend/executors/policies.py	92.39% <91.66%> (-0.13%)	⬇️
reframe/core/pipeline.py	93.75% <95.00%> (-0.03%)	⬇️
reframe/core/decorators.py	94.66% <100.00%> (+0.07%)	⬆️
reframe/frontend/cli.py	73.71% <100.00%> (+0.04%)	⬆️
reframe/frontend/loader.py	96.07% <100.00%> (+0.05%)	⬆️
reframe/frontend/testgenerators.py	96.00% <100.00%> (ø)
reframe/frontend/statistics.py	95.18% <0.00%> (-0.89%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[vkarak]

Besides my individual comments in the code, I think the proposed changes are very intrusive. I don't think we should touch the scheduler backends. The dry-run mode mechanics should stop in the RegressionTest. If the test is in dry-run mode, we should simply skip the job submission. Similarly, the compile|run_wait() should return immediately and the compile|run_complete() should always return True. As for the sanity and performance phases, I suggest that we enter normally in those stages and exit just before we start the evaluation of the sanity or performance functions. That would allow to reveal test problems even in dry-run mode. Finally, we should probably omit completely the creation of the output directory as nothing will be eventually copied there and the "cleanup" stage should just return immediately. The idea of the dry-run mode is primarily to be able to inspect the generated scripts, which will be left in the stage dir. I would also claim that the copy from sourcesdir to the stagedir should still happen, as it will also reveal potential problems without actually running the test.

[vkarak]

@ekouts The PR is now fine from my side.

[ekouts]

I cannot approve because it's my PR, but @vkarak lgtm 🎉