Remove criticality concept in favor of skip status #3624
Comments
|
Removing criticality is a controversial decision and not everyone is happy about it. Most people rather have skip than criticality, but there are those who'd like to have both. For them #3625 proposes a way to add custom statuses that make it possible to separate tests skipped for different reasons. |
|
Linking to the pull request about removing this #3662 |
|
Things missing from the first PR:
|
|
What's @yanne the situation with |
|
I updated --help documentation related to --critical/--noncritical and also explicitly deprecated both of them in 594ba01. Now the remaining task ought to be updating the User Guide. |
- Add new "Test and suite statuses" section explaining PASS, FAIL and SKIP. - Move existing SKIP related documentation under the new section. - Move also existing docs related to Pass Execution there. - Remove "Old criticality functionality" section in favor of explaining all that in the SKIP section. - Reorganize and slightly enhance docs about provided special exceptions to e.g. skip tests. After these documentation enhancements the new SKIP status (#3622) and removal of criticality (#3624) ought to be adequately documented.
|
Criticality has been removed for some time now and now its also documented. This issue is done. |
- Add new "Test and suite statuses" section explaining PASS, FAIL and SKIP. - Move existing SKIP related documentation under the new section. - Move also existing docs related to Pass Execution there. - Remove "Old criticality functionality" section in favor of explaining all that in the SKIP section. - Reorganize and slightly enhance docs about provided special exceptions to e.g. skip tests. After these documentation enhancements the new SKIP status (robotframework#3622) and removal of criticality (robotframework#3624) ought to be adequately documented.
Robot Framework currently has a concept of criticality that makes it possible to run tests so that their failures don't affect the overall test execution verdict. The motivation for this feature is acceptance test driven development (ATDD) where you create tests before implementing features and those tests naturally cannot pass initially. In addition to that, this functionality has been used for emulating skipping tests by dynamically marking them non-critical before failing. The system works by using
--criticaland--noncriticaloptions matching tests by tags.Although this functionality has worked ok in its designed usage, it also has several problems and it will also become redundant when RF 4.0 adds real skip status (#3622). This issue first goes through the aforementioned problems in more detail and then explains how the forthcoming skip functionality can be used in the ATDD context to get same functionality that criticality currently provides.
Problems with criticality
RF 4.0 will introduce real skip status (New
SKIPstatus #3622) which is conceptually very close to the criticality functionality. There are some differences, but these featues are so close that having both doesn't add much benefits but instead causes confusion and adds unnecessary complexity.Criticality makes the final outcome of a test two dimensional so that one axis is the actual status and the other is criticality. Even with only pass and fail statuses we end up with four different end results "critical pass", "critical fail", "non-critical pass" and "non-critical fail", and adding the skip status to the mix would add "critical skip" and "non-critical skip". Most of these final statuses make no sense and everything is a lot easier if there's only "pass", "fail" and "skip".
When looking at suite statistics in reports and logs, Robot only shows the number of all passed and failed tests without any indication are failures critical or not. We have experimented showing statistics separately both for critical and non-critical tests but that didn't work well at all. This is similar problem as the one above and having just pass, fail and skip statuses resolved this one as well.
Criticality is a unique feature in Robot Framework. Unique tool features can be really useful, but they also require learning by new (and old) users and they don't always play nicely together with other tools. In this particular case skip is a familiar feature for most people working with automation and it's also a functionality that external tools like test management systems generally support.
Related to the above, having statistics both for "Critical Tests" and "All Tests" in reports and logs is rather strange especially for new users. Just having single statistics with pass, skip and fail statuses is a lot simpler and intuitive.
Migrating from criticality to skipping
Part of the new skip feature (#3622) is adding
--skiponfailurecommand line option that automatically changes status of tests having a matching tag to skip if they fail. This works very much like the current--noncriticaloption that marks tests non-critical and thus their failures are in practice ignored. To make migration to skipping easier,--noncriticaland also--criticalwill be preserved as aliases to--skiponfailurein RF 4.0.Although
--noncriticaland--criticalwill continued to work mostly like earlier, there are various other changes affecting the current criticality users. Especially visible are changes in reports and logs where critical/non-critical distinction will be gone. It will, however, be easy to see statistics for all tests having a critical or non-critical tag in the tag statistics table. Other changes include removing thecriticalattribute fromtestelements in output.xml and changes to the result related APIs.Migrating to skipping very importantly requires changes to integration with external tools. This will certainly add some work to projects providing such integration (e.g. Robot Framework Jenkins Plugin), but in the end using commonly used skip status and not the unique criticality is likely to make things easier.
The text was updated successfully, but these errors were encountered: