Track implementation for MC/DC

[ZhuUx]

Introduction

Modified condition/decision coverage (MC/DC) is a code coverage criterion used widely in safety critical software components and is required by standards such as DO-178B and ISO26262.

Terminology
condition: boolean expressions that have no binary logical operators. For example, a || b is not "condition" because it has an or operator while a==b is.
decision: boolean expressions composed of conditions and binary boolean expressions.

MC/DC requires each condition in a decision is shown to independently affect the outcome of the decision.
e.g Suppose we have code like

if (a || b) && c {
    todo!();
}

Here (a || b) && c is a decision and a,b,c are conditions.

• If test cases are (a=true, b=false, c=true) and (a=false, b=false, c=true), we say a can independently affect the decision because the value of decision is changed as a changes while keep b and c unchanged. So that we get 1/3 MC/DC here (1 for a and 3 for a,b,c).
• Test cases (a=false, b=true, c=true) and (a=false, b=false, c=false) also show b can independently affect the decision because in the later case c is short-circuited and has no impacts (thus we can view it as same as c=true). However, c is not acknowledged due to change of b. Plus the two cases before we get 2/3 MC/DC.
• Test cases (a=true,b=false,c=true) and (a=true,b=false,c=false) show c can do the same. By now we get 3/3.
  Notice that there are duplicate cases, so test cases collection {(a=true, b=false, c=true),(a=false, b=false, c=true),(a=false, b=true, c=true), (a=false, b=false, c=false),(a=true,b=false,c=false)} is sufficient to prove 3/3 MC/DC.
  In fact we can use at least n+1 cases to prove 100% MC/DC of decision with n conditions. (In this example, {(a=true,b=false,c=true),(a=false,b=false,c=true),(a=false,b=true,c=true),(a=true,b=false,c=false)} is enough)

Progress

A basic implementation for MC/DC is filed on #123409 , which has some limits. There are still several cases need to handle:

• Support lazy boolean expressions like let val = a || b. For now this is impacted by branch coverage and work in a different style with certification standards where usually MC/DC is required. To solve this, 124120 is the prerequisite.
  Draft: Coverage: Add condition coverage #124402
• Support if-let pattern. Ferrous suggests that refutable pattern matching should be considered as decisions, which sounds certainly reasonable as these are considered as branches too. This depends on support from branch coverage (see 124118).
  Draft: Support mcdc analysis for pattern matching #124278
  Besides, decisions in match guard might still be left to discuss.
• Support nested decisions like if (a || b) || inner_decision(c || d)
  Done by MCDC coverage: support nested decision coverage #124255
• So far only decisions with at most 6 conditions are counted due to resource cost. This limit should be relaxed once if the llvm backend had this optimization.

[ZhuUx]

@rustbot label +A-code-coverage

[workingjubilee]

The tracking issue should probably include a (brief) definition of MCDC, as otherwise this is untrackable by anyone who does not already know what the issue is for.

[RenjiSann]

condition: boolean expressions that have no binary operators. For example, a || b is not "condition" because it has an or operator while a==b is.

nit on the terminology, but I'd rather say that conditions are boolean expressions that have no logical operator.
cf the BinOp and LogicalOp definitions.

[workingjubilee]

Excellent, that's much better, thank you! The teams every now and then review tracking issues and it is helpful if whoever is participating in the review can understand what the tracking issue is about without any domain expertise (even if only so they can fetch the appropriate subject-matter expert).

[ZhuUx]

nit on the terminology, but I'd rather say that conditions are boolean expressions that have no logical operator.

Thanks for correctness! I should have written "binary logical operators". It's a bit different with normal definition because I think treat conditions + unary operators as condition does not lead to different results in MC/DC.

[ZhuUx]

Upon implementing mcdc for pattern match, I found that decisions here might be confusing.
Consider pattern if let (A | B, C) = value, intuitively the decision here likes (matches(A) || matches(B)) && matches (C).
However, compiler will rearrange it and make its CFG look like matches (C) && (matches(A) || matches(B)) to reduce complexity.
This impacts mcdc analysis because the condition short-circuited is changed. We should find a way to determine the rearranged decision.

[RenjiSann]

I wonder how should let-chains be instrumented.

for example:

if let (A | B, C) = value && let Some(X | Y) = other && (x || y) {
}

Here, I can't decide whether we should insert 1 or 3 MCDC decision records here

• 3 decisions: 1 for each refutable pattern matching, and 1 for the top-level boolean expression
• 1 decision with

Maybe the 3 decisions would be better and easier to do, but I fear this would add a lot of verbosity to the report, and also increase the number of conditions in decisions, which therefore increase the complexity of the testing.

What do you think ?

[ZhuUx]

I wonder how should let-chains be instrumented.

for example:

if let (A | B, C) = value && let Some(X | Y) = other && (x || y) {
}

Here, I can't decide whether we should insert 1 or 3 MCDC decision records here

* 3 decisions: 1 for each refutable pattern matching, and 1 for the top-level boolean expression

* 1 decision with

Maybe the 3 decisions would be better and easier to do, but I fear this would add a lot of verbosity to the report, and also increase the number of conditions in decisions, which therefore increase the complexity of the testing.

What do you think ?

I'd tend to 3 decisions as it's more clear and easy to be implemented.
I have managed to construct decisions for refutable pattern matching and found it is very different with normal boolean expressions (eg. rustc does not promise evaluated order of pattern matching). Thus I'd rather view refutable pattern matching as something like try_extract_data_from(&mut bindings...) -> bool.

Besides, if we want to present consistent results in mcdc, we would better take all refutable pattern matching as decisions or not, otherwise we might face paradox like branch coverage in lazy expressions now.

[RenjiSann]

I'd tend to 3 decisions as it's more clear and easy to be implemented.
I have managed to construct decisions for refutable pattern matching and found it is very different with normal boolean expressions (eg. rustc does not promise evaluated order of pattern matching). Thus I'd rather view refutable pattern matching as something like try_extract_data_from(&mut bindings...) -> bool.

That makes sense, indeed. Let's go with this strategy then :)

Also, I think it would make sense to allow the user to disable pattern matching instrumentation if needed (for example to reduce the binary size if needed). Maybe we could do this with a flag -Z coverage-options=no-mcdc-in-pattern-matching (or smth shorter if possible).

[ZhuUx]

Maybe we could do this with a flag -Z coverage-options=no-mcdc-in-pattern-matching (or smth shorter if possible).

Good idea, -no-pattern-matching may be enough since normal branch coverage might also use this option and it is mutual exclusive with mcdc to some extent for now.

[Zalathar]

I have a few cleanup steps planned for after #123409/#124255:

• coverage: Replace boolean options with a CoverageLevel enum #124507
• coverage: Avoid hard-coded values when visiting logical ops #124508
• I want to split the compiler/rustc_mir_transform/src/coverage/spans.rs file into two files, one containing the existing span-processing code, and a separate file containing BcbMapping and the code that extracts branch/mcdc info from MIR.
  • coverage: Split off mappings.rs from spans.rs and from_mir.rs #124545
• I would like to slightly change the way that nested condition bitmaps are counted, because I think the current way makes it a bit too easy to make off-by-1 mistakes.
• Inside mcdc_parameters there's a big chunk of code inside an unsafe block, which I would like to try to split into smaller and safer pieces.
  • coverage: Clean up creation of MC/DC condition bitmaps #124555