|
| 1 | +# Assembly Tests |
| 2 | + |
| 3 | +The Benchmark library provides a number of functions whose primary |
| 4 | +purpose in to affect assembly generation, including `DoNotOptimize` |
| 5 | +and `ClobberMemory`. In addition there are other functions, |
| 6 | +such as `KeepRunning`, for which generating good assembly is paramount. |
| 7 | + |
| 8 | +For these functions it's important to have tests that verify the |
| 9 | +correctness and quality of the implementation. This requires testing |
| 10 | +the code generated by the compiler. |
| 11 | + |
| 12 | +This document describes how the Benchmark library tests compiler output, |
| 13 | +as well as how to properly write new tests. |
| 14 | + |
| 15 | + |
| 16 | +## Anatomy of a Test |
| 17 | + |
| 18 | +Writing a test has two steps: |
| 19 | + |
| 20 | +* Write the code you want to generate assembly for. |
| 21 | +* Add `// CHECK` lines to match against the verified assembly. |
| 22 | + |
| 23 | +Example: |
| 24 | +```c++ |
| 25 | + |
| 26 | +// CHECK-LABEL: test_add: |
| 27 | +extern "C" int test_add() { |
| 28 | + extern int ExternInt; |
| 29 | + return ExternInt + 1; |
| 30 | + |
| 31 | + // CHECK: movl ExternInt(%rip), %eax |
| 32 | + // CHECK: addl %eax |
| 33 | + // CHECK: ret |
| 34 | +} |
| 35 | + |
| 36 | +``` |
| 37 | +
|
| 38 | +#### LLVM Filecheck |
| 39 | +
|
| 40 | +[LLVM's Filecheck](https://llvm.org/docs/CommandGuide/FileCheck.html) |
| 41 | +is used to test the generated assembly against the `// CHECK` lines |
| 42 | +specified in the tests source file. Please see the documentation |
| 43 | +linked above for information on how to write `CHECK` directives. |
| 44 | +
|
| 45 | +#### Tips and Tricks: |
| 46 | +
|
| 47 | +* Tests should match the minimal amount of output required to establish |
| 48 | +correctness. `CHECK` directives don't have to match on the exact next line |
| 49 | +after the previous match, so tests should omit checks for unimportant |
| 50 | +bits of assembly. ([`CHECK-NEXT`](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-next-directive) |
| 51 | +can be used to ensure a match occurs exactly after the previous match). |
| 52 | +
|
| 53 | +* The tests are compiled with `-O3 -g0`. So we're only testing the |
| 54 | +optimized output. |
| 55 | +
|
| 56 | +* The assembly output is further cleaned up using `tools/strip_asm.py`. |
| 57 | +This removes comments, assembler directives, and unused labels before |
| 58 | +the test is run. |
| 59 | +
|
| 60 | +* The generated and stripped assembly file for a test is output under |
| 61 | +`<build-directory>/test/<test-name>.s` |
| 62 | +
|
| 63 | +* Filecheck supports using [`CHECK` prefixes](https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes) |
| 64 | +to specify lines that should only match in certain situations. |
| 65 | +The Benchmark tests use `CHECK-CLANG` and `CHECK-GNU` for lines that |
| 66 | +are only expected to match Clang or GCC's output respectively. Normal |
| 67 | +`CHECK` lines match against all compilers. (Note: `CHECK-NOT` and |
| 68 | +`CHECK-LABEL` are NOT prefixes. They are versions of non-prefixed |
| 69 | +`CHECK` lines) |
| 70 | +
|
| 71 | +* Use `extern "C"` to disable name mangling for specific functions. This |
| 72 | +makes them easier to name in the `CHECK` lines. |
| 73 | +
|
| 74 | +
|
| 75 | +## Problems Writing Portable Tests |
| 76 | +
|
| 77 | +Writing tests which check the code generated by a compiler are |
| 78 | +inherently non-portable. Different compilers and even different compiler |
| 79 | +versions may generate entirely different code. The Benchmark tests |
| 80 | +must tolerate this. |
| 81 | +
|
| 82 | +LLVM Filecheck provides a number of mechanisms to help write |
| 83 | +"more portable" tests; including [matching using regular expressions](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-pattern-matching-syntax), |
| 84 | +allowing the creation of [named variables](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-variables) |
| 85 | +for later matching, and [checking non-sequential matches](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive). |
| 86 | +
|
| 87 | +#### Capturing Variables |
| 88 | +
|
| 89 | +For example, say GCC stores a variable in a register but Clang stores |
| 90 | +it in memory. To write a test that tolerates both cases we "capture" |
| 91 | +the destination of the store, and then use the captured expression |
| 92 | +to write the remainder of the test. |
| 93 | +
|
| 94 | +```c++ |
| 95 | +// CHECK-LABEL: test_div_no_op_into_shr: |
| 96 | +extern "C" void test_div_no_op_into_shr(int value) { |
| 97 | + int divisor = 2; |
| 98 | + benchmark::DoNotOptimize(divisor); // hide the value from the optimizer |
| 99 | + return value / divisor; |
| 100 | +
|
| 101 | + // CHECK: movl $2, [[DEST:.*]] |
| 102 | + // CHECK: idivl [[DEST]] |
| 103 | + // CHECK: ret |
| 104 | +} |
| 105 | +``` |
| 106 | + |
| 107 | +#### Using Regular Expressions to Match Differing Output |
| 108 | + |
| 109 | +Often tests require testing assembly lines which may subtly differ |
| 110 | +between compilers or compiler versions. A common example of this |
| 111 | +is matching stack frame addresses. In this case regular expressions |
| 112 | +can be used to match the differing bits of output. For example: |
| 113 | + |
| 114 | +```c++ |
| 115 | +int ExternInt; |
| 116 | +struct Point { int x, y, z; }; |
| 117 | + |
| 118 | +// CHECK-LABEL: test_store_point: |
| 119 | +extern "C" void test_store_point() { |
| 120 | + Point p{ExternInt, ExternInt, ExternInt}; |
| 121 | + benchmark::DoNotOptimize(p); |
| 122 | + |
| 123 | + // CHECK: movl ExternInt(%rip), %eax |
| 124 | + // CHECK: movl %eax, -{{[0-9]+}}(%rsp) |
| 125 | + // CHECK: movl %eax, -{{[0-9]+}}(%rsp) |
| 126 | + // CHECK: movl %eax, -{{[0-9]+}}(%rsp) |
| 127 | + // CHECK: ret |
| 128 | +} |
| 129 | +``` |
| 130 | +
|
| 131 | +## Current Requirements and Limitations |
| 132 | +
|
| 133 | +The tests require Filecheck to be installed along the `PATH` of the |
| 134 | +build machine. Otherwise the tests will be disabled. |
| 135 | +
|
| 136 | +Additionally, as mentioned in the previous section, codegen tests are |
| 137 | +inherently non-portable. Currently the tests are limited to: |
| 138 | +
|
| 139 | +* x86_64 targets. |
| 140 | +* Compiled with GCC or Clang |
| 141 | +
|
| 142 | +Further work could be done, at least on a limited basis, to extend the |
| 143 | +tests to other architectures and compilers (using `CHECK` prefixes). |
| 144 | +
|
| 145 | +Furthermore, the tests fail for builds which specify additional flags |
| 146 | +that modify code generation, including `--coverage` or `-fsanitize=`. |
| 147 | +
|
0 commit comments