-
-
Notifications
You must be signed in to change notification settings - Fork 367
/
code_coverage.dd
272 lines (226 loc) · 7.98 KB
/
code_coverage.dd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
Ddoc
$(D_S Code Coverage Analysis,
$(P A major part of the engineering of a professional software project
is creating a test suite for it. Without some sort of test suite,
it is impossible to know if the software works at all.
The D language has
many features to aid in the creation of test suites, such as
$(DDLINK spec/unittest, Unit Tests, unit tests) and
$(DDLINK spec/contracts, Contract Programming, contract programming).
But there's the issue of how thoroughly the test suite tests
the code.
The $(LINK2 http://www.digitalmars.com/ctg/trace.html, profiler)
can give valuable information on which functions were called, and
by whom. But to look inside a function, and determine which statements
were executed and which were not, requires a code coverage analyzer.
)
$(P A code coverage analyzer will help in these ways:)
$(OL
$(LI Expose code that is not exercised by the test suite.
Add test cases that will exercise it.)
$(LI Identify code that is unreachable. Unreachable code is often
the leftover result of program design changes.
Unreachable code should be removed,
as it can be very confusing to the maintenance programmer.)
$(LI It can be used to track down why a particular section of code
exists, as the test case that causes it to execute will
illuminate why.)
$(LI Since execution counts are given for each line, it is possible
to use the coverage analysis to reorder the basic blocks in
a function to minimize jmps in the most used path, thus
optimizing it.)
)
$(P Experience with code coverage analyzers show that they dramatically
reduce the number of bugs in shipping code.
But it isn't a panacea, a code coverage analyzer won't help with:)
$(OL
$(LI Identifying race conditions.)
$(LI Memory consumption problems.)
$(LI Pointer bugs.)
$(LI Verifying that the program got the correct result.)
)
$(P Code coverage analysers are available for many popular languages,
but they are often third party products that integrate
poorly with the compiler, and are often very expensive.
A big problem with third party products is, in order to instrument
the source code, they must include what is essentially a full blown
compiler front end for the same language. Not only is this an expensive
proposition, it often winds up out of step with the various compiler
vendors as their implementations change and as they evolve various extensions.
($(LINK2 http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html, gcov),
the Gnu coverage analyzer, is an exception as it is both free
and is integrated into gcc.)
)
$(P The D code coverage analyser is built in as part of the D compiler.
Therefore, it is always in perfect synchronization with the language
implementation. It's implemented by establishing a counter for each
line in each module compiled with the $(DDSUBLINK dmd,switch-cov, $(B -cov)) switch.
Code is inserted
at the beginning of each statement to increment the corresponding counter.
When the program finishes, the runtime collects all
the counters, merges it with the source files, and writes the reports out
to listing (.lst) files.)
$(P For example, consider the Sieve program:)
----------------------
/* Eratosthenes Sieve prime number calculation. */
import std.stdio;
bool flags[8191];
int main()
{
int i, prime, k, count, iter;
writeln("10 iterations");
for (iter = 1; iter <= 10; iter++)
{
count = 0;
flags[] = true;
for (i = 0; i < flags.length; i++)
{
if (flags[i])
{
prime = i + i + 3;
k = i + prime;
while (k < flags.length)
{
flags[k] = false;
k += prime;
}
count += 1;
}
}
}
writefln("%d primes", count);
return 0;
}
----------------------
$(P Compile and run it with:)
$(CONSOLE
dmd sieve -cov
sieve
)
$(P The output file will be created called $(D sieve.lst), the contents of
which are:)
$(CONSOLE
|/* Eratosthenes Sieve prime number calculation. */
|
|import std.stdio;
|
|bool flags[8191];
|
|int main()
|{
5| int i, prime, k, count, iter;
|
1| writeln("10 iterations");
22| for (iter = 1; iter <= 10; iter++)
| {
10| count = 0;
10| flags[] = true;
163840| for (i = 0; i < flags.length; i++)
| {
81910| if (flags[i])
| {
18990| prime = i + i + 3;
18990| k = i + prime;
168980| while (k < flags.length)
| {
149990| flags[k] = false;
149990| k += prime;
| }
18990| count += 1;
| }
| }
| }
1| writefln("%d primes", count);
1| return 0;
|}
sieve.d is 100% covered
)
$(P The numbers to the left of the $(B |) are the execution counts for that
line. Lines that have no executable code are left blank.
Lines that have executable code, but were not executed, have a "0000000"
as the execution count.
At the end of the .lst file, the percent coverage is given.
)
$(P There are 3 lines with an exection count
of 1, these were each executed once. The declaration line for $(D i, prime),
etc., has 5 because there are 5 declarations, and the initialization of
each declaration counts as one statement.)
$(P The first $(D for) loop shows 22. This is the sum of the 3 parts
of the for header. If the for header is broken up into 3 lines, the
data is similarly divided:)
$(CONSOLE
1| for (iter = 1;
11| iter <= 10;
10| iter++)
)
$(P which adds up to 22.)
$(P $(D e1&&e2) and $(D e1||e2) expressions conditionally
execute the right-hand operand $(D e2).
Therefore, the right-hand operand is treated as a separate statement with its
own counter:)
$(CONSOLE
|void foo(int a, int b)
|{
5| bar(a);
8| if (a && b)
1| bar(b);
|}
)
$(P By putting the right-hand operand on a separate line, this illuminates
things:)
$(CONSOLE
|void foo(int a, int b)
|{
5| bar(a);
5| if (a &&
3| b)
1| bar(b);
|}
)
$(P Similarly, for the $(D e?e1:e2) expressions, $(D e1) and
$(D e2) are treated as separate statements.)
$(H3 Controlling the Coverage Analyser)
$(COMMENT The behavior of the coverage analyser can be controlled through
the $(DRUNTIMESRC rt/cover.d) module.)
$(P When the $(DDSUBLINK dmd,switch-cov, $(B -cov)) switch is thrown,
the $(DDSUBLINK spec/version, PredefinedVersions, version identifier)
$(B D_Coverage) is defined.)
$(P The standard runtime option passing mechanism can also be used to control how
code coverage reports are generated at runtime. The command-line option $(D
--DRT-covopt) or the environment variable $(D DRT_COVOPT) can be used to pass
options. Options are passed separated by spaces as key,value with the following
format: $(D key:value). Current options are:)
$(DL
$(SWITCH $(SWNAME merge),
Merge the current run with existing reports if `1`, or overwrite the existing reports if `0`.
)
$(SWITCH $(SWNAME srcpath),
Set path to where source files are located.
)
$(SWITCH $(SWNAME dstpath),
Set path to where listing files are to be written (must exist).
)
)
$(H4 Example:)
$(CONSOLE
sieve --DRT-covopt="merge:1"
mkdir reports
sieve --DRT-covopt="merge:1 dstpath:reports"
)
$(P Will merge the results of this run with the previous run in the $(D
sieve.lst) file in the current directory. While:)
$(CONSOLE
mkdir reports
sieve --DRT-covopt="dstpath:reports"
)
$(P Will create a new report in $(D reports/sieve.lst). Another run adding $(D
merge) will add to the new report:)
$(CONSOLE
sieve --DRT-covopt="merge:1 dstpath:reports"
)
$(H3 References)
$(LINK2 https://en.wikipedia.org/wiki/Code_coverage, Wikipedia)
)
Macros:
TITLE=Code Coverage Analysis
SUBNAV=$(SUBNAV_ARTICLES)