My Copy of Bugbench

Bugbench is a benchmark suite created by Shan Lu.

Bug signature example is in this separate file.

Why creating this repo?

• I need to access bugbench many times
• The copy I got needs some modification (compiler flags) to run on Linux
• I need take some notes
• I want to create bug signature along with it

Statistics

benchmark	deterministic	original benchmark	patched version	manual bug signature	patched version	Comment
gzip-1.2.4	Y	Y	N (official patch)	Y	N
ncompress-4.2.4	Y	Y	N (manual patch)	Y	N
polymorph-0.4.0 (bug 1)	Y	Y	N (manual patch)	Y	N
bc-1.06 (bug 3)	Y	Y	N (manual patch)	N/A	N/A	Too complicated. code is generated by flex and bison
man-1.5h1	Y	Y	N (manual patch)	Y	N

benchmark	deterministic	original benchmark	patched version	manual bug signature	patched version	Comment
bc-1.06 (bug 1)		N/A				No bug triggering input
bc-1.06 (bug 2)		N/A				No bug triggering input
polymorph-0.4.0 (bug 2)		N/A				No bug triggering input
squid-2.3		N/A				Complicate to run, don’t know how to start and connect squid server
cvs-1.11.4		N/A				Require running cvs server, no exploit-cvs.c file found

Directory layout

In each benchmark directory, the three folders are added by me:

• ./*slice.txt slice performed on the property violation line (the criteria file is ./src/slicing-criteria.txt)
• ./patch the patch that can fix the bug
• ./helium the folder containing bug signature.
• ./helium/addition the bug signature created manually
• ./helium/slicing the bug signature created based on slice

Slicing experiment

Overview

benchmark	LOC	full slice size	data slice	control slice	comment
gzip	5225	1982	1	55	data slice is actually the criteria itself
ncompress	1436	450	1	63
polymorph	404	20	1	19
man	3036	1992	1206	90

Conclusion from experiment

1. all statements in manual created bug signature are in the slice
2. full slice is much bigger than bug signature
3. understand the reasons that full slice can not simply built (see next sub-section)
4. It is possible to carefully remove statements not in slice to make slice built, and can trigger the bug.
5. The reason for slice to be so big. The reasons can be 1) control slice 2) correct path 3) compute irrelevant results. The first and third reason seem to be primary reasons for these benchmarks.

Reasons for slice not built

1. slice will not contain the syntax meaningless constructs, like
   • parenthesis,
   • else clause,
   • multi-line statements.
   • do while

   These hinder building. The use of AST can help this.

2. slicing may not include the declaration of a variable, results in compile error.
3. typedef is not included in slice
4. Some global variables are not in slice, but is used in many places, including some statements in slice.
5. if branches contains only one statement, which is also not in slice. Cannot simply delete it
6. …