Differential Debugger Testing
Differential testing is an automatic testing technique where generated inputs are given to two different programs whose behavior is then compared. We apply this idea to debuggers, where the inputs are a program-to-debug and generated debugging actions, e.g., "set breakpoint in line X" or "step into" and the compared debugger behavior is, for example, where the debugger paused or the names and values of local variables.
For more details on our approach, see our paper (to appear):
Feedback-Directed Differential Testing of Interactive Debuggers Daniel Lehmann and Michael Pradel The 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018)
Directory Structure and Overview
This repository is structured into several subdirectories, some of which are only created upon building the tool or during execution (i.e., when the automated tests on Firefox and Chromium are performed).
./: At the top level you find our implementation as an NPM project (see the
package.jsonfile and the
yarn.lockfile which captures the dependencies' exact version numbers), this readme, installation instructions (
INSTALL.md) and a script to install all prerequisites (
install.sh, tested for Ubuntu 18.04 LTS), and the artifact status for which we apply for the FSE artifact evaluation (
browsers/: This directory contains a
download-and-extract.shscript you can use this to get the exact versions of Firefox and Chromium we used for testing. They will be downloaded and extracted into this directory and not installed system wide (i.e., your installations of Chromium and Firefox are unaffected).
It also contains a pre-setup Firefox profile
firefox-profile/prefs.jswhich (among other things) enables the remote debugging protocol. The browsers are run with temporary profiles, i.e., should not affect your settings of the locally installed browsers.
bugreports/: In this directory we collected all bug reports we filed in the bug trackers of Firefox and Chromium. In particular, the
README.mdlists all bugs with a short description/intuition when/how it manifests, a link to the actual issue entry, and its status (one of Reported/Confirmed/Fixed).
For each of the listed bugs, there is also a subdirectory, e.g.,
Note that, as described in the paper, between automatic finding of differences between the debuggers (i.e., the differential testing implemented by our tool) and filing of the bug reports some manual effort was involved; namely condensing the program-to-debug into a small script, ideally with a short example where the bug appears vs. a short example where it doesn't but that is very similar. This is one aspect where we would like to further improve/automate our approach.
build/eval/run-tests.jswith Node.js (also see the installation instructions).
sunspider-determ/: The largest group (26 programs), taken from the SunSpider benchmarks, version 1.0.2.
Since some SunSpider benchmarks use non-deterministic operations that would show up as differences when comparing the debugger outputs (e.g.,
string-validate-input.js), we fixed these sources of non-determinism. The changes are minor, namely replacing
Math.random()with a deterministic function and constructing a fixed
9.js, where a function appears with multiple parameters of the same name.
results/: This directory is generated upon execution of our tool and contains the output of the testing. It contains one subdirectory per set of testing parameters, e.g.,
results/breakpointsPerLine0.1-breakpointRemoveProb0.2-maxActions20/with the default setting of 0.1 breakpoints per line, a probability of 0.2 of removing a just-set breakpoint, and a maximum number of 20 actions (i.e., continue or steps).
Inside, there are again directories for each program-to-debug, e.g.,
/programs-to-debug/sunspider-determ/3d-cube.js. These directories finally contain the traces in the format described in the paper, e.g., the recorded trace of the first generated test is
seed0.logand contains (excerpt)
... set breakpoint @ line 309 Chrome: set @ line 311, id: 31:308:0 Firefox: set @ line 311, id: server1.conn0.child1/breakpoint39 remove breakpoints from previous step Chrome: removed @ line 311, id: 31:308:0 Firefox: removed @ line 311, id: server1.conn0.child1/breakpoint39 ... set breakpoint @ line 275 Chrome: set @ line 279, id: 31:274:0 Firefox: set @ line 275, id: server1.conn0.child1/breakpoint44 breakpoint difference
which indicates that a difference in the location of a set breakpoint was found between Firefox and Chromium.
An overview of all runs performed with the given parameter set, can be found in
runs.csv. This also contains a column for the type of difference that was found (cf. Figure 7 in the paper).
Note again that some manual efforts were involved in filing bug reports out of these found warnings. The basic workflow for this part is opening a trace, looking for the affected lines in the program-to-debug and the executed actions, reproducing the difference in the debugger GUIs of Firefox and Chromium, and then manually minimizing the program-to-debug to find the root cause. See
bugreports/for the bug reports we filed in the official issue trackers.
src/: Finally, this contains our implementation in TypeScript. Its subdirectories are:
chrome-remote-interface), but for Firefox, we had to implement the debugging protocol ourselves (see
firefox-protocol.ts). Since the official documentation is sometimes incomplete or out of date (https://wiki.mozilla.org/Remote_Debugging_Protocol), this might be a useful resource for others trying to interface with the Firefox debugger through its RDP.
interface.tscontains our common abstraction over the debugger protocols of Firefox and Chromium. On top of this interface, we issued the debugger actions mentioned in the paper.
await), and generating the debugger traces.
eval/the actual code for generating the debugging actions, loading the
programs-to-debug, and comparing the debugger behavior between Firefox and Chromium. In particular, run
run-tests.jswith Node.js to start the testing (and modify the set of parameters
paramsValuesif you want to run with a different number of breakpoints or actions). Also, by editing
testcases.tsyou can add/change the programs-to-debug for which actions are generated.
Please check INSTALL.md for instructions.
License, Usage, and Modification
This project is licensed under the permissive open-source MIT license, see LICENSE.txt for the full license. If you
have any questions on how our tool works, on how to get it to run, or how to modify it for your own purposes, please
write us a mail at
firstname.lastname@example.org. We are happy to help!