FauxPy (pronounced: "foh pie") is an automated fault localization tool for Python programs. This repository includes FauxPy's source code, and instructions to install and use the tool.
FauxPy supports seven classic fault-localization techniques in four families:
-
SBFL (spectrum-based) techniques Tarantula, Ochiai, and DStar.
-
MBFL (mutation-based) techniques Metallaxis and Muse.
-
PS (predicate switching) fault localization.
-
ST (stack-trace) fault localization.
It supports fault localization at the level of statements (statement-level granularity) and at the level of functions (function-level granularity).
FauxPy is based on dynamic analysis, and can use tests written in the format of Pytest, Unittest, and Hypothesis.
FauxPy is on PyPI,
so you can install it using pip
:
pip install fauxpy
To install the latest (unreleased) version, use the following command:
pip install git+https://github.com/atom-sw/fauxpy
We mainly tested FauxPy with Python 3.6, 3.7, and 3.8, but it should also work on later Python versions.
Here is a short demo video of FauxPy in action.
Directory examples/triangle_area includes a tutorial example of using FauxPy. Follow the instructions in the README.md
.
FauxPy is implemented as a Pytest plugin, thus using FauxPy boils down to passing some custom options to Pytest.
To run FauxPy, you must first cd
to the project's directory $PROJECT
:
cd $PROJECT
Then, the basic command to run FauxPy is the following, where
$SOURCE
is the relative path to a Python package or module
inside $PROJECT
:
python -m pytest --src $SOURCE
This performs statement-level spectrum-based fault localization on the
Python project in directory $PROJECT
, using any Pytest tests in
there.
The output is a list of program entities with their suspiciousness
score, sorted from most to least suspicious. The output is printed on
screen, and also saved in a directory FauxPyReport_...
created in
$PROJECT
's parent directory.
This is the complete list of command-line arguments to control FauxPy.
python -m pytest \
$TESTS \
--src $SRC \
--family $FAMILY \
--exclude "[$EXCLUDE1, $EXCLUDE2, ...]" \
--granularity $GRANULARITY \
--top-n $N \
--failing-list "[$FAIL1, $FAIL2, ...]" \
--failing-file $FAIL
Option --src $SRC
runs FauxPy on the project considering only
the program entities under relative path
$SRC
. Precisely, $SRC
can point to a whole project, or an
individual package (subdirectory) or modules (source-code file) within
it.
In particular, option --src .
runs FauxPy on the project in the
current directory, considering the program entites in all
the Python modules and packages existing within project.
Option --src
is the only mandatory argument to run FauxPy.
Option --family $FAMILY
runs FauxPy using the $FAMILY
fault
localization family of techniques.
FauxPy currently supports families: sbfl
(the default), mbfl
, ps
, and st
.
Option --granularity $GRANULARITY
runs FauxPy using $GRANULARITY
as program entities to localize. FauxPy currently supports
granularities: statement
(the default), and function
.
With statement-level granularity, FauxPy outputs a list of program locations (i.e., line numbers) that may be responsible for the fault. With function-level granularity, FauxPy outputs a list of functions that may be responsible for the fault.
Option --exclude "[$EXCLUDE1, $EXCLUDE2, ...]"
ignores entities in $EXCLUDE1
,
$EXCLUDE2
, and so on when performing fault localization.
Each element of the comma-separated list must be a path relative to the analyzed project's root directory of a directory (package) or Python source file (module).
For instance, the following command runs fault localization on the
project in the current directory, skipping directories env
and
tests
, and module utilities
:
python -m pytest --src . --exclude "[env, tests, utilities.py]"
Option --failing-list "[$FAIL1, $FAIL2, ...]
only uses tests
$FAIL1
, $FAIL2
, and so on as failing tests when performing fault
localization.
Each element of the comma-separated list must be
the fully-qualified name of a test function in the analyzed project,
using the Pytest format <FILE_PATH>::<CLASS_NAME>::<FUNCTION_NAME>
,
where the <CLASS_NAME>::
can be omitted if the test function is
top-level.
For instance, the following command runs fault localization on the
project in the current directory, using only test function
test_read_file
in class Test_IO
as failing test:
python -m pytest --src . \
--failing-list "[`test/test_common/test_file.py::Test_IO::test_read_file`]"
Selecting specific failing tests is especially useful when there are multiple, different faults, triggered by different tests. Fault localization techniques are usually designed to work under the assumption that they analyze each fault in isolation. If the analyzed project includes multiple faults, it is advisable to select a subset of the failing tests that trigger a single fault, so that fault localization can perform more accurately.
Option --failing-file $FAIL
is the same as option --failing-list
.
But instead of taking a list of failing tests,
it takes the path of a file relative to the analyzed project's root directory.
In file $FAIL
, every failing test must be in a separate line.
Option --top-n $N
only reports up to $N
suspicious program
entities (statements or functions). $N
must be a positive integer,
or -1
(the default: no limit).
Optional positional argument $TESTS
, specified just after pytest
,
runs FauxPy using the tests found under path $TESTS
.
If this argument is missing, FauxPy will use any tests found in the
analyzed project.
$TESTS
must be a path relative to the analyzed project's root
directory of a directory (package), a Python source file (module), or
the fully-qualified name of a test function in the analyzed project,
using the Pytest format <FILE_PATH>::<CLASS_NAME>::<FUNCTION_NAME>
,
where the <CLASS_NAME>::
can be omitted if the test function is
top-level.
The positional argument can be repeated to select tests at different locations.
For instance, the following command runs FauxPy using only tests
in package tests/package_x
, module tests/test_y.py
,
and test function tests/test_z.py::test_function_t
.
python -m pytest tests/package_x \
tests/test_y.py \
tests/test_z.py::test_function_t \
--src $SRC
Stack-trace and predicate switching fault localization only need to run failing tests. If a project has many tests, but only a few are failing, ST and PS fault localization will run more quickly if we pass the failing tests' location using this feature. If we don't, FauxPy will still have to run all tests, just to discover which ones are failing and can be used for ST or PS fault localization.
The current version of FauxPy has a couple of known limitations:
-
The implementation of SBFL and MBFL uses Coverage.py through its API to collect execution traces. Thus, FauxPy inherits any limitation of Coverage.py.
-
The implementation of ST does not work properly if Pytest's option
--tb=native
is enabled (as a command-line argument or in apytest.ini
file). -
FauxPy is incompatible with Pytest plugin
pytest-sugar
. If you want to use FauxPy, removepytest-sugar
from your Python environment.
The paper An Empirical Study of Fault Localization in Python Programs describes an empirical study where we applied FauxPy to 135 bugs in 13 real-world Python programs (from the BugsInPy curated collection).
You can cite this empirical work as follows:
Mohammad Rezaalipour, Carlo A. Furia: An Empirical Study of Fault Localization in Python Programs. CoRR abs/2305.19834 (2023)
@misc{PythonFL-FauxPy,
title={An Empirical Study of Fault Localization in {P}ython Programs},
author={Mohammad Rezaalipour and Carlo A. Furia},
year={2023},
eprint={2305.19834},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2305.19834}
}
This repository is a public mirror of (part of) FauxPy's private development repository. There are two public mirrors, whose content is identical: