pip install pnu-b2bt
b2bt - back-to-back testing
b2bt [-f|--force] [-k|--keep] [-n|--newpath PATH] [-o|--origpath PATH] [-q|--quiet] [-s|--skip] [-t|--timeout VALUE] [-y|--autoconfirm] [-N|--nocolors] [--debug] [--help|-?] [--version] [--] filename [...]
The b2bt utility aims to automate:
- back-to-back testing (testing a command-line utility against another implementation)
- non regression testing (testing a command-line utility against a previous version)
It processes one or more XML files describing the test suite for a command-line utility, with its different test cases, executing the 2 commands versions and reporting potential differences (on return codes, standard output, standard error output, plus custom post processing output in order to check other kind of impacts, on file systems, execution time, etc.) for each test case.
In a test suite it is possible to specify:
- optional shell commands to execute before the test, typically to setup the test environment in an automatically generated test sub directory
- optional standard input to be injected in the command tested
- mandatory one line shell command-line for the command to be tested
- optional shell commands to execute after the test, typically to analyze the tested command impact on the test environment. The output of these "post" commands is captured before automatic removal of the test sub directory
Each test case can have a name and a potential timeout (in seconds or fraction of seconds).
By defaut, this utility:
- executes the original command, unless you use the -s option to skip it (useful when you are testing the new command on another system);
- executes the original command from the PATH, unless you use the -o option to specify another location;
- uses a 120.0 s timeout for commands execution, unless you use the -t option to modify it;
- discards the test results after each comparison, unless you use the -k option to keep them;
- details the differences between the 2 commands runs, unless you use the -q option to limit it;
- does not execute again tests when you have kept previous results, unless you use the -f option to overwrite them;
- asks for confirmation before each test case execution, unless you use the -y option to auto confirm them;
- uses ANSI colours in the output, unless you use the -N option to disable it;
- does not execute the new command if it has the same MD5 digest than the original one.
The new command location has to be specified with the -n option if you want a comparison to occur.
| Options | Use |
|---|---|
| -f|--force | Overwrite results directories |
| -k|--keep | Keep results directories after running |
| -n|--newpath PATH | Path of the new command |
| -o|--origpath PATH | Path of the original command if not in $PATH |
| -q|--quiet | Don't detail run differences |
| -s|--skip | Skip original command processing |
| -t|--timeout VALUE | Set default timeout (120.0 s) to a new value, with a 30.0 s minimum |
| -y|--autoconfirm | Don't ask for confirmation before test case execution |
| -N|--nocolors | Don't use colors in output |
| --debug | Enable debug mode |
| --help|-? | Print usage and a short help message and exit |
| --version | Print version and exit |
| -- | Options processing terminator |
The B2BT_OPTIONS environment variable can be set with a combination of the following command-line options letters:
- f in order to overwrite results directories if they exist
- k in order to keep results directories after running
- q in order to disable details in run differences
- s in order to skip original command processing
- y in order to answer yes to all confirmation requests
- N in order to disable colors in output
The B2BT_DEBUG environment variable can also be set to any value to enable debug mode.
These 2 environment variables are superseded by command line options.
The format of the XML files processed is described in the b2bt(5) manual page.
The b2bt utility exits 0 on success, and >0 if an error occurs.
The b2bt utility is not a standard UNIX/POSIX command.
It tries to follow the PEP 8 style guide for Python code.
This utility was made for the PNU project in order to test the rewritten commands against the installed ones.
This utility is available under the 3-clause BSD license.
If you are comparing commands execution across operating systems, for example between Unix-like and Windows systems, the output might be different due to the path separators ("/" versus "\").
Comparing timeout interrupted commands output is hazardous...
This utility processes XML files describing commands to be shell executed on your system. If you are not the author of a test suite or are using an untrusted source, this can be dangerous!
To mitigate the risks, the utility will:
- Warn you if you are using a privileged account and advise you not to do so.
- Show you every command to be executed and ask for prior confirmation.
Visual inspection of the XML files to process is recommended, and useful anyway if you are rewriting an existing command.
The program is using MD5 file digests but not for security purposes. It is also calling the what(1) and ident(1) commands from their PATH location, if they are available, assuming system directories precede user directories in the PATH.