Automated testing of commandline interfaces
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
tests
.gitignore
GNUmakefile
LICENSE
README.rest
run-test.sh
run-tests.sh
update-expected.sh

README.rest

rnt

Automated testing of command line interfaces

Author: Roman Neuhauser
Contact: rneuhauser@suse.cz
Copyright: This document is in the public domain.

Overview

rnt is a tool for automated testing of command line interfaces. It comprises two shell scripts portable enough to run at least on FreeBSD, GNU/Linux and MSYS (Windows).

rnt has notions of test suite and test case.

Test Suite

A test suite is a directory containing zero or more test cases. A test suite may contain other entries beside test cases, they will be ignored.

Test Case

A test case is an immediate test suite subdirectory with a name matching ^[0-9]{3,}(-[a-z0-9]+)+$. It should contain these files:

README
This file is output at the terminal if the test fails.
cmd
A script with the command(s) under test.
exit.expected
Expected exit code.
out.expected
Expected contents of standard output.
err.expected
Expected contents of standard output.

cmd is required, the remaining files are optional (with a caveat about exit.expected, see below). If any of the *.expected files is missing, it's treated as if it was present and empty. A test case may contain other files matching *.expected, these are treated the same as the usual *.expected files listed above (see below). Any number of other files and directories may be present in the test case directory.

Behavior

rnt runs each test case in the given test suite, printing . (a dot) for a successful case, F for a failed one. After all test cases have been run, rnt prints details of failures if any: for each failed test case, its README file is output if it exists, followed by the diffs.

For each test case, rnt chdirs into the test case directory and runs cmd, storing stdout, stderr and the exit code in out.actual, err.actual and exit.actual, respectively.

Then, rnt gathers all *.actual and *.expected files in the test case directory. The list of files considered is not limited to the out.*, err.* and exit.* files listed above: all files with either of those two extensions are included. It's not necessary for both foo.expected and foo.actual to be present; if either side is missing, it's treated as present and empty.

Each pair of foo.expected, foo.actual gets diffed; if any pair has non-empty diff, the test is considered to have failed. Note that since the actual exit code is always a number (never the empty string), a test case without exit.expected will always fail.

*.actual and *.diff files of successful tests are removed, while failed test cases are left as-is for examination.

Installation

Git Subtree

Use git subtree in a project using Git:

git remote add rnt https://github.com/roman-neuhauser/rnt.git
git subtree add -P rnt --squash rnt master

Git Submodule

This method has several disadvantages: submodule trees do not show up in git-archive(1) output, and working copies of such a repository need to be created with git clone --recursive, otherwise git submodule update --init is necessary to clone and check out the submodule.

git submodule add https://github.com/roman-neuhauser/rnt.git rnt
git submodule update --init

Plain Old Copy

Using another versioning control system?

mkdir rnt
curl -s https://github.com/roman-neuhauser/rnt/tarball/master \
| tar -xzf - -C rnt --strip-components=1
svn add rnt
svn ci

Integration

Modify your Makefile to include a check target of the following structure:

check: all
  SHELL=$(SHELL) $(SHELL) <rnt>/run-tests.sh <suite> [args...]
  • <rnt> is the path to rnt.
  • <suite> is the path to your tests.
  • [args...] is any number of arguments (possibly none), these are passed to each test.

Alternatives

rnt is quite different from a number of other shell-oriented testing tools like