Skip to content

Home

Jump to bottom
Emil Karlén edited this page Feb 7, 2021 · 46 revisions

Exactly tests a command line program by executing it in a temporary sandbox directory and inspecting its result.

Example

A test case is as a plain text file

[setup]

stdin = -contents-of an-address-book.txt

[act]

addressbook --get-email-of --name 'Test Testingson'

[assert]

exit-code == 0

stdout equals <<EOF
expected@email.org
EOF

If the file 'addressbook.case' contains this test case, then Exactly can execute it:

> exactly addressbook.case
PASS

"PASS" means that the two assertions were satisfied.

This test assumes that

  • the system under test - the addressbook program - is is found in the same directory as the test case file
  • the file "an-address-book.txt" (that is referenced from the test case) is found in the same directory as the test case file

The home instruction can be used to change where Exactly looks for files referenced from the test case.

Using shell commands

Shell commands can be used both as the sut (system under test), and as assertions.

$ echo ${PATH}

[assert]

$ < ../result/stdout tr ':' '\n' | grep '^/usr/local/bin$'

About

A test is organized into "phases", where each phase is a sequence of instructions, that behave a bit like a script.

The phases are:

  • conf
  • setup
  • act
  • before-assert
  • assert
  • cleanup

The test fixture and assertions are written using "instructions" that mimics the syntax of unix shell commands, with options and arguments.

Exactly supports individual test cases and test suites.

Exactly has a built in help system.

Exactly has a Reference manual.

Exactly is a Python 3 program, with no dependencies on external libraries.

Exactly has been tested on Linux and OS X.

Most, or all, features work on Windows, but Exactly's own test suite has not been completely ported to Windows.

Some features

[act] is the default phase

[act] is not needed to indicate what is being checked, since the "act" phase is the default "phase".

The following is a valid test case, and if run by Exactly, it won't remove anything (since it is executed inside a temporary sandbox directory):

$ rm -rf *

Print output from the tested program

If --act is used, the output of the tested program (the "act" phase) will become the output of Exactly - stdout, stderr and exit code.

$ echo Hello World

[assert]

stdout any line : contents matches Hello

Then:

> exactly --act hello-world.case
Hello World

The test case is executed in the sandbox, as usual.

Keeping the sandbox directory for later inspection

If --keep is used, the sandbox directory will not be deleted, and its name will be printed.

This can be used to inspect the outcome of the "setup" phase, e.g.

[setup]

file my-file.txt

[act]

my-prog my-file

[assert]

exit-code == 0

The act directory is the current directory when the test runs. The file instruction has put the file my-file.txt there:

> exactly --keep my-test.case
/tmp/exactly-1strbro1

> find /tmp/exactly-1strbro1
/tmp/exactly-1strbro1
/tmp/exactly-1strbro1/tmp
/tmp/exactly-1strbro1/tmp/user
/tmp/exactly-1strbro1/tmp/internal
/tmp/exactly-1strbro1/testcase
/tmp/exactly-1strbro1/act
/tmp/exactly-1strbro1/act/my-file.txt
/tmp/exactly-1strbro1/result
/tmp/exactly-1strbro1/result/exitcode
/tmp/exactly-1strbro1/result/stderr
/tmp/exactly-1strbro1/result/stdout
/tmp/exactly-1strbro1/log

A complex example

The following test case displays a potpurri of functionality. (Beware that this test case does not make sense! - it just displays some of Exactly's functionality.)

[conf]

# This test case must be SKIPed, since it doesn't make sense.
# It just displays some instructions and how they might could be used.

status = SKIP


act-home = ../..

# "act home" is one of the directories in the home directory structure
# it tells where the program run by [act] is located.

home = ..

# "case home" is one of the directories in the home directory structure
# it is the default location for predefined resources referenced from a test


[setup]


copy this-is-an-existing-file-in-same-dir-as-test-case.txt

dir first/second/third

file in/a/dir/file-name.txt =
<<EOF
contents of the file
EOF

dir root-dir-for-act-phase

cd root-dir-for-act-phase
# This will be current directory for the [act] phase.

def string THIS_MUST_BE_PART_OF_STDIN = 'some important data'

stdin = <<EOF
this will be stdin for the program in the "act" phase
@[THIS_MUST_BE_PART_OF_STDIN]@
EOF
# (It is also possible to have stdin redirected to an existing file.)

env MY_VAR = 'value of my environment variable'

env unset VARIABLE_THAT_SHOULD_NOT_BE_SET

run my-prog--located-in-same-dir-as-test-case--that-does-some-more-setup 'with an argument'


def string A_SUT_ARGUMENT = 'an argument'

def list   MORE_SUT_ARGUMENT = first 'second argument'

def list   ALL_SUT_ARGUMENTS = @[A_SUT_ARGUMENT]@ @[MORE_SUT_ARGUMENT]@


[act]


the-system-under-test @[ALL_SUT_ARGUMENTS]@ 'and finally one more argument'


[before-assert]


cd ..
# Moves back to the original current directory.

$ sort root-dir-for-act-phase/output-from-sut.txt > sorted.txt


[assert]


exit-code != 0

stdout equals <<EOF
This is the expected output from the-system-under-test
EOF

stderr is-empty

contents a-file.txt : is-empty

contents a-second-file.txt : ! is-empty

contents another-file.txt :
         -transformed-by REPLACE_TEST_CASE_DIRS
         equals
         -contents-of expected-content.txt

exists actual-file : type file

dir-contents sql-dir : -selection ( ! name *.sql ) is-empty

cd this-dir-is-where-we-should-be-for-the-following-assertions

run my-prog--located-in-same-dir-as-test-case--that-does-some-assertions


[cleanup]


$ umount my-test-mount-point

run my-prog-that-removes-database 'my test database'

More examples

See the examples/ directory of the source repository for examples.

Author

Emil Karlén

emil@member.fsf.org

Clone this wiki locally