asparagus - a framework for human-readable testsuites
-----------------------------------------------------

Copyright (C) 2014 - 2015 Andreas Grapentin
See the end for copying conditions

Please send asparagus bug reports via
<http://github.com/oaken-source/asparagus/issues/>.

Or mail us at <asparagus@grapentin.org>!

 Table of Contents:
  1. Introduction
  2. Integration
  3. Writing Tests
  4. Supported Steps
  5. Writing Step Definitions


1. Introduction
---------------

 Asparagus has been designed to make testing simple. It is supposed to provide
 a means of defining tests via a middleware that allows the definition of test
 cases to be as human-readable as possible, while being as lightweight as
 possible.
 For the development of asparagus, we drew inspiration from the testing frame-
 work for web applications called cucumber, where tests could be defined in
 human-readable paragraphs, which were internally mapped to a selection of
 step definitions, which were then executed in order and verified that the
 assertions made by the tests were met. With asparagus, we wanted to achieve
 the same effect, only for command line applications, in order to bring the
 elegance of the human-readable testing approach down to a new level.
 In order to make integrating asparagus into existing projects as easy as
 possible, we wanted to avoid using powerful general-purpose programming
 languages like python or ruby for the step definitions, and instead settled
 for the TCL/Expect/DejaGnu testing framework and implemented our middleware
 on top of this stack.

 The rest of this doucument describes how to use asparagus in your own
 projects, and how to write yor own tests. For a quick reference, take a look
 at the example test definition in example/example-text.exp, and run it via

  $> DEJAGNU=asparagus.exp VERBOSE=1 runtest --tool example

 where the runtest binary is provided by DejaGnu.

 The information in this file, as well as some additional documentation, can
 be found on <http://asparagus.grapentin.org>.

 If you have found a bug, please do not hesitate to report it at asparagus'
 bug tracker found on <http://github.com/asparagus/issues>.

 If you have any suggestions, on how asparagus can be improved, or want to
 submit a bug directly, or just want to chat with us, send an email to
 <asparagus@grapentin.org>.

 If you want to support asparagus, or its developers, then feel free to do so.
 We can always use a hand maintaining and improving the project, or its
 representaion.


2. Integration
--------------

 To run a single test file, run the following command with dejagnu installed:

  $> DEJAGNU=<path-to-asparagus.exp> runtest --tool <tool> --srcdir <srcdir>

 Use this schema to implement a `make check` step for your makefiles.

 Integration with autoconf/automake managed projects is straightforward as
 well. Just copy asparagus.exp and the lib/ and steps/ directory to your tests
 directory and add the following block to Makefile.am in your tests directory:

  EXTRA_DIST = asparagus.exp lib/feedback.tcl lib/globals.tcl lib/string.tcl \
               steps/dispatcher.tcl steps/default.tcl \
	       [ ... your test files here ...]

  AUTOMAKE_OPTIONS = dejagnu

  DEJAGNU  = $(top_srcdir)<path to asparagus.exp>
  DEJATOOL = <list of your test groups>

  export DEJAGNU

 Group your test files in directories named <tool>.tests/ and add <tool> to
 DEJATOOL in Makefile.am, if you want them executed. In the <tool>.tests/
 directory, create your test files <group>.exp which contain your tests.

 The tests should be executed when you run `make check` or `make distcheck`.


3. Writing Tests
----------------

 Test definitions are separated into steps, which can be separated into three
 groups.

 `Given' steps are usually the first step of a test, and describe the
 prerequesites for the following steps, for example:

  Given an executable "ls"

 `When' steps describe events that usually describe input to the tested
 entity. An example that is commonly used in asparagus is:

  When I run with parameters "-a"

 `Then' steps describe assertions, that result from the preceding When steps.
 A common example is:

  Then It should return 0

 There is one more step family, which is the `And' steps. And steps mimic the
 family preceding the current step, which allows for more pleasing semantic
 test definitions.

 Chaining these steps together creates a test, where each step may either fail
 or succeed. An example for a complete test would be:

  Given an executable "ls"
    When I run with parameters "-a"
    Then I should see ".."
    And it should return 0

 The step keywords Given, When, Then and And may be written all lowercase, or
 with an uppercase first letter.


4. Supported Steps
------------------

 Asparagus comes with a number of different steps which are supported by
 default, which are outlined below. For code reference, see the file
 steps/default.tcl.

  Given an executable ?exe?
    This step sets the currently used executable, which is used by the
    `When I run' family of steps.

  Given the c code ?source?
    Specifiy a source code snippet to be consumed by the `When I compile`
    family of steps.

  When I run with parameters ?parameters?
    This step tries to spawn a process from the currently set executable
    while passing the given command line arguments.

  When I run
    This step tries to spawn a process from the currently set executable.

  When I send ?string?
    This step sends the given input to the stdin of the spawned process.

  When I compile with cflags ?cflags?
    This step attempts to compile the source code specified by a `Given the
    c code` step into a binary using the given cflags, and sets the current
    executable to this binary.

  When I compile
    This step attempts to compile the source code specified by a `Given the
    c code` step into a binary and sets the current executable to this
    binary.

  Then I should see ?string?
    This step asserts that the given string is part of the output of the
    spawned process. All output until the given string is found is discarded.

  Then I should not see ?string?
    This step asserts that the given string is not part of the output of the
    spawned process. All output in the expect buffer is discarded.

  Then it should return ?code?
    This step asserts that the spawned process terminates with the given
    return code.

  Then it should not return ?code?
    This step asserts that the spawned process terminates with another than
    the given return code.

  Then write the output to log
    This step captures all output produced by the spawned process and sends
    it to the log file until the process terminates.


5. Writing Step Definitions
---------------------------

 Extending the functionality of asparagus with custom step definitions
 requires some expertise with expect and tcl. For some examples, see
 steps/default.tcl.

 What you need to do, is create TCL procedures for the behaviour of your steps
 and call `asparagus_register_step <your step proc> "<your step name>"` from
 a file sourced by asparagus.exp.
 See asparagus.exp for an example.


----------------------------------------------------------------------
Copyright information:

Copyright (C) 2014 - 2015 Andreas Grapentin

   Permission is granted to anyone to make or distribute verbatim copies
   of this document as received, in any medium, provided that the
   copyright notice and this permission notice are preserved,
   thus giving the recipient permission to redistribute in turn.

   Permission is granted to distribute modified versions
   of this document, or of portions of it,
   under the above conditions, provided also that they
   carry prominent notices stating who last changed them.