Scripted Testing

Introduction

To set the scene, the Ada software under test (SUT) forms part of a system. Generally, the system will be constructed using a layered approach, and the other parts of the system that interact with the SUT are higher-level (which can call interface subprograms presented by the SUT) or lower-level (whose interfaces can be called by the SUT, or which can provide callbacks to the SUT).

This package is intended for testing at a level between unit testing (using, for example, AUnit) and integration testing (using real hardware). Unit testing tends to be fragile, requiring a lot of rework for even minor functional changes in the requirements.

The scripting language supported is Tcl, and specifically the Ada binding Tash. The reason for choosing Tcl rather than Python or Lua is that Tcl's interface is entirely string-based; this is important, considering the need to specify values of enumerated types.

Genesis

The project from which this project is derived is the author's ColdFrame; see that project's stubbing facilities.

ColdFrame is a code generation framework, and has its own approach to stubbing (mocking) subprograms. This package has a different, but related, approach.

Facilities

The package provides facilities to write new commands in Tcl to

• call the interface subprograms presented by the SUT,

• set up values to be returned by calls the SUT makes to lower-level system components,

• call callbacks provided by lower-level system components,

• delay for appropriate periods, and

• check that the proper calls have been made to lower-level system components, with the required values.

It's assumed that the interface subprograms of the lower-level subsystems are stubbed so that:

• in and in out parameters can be recorded for later checking,

• out (and in out) parameter values and function return values can be provided to be returned to the SUT,

• exceptions can be raised when required,

• the number of calls to the subprogram can be checked.

The components of the package are Commands, Actions, and an Action Queue.

Commands

A Command implements a Tcl command.

It creates an Action to be executed at run time, and posts it on the Action Queue.

Some commands are provided by this package. Other commands are to be provided to support the specific application to be tested: typically, call_procedure param1 param2 ... (where the parameters are those required by procedure) and check_subprogram parameter value (to check the value passed to procedure in parameter on the last call).

Commands have to be Registered with this package, because once Tcl has been started (using Start, which doesn't return) no more Commands can be added. Registration would normally be done during elaboration of the package in which the Command is defined (see test/test-first.adb).

Actions

An Action carries the data required to enact the command at run time.

When it is executed, it performs the required action.

• If all is well, it completes normally.

• If some condition fails, it raises an exception (Execution_Failure) with a message stating the problem.

• Any other exceptions are propagated.

Action Queue

When the Action Queue is started (using the go command), it repeatedly picks the next Action and executes it, until either the end of the queue is reached (which implies that the script has succeeded) or an exception is propagated (which implies that the script has failed).

Provided commands

The commands provided by this package are

echo "string"

outputs string to the terminal at run time. Useful to report the script's progress.

wait duration

delays for duration.

mark name

notes the time at which the command was executed.

wait_from_mark name duration

delays until duration after the named mark. It is an error if the indicated time has already passed. The mark can be re-used.

go

start executing the script.

Building

This is an Alire crate; to build, say alr build.

Stub generation

See a worked example in the demo/ directory.