-
Notifications
You must be signed in to change notification settings - Fork 7
Shell Test API
Note
|
See also Test Writing Guidelines, C Test API. |
LTP supports testcases to be written in a portable shell too.
There is a shell library modeled closely to the C interface at testcases/lib/tst_test.sh.
Warning
|
All identifiers starting with TST_ or tst_ are reserved for the test library. |
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# This is a basic test for true shell builtin
TST_TESTFUNC=do_test
. tst_test.sh
do_test()
{
true
ret=$?
if [ $ret -eq 0 ]; then
tst_res TPASS "true returned 0"
else
tst_res TFAIL "true returned $ret"
fi
}
tst_run
Tip
|
To execute this test the tst_test.sh library must be in $PATH. If you are executing the test from a git checkout you can run it as PATH="$PATH:../../lib" ./foo01.sh |
The shell library expects test setup, cleanup and the test function executing the test in the $TST_SETUP, $TST_CLEANUP and $TST_TESTFUNC variables.
Both $TST_SETUP and $TST_CLEANUP are optional.
The $TST_TESTFUNC may be called several times if more than one test iteration was requested by passing right command line options to the test.
The $TST_CLEANUP may be called even in the middle of the setup and must be able to clean up correctly even in this situation. The easiest solution for this is to keep track of what was initialized and act accordingly in the cleanup.
Warning
|
Similar to the C library, calling tst_brk in the $TST_CLEANUP does not exit the test and TBROK is converted to TWARN. |
Notice also the tst_run shell API function called at the end of the test that actually starts the test.
Warning
|
cleanup function is called only after tst_run has been started. Calling tst_brk in shell libraries, e.g. tst_test.sh or tst_net.sh does not trigger calling it. |
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in separate functions
TST_TESTFUNC=test
TST_CNT=2
. tst_test.sh
test1()
{
tst_res TPASS "Test $1 passed"
}
test2()
{
tst_res TPASS "Test $1 passed"
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
If $TST_CNT is set, the test library looks if there are functions named ${TST_TESTFUNC}1, …, ${TST_TESTFUNC}${TST_CNT} and if these are found they are executed one by one. The test number is passed to it in the $1.
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function
TST_TESTFUNC=do_test
TST_CNT=2
. tst_test.sh
do_test()
{
case $1 in
1) tst_res TPASS "Test $1 passed";;
2) tst_res TPASS "Test $1 passed";;
esac
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
Otherwise, if $TST_CNT is set but there is no ${TST_TESTFUNC}1, etc., the $TST_TESTFUNC is executed $TST_CNT times and the test number is passed to it in the $1.
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function, using $TST_TEST_DATA and
# $TST_TEST_DATA_IFS
TST_TESTFUNC=do_test
TST_TEST_DATA="foo:bar:d dd"
TST_TEST_DATA_IFS=":"
. tst_test.sh
do_test()
{
tst_res TPASS "Test $1 passed with data '$2'"
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 1 passed with data 'bar'
# foo 3 TPASS: Test 1 passed with data 'd dd'
It’s possible to pass data for function with $TST_TEST_DATA. Optional $TST_TEST_DATA_IFS is used for splitting, default value is space.
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
TST_TESTFUNC=do_test
TST_CNT=2
TST_TEST_DATA="foo bar"
. tst_test.sh
do_test()
{
case $1 in
1) tst_res TPASS "Test $1 passed with data '$2'";;
2) tst_res TPASS "Test $1 passed with data '$2'";;
esac
}
tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 2 passed with data 'foo'
# foo 3 TPASS: Test 1 passed with data 'bar'
# foo 4 TPASS: Test 2 passed with data 'bar'
$TST_TEST_DATA can be used with $TST_CNT. If $TST_TEST_DATA_IFS not specified, space as default value is used. Of course, it’s possible to use separate functions.
Similarily to the C library various checks and preparations can be requested simply by setting right $TST_NEEDS_FOO.
Variable name | Action done |
---|---|
TST_NEEDS_ROOT |
Exit the test with TCONF unless executed under root. |
Alternatively the tst_require_root command can be used. |
|
TST_NEEDS_TMPDIR |
Create test temporary directory and cd into it. |
TST_NEEDS_DEVICE |
Prepare test temporary device, the path to testing device is stored in $TST_DEVICE variable. The option implies TST_NEEDS_TMPDIR. |
TST_NEEDS_CMDS |
String with command names that has to be present for the test (see below). |
TST_NEEDS_MODULE |
Test module name needed for the test (see below). |
TST_NEEDS_DRIVERS |
Checks kernel drivers support for the test. |
TST_TIMEOUT |
Maximum timeout set for the test in sec. Must be int >= 1,
or -1 (special value to disable timeout), default is 300.
Variable is meant be set in tests, not by user.
It’s an equivalent of |
Function name | Action done |
---|---|
tst_set_timeout(timeout) |
Maximum timeout set for the test in sec. See TST_TIMEOUT variable. |
Note
|
Network tests (see testcases/network/README.md) use additional variables and functions in tst_net.sh. |
#!/bin/sh
...
TST_NEEDS_CMDS="modinfo modprobe"
. tst_test.sh
...
Setting $TST_NEEDS_CMDS to a string listing required commands will check for existence each of them and exits the test with TCONF on first missing.
Alternatively the tst_require_cmds() function can be used to do the same on runtime, since sometimes we need to the check at runtime too.
tst_check_cmds() can be used for requirements just for a particular test as it doesn’t exit (it issues tst_res TCONF). Expected usage is:
#!/bin/sh
TST_TESTFUNC=do_test
. tst_test.sh
do_test()
{
tst_check_cmds cmd || return
cmd --foo
...
}
tst_run
The LTP build system can build kernel modules as well, setting $TST_NEEDS_MODULE to module name will cause the library to look for the module in a few possible paths.
If module was found the path to it will be stored into $TST_MODPATH variable, if module wasn’t found the test will exit with TCONF.
Alternatively the tst_require_module() function can be used to do the same at runtime.
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Optional test command line parameters
TST_OPTS="af:"
TST_USAGE=usage
TST_PARSE_ARGS=parse_args
TST_TESTFUNC=do_test
. tst_test.sh
ALTERNATIVE=0
MODE="foo"
usage()
{
cat << EOF
usage: $0 [-a] [-f <foo|bar>]
OPTIONS
-a Enable support for alternative foo
-f Specify foo or bar mode
EOF
}
parse_args()
{
case $1 in
a) ALTERNATIVE=1;;
f) MODE="$2";;
esac
}
do_test()
{
...
}
tst_run
The getopts string for optional parameters is passed in the $TST_OPTS variable. There are a few default parameters that cannot be used by a test, these can be listed with passing help -h option to any test.
The function that prints the usage is passed in $TST_USAGE, the help for the options implemented in the library is appended when usage is printed.
Lastly the function $PARSE_ARGS is called with the option name in the $1 and, if option has argument, its value in the $2.
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Optional test positional parameters
TST_POS_ARGS=3
TST_USAGE=usage
TST_TESTFUNC=do_test
. tst_test.sh
usage()
{
cat << EOF
usage: $0 [min] [max] [size]
EOF
}
min="$1"
max="$2"
size="$3"
do_test()
{
...
}
tst_run
You can also request a number of positional parameters by setting the $TST_POS_ARGS variable. If you do, these will be available as they were passed directly to the script in $1, $2, …, $n.
You may need to retrieve configuration values such as PAGESIZE, there is getconf but as some system may not have it, you are advised to use tst_getconf instead. Note that it implements subset of getconf system variables used by the testcases only.
# retrieve PAGESIZE
pagesize=`tst_getconf PAGESIZE`
Albeit there is a sleep command available basically everywhere not all implementations can support sleeping for less than one second. And most of the time sleeping for a second is too much. Therefore LTP includes tst_sleep that can sleep for defined amount of seconds, milliseconds or microseconds.
# sleep for 100 milliseconds
tst_sleep 100ms
Sometimes an LTP test needs to retry a function call multiple times because the system is not ready to process it successfully on the first try. The LTP library has useful tools to handle the call retry automatically. TST_RETRY_FUNC() will keep retrying for up to 1 second. If you want a custom time limit use TST_RETRY_FN_EXP_BACKOFF(). Both methods return the value returned by the last FUNC call.
The delay between retries starts at 1 microsecond and doubles after each call. The retry loop ends when the function call succeeds or when the next delay exceeds the specified time (1 second for TST_RETRY_FUNC()). The maximum delay is multiplied by TST_TIMEOUT_MUL. The total cumulative delay may be up to twice as long as the adjusted maximum delay.
The C version of TST_RETRY_FUNC() is a macro which takes two arguments:
-
FUNC is the complete function call with arguments which should be retried multiple times.
-
SUCCESS_CHECK is a macro or function which will validate FUNC return value. FUNC call was successful if SUCCESS_CHECK(ret) evaluates to non-zero.
Both retry methods clear errno before every FUNC call so your SUCCESS_CHECK can look for specific error codes as well. The LTP library also includes predefined SUCCESS_CHECK macros for the most common call conventions:
-
TST_RETVAL_EQ0() - The call was successful if FUNC returned 0 or NULL
-
TST_RETVAL_NOTNULL() - The call was successful if FUNC returned any value other than 0 or NULL.
-
TST_RETVAL_GE0() - The call was successful if FUNC returned value >= 0.
/* Keep trying for 1 second */
TST_RETRY_FUNC(FUNC, SUCCESS_CHECK)
/* Keep trying for up to 2*N seconds */
TST_RETRY_FN_EXP_BACKOFF(FUNC, SUCCESS_CHECK, N)
The shell version of TST_RETRY_FUNC() is simpler and takes slightly different arguments:
-
FUNC is a string containing the complete function or program call with arguments.
-
EXPECTED_RET is a single expected return value. FUNC call was successful if the return value is equal to EXPECTED_RET.
# Keep trying for 1 second
TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET"
# Keep trying for up to 2*N seconds
TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N"
# returns zero if passed an integer parameter, non-zero otherwise
tst_is_int "$FOO"
# returns zero if passed an integer or floating point number parameter,
# non-zero otherwise
tst_is_num "$FOO"
There is no $RANDOM in portable shell, use tst_random instead.
# get random integer between 0 and 1000 (including 0 and 1000)
tst_random 0 1000
The tst_mkfs helper will format device with the filesystem.
# format test device with ext2
tst_mkfs ext2 $TST_DEVICE
# default params are $TST_FS_TYPE $TST_DEVICE
tst_mkfs
# optional parameters
tst_mkfs ext4 /dev/device -T largefile
The tst_mount and tst_umount helpers are a safe way to mount/umount a filesystem.
The tst_mount mounts $TST_DEVICE of $TST_FS_TYPE (optional) to $TST_MNTPOINT (defaults to mntpoint), optionally using the $TST_MNT_PARAMS. The $TST_MNTPOINT directory is created if it didn’t exist prior to the function call.
If the path passed (optional, must be absolute path, defaults to $TST_MNTPOINT) to the tst_umount is not mounted (present in /proc/mounts) it’s noop. Otherwise it retries to umount the filesystem a few times on failure. This is a workaround since there are daemons dumb enough to probe all newly mounted filesystems, and prevents them from being umounted shortly after they were mounted.
These functions supply the SAFE_MACROS used in C although they work and are named differently.
ROD_SILENT command arg1 arg2 ...
# is shorthand for:
command arg1 arg2 ... > /dev/null 2>&1
if [ $? -ne 0 ]; then
tst_brk TBROK "..."
fi
ROD command arg1 arg2 ...
# is shorthand for:
ROD arg1 arg2 ...
if [ $? -ne 0 ]; then
tst_brk TBROK "..."
fi
Warning
|
Keep in mind that output redirection (to a file) happens in the caller rather than in the ROD function and cannot be checked for write errors by the ROD function. |
As a matter of a fact doing ROD echo a > /proc/cpuinfo
would work just fine
since the ROD function will only get the echo a
part that will run just
fine.
# Redirect output to a file with ROD
ROD echo foo \> bar
Note the > is escaped with \, this causes that the > and filename are passed to the ROD function as parameters and the ROD function contains code to split $@ on > and redirects the output to the file.
EXPECT_PASS command arg1 arg2 ... [ \> file ]
EXPECT_FAIL command arg1 arg2 ... [ \> file ]
EXPECT_PASS calls tst_res TPASS if the command exited with 0 exit code, and tst_res TFAIL otherwise. EXPECT_FAIL does vice versa.
Output redirection rules are the same as for the ROD function. In addition to that, EXPECT_FAIL always redirects the command’s stderr to /dev/null.
There are also EXPECT_PASS_BRK and EXPECT_FAIL_BRK, which works the same way except breaking a test when unexpected action happen.
It’s possible to detect whether expected value happened:
if ! EXPECT_PASS command arg1 2\> /dev/null; then
continue
fi
This command compares the currently running kernel version given conditions with syntax similar to the shell test command.
# Exit the test if kernel version is older or equal to 2.6.8
if tst_kvcmp -le 2.6.8; then
tst_brk TCONF "Kernel newer than 2.6.8 is needed"
fi
# Exit the test if kernel is newer than 3.8 and older than 4.0.1
if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then
tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1"
fi
expression | description |
---|---|
-eq kver |
Returns true if kernel version is equal |
-ne kver |
Returns true if kernel version is not equal |
-gt kver |
Returns true if kernel version is greater |
-ge kver |
Returns true if kernel version is greater or equal |
-lt kver |
Returns true if kernel version is lesser |
-le kver |
Returns true if kernel version is lesser or equal |
-a |
Does logical and between two expressions |
-o |
Does logical or between two expressions |
The format for kernel version has to either be with one dot e.g. 2.6 or with two dots e.g. 4.8.1.
#!/bin/sh
...
# whether current directory has 100MB free space at least.
if ! tst_fs_has_free . 100MB; then
tst_brkm TCONF "Not enough free space"
fi
...
The tst_fs_has_free shell interface returns 0 if the specified free space is satisfied, 1 if not, and 2 on error.
The second argument supports suffixes kB, MB and GB, the default unit is Byte.
#!/bin/sh
...
# Retry ping command three times
tst_retry "ping -c 1 127.0.0.1"
if [ $? -ne 0 ]; then
tst_resm TFAIL "Failed to ping 127.0.0.1"
else
tst_resm TPASS "Successfully pinged 127.0.0.1"
fi
...
The tst_retry function allows you to retry a command after waiting small amount of time until it succeeds or until given amount of retries has been reached (default is three attempts).
Restarting system daemons is a complicated task for two reasons.
-
There are different init systems (SysV init, systemd, etc…)
-
Daemon names are not unified between distributions (apache vs httpd, cron vs crond, various syslog variations)
To solve these problems LTP has testcases/lib/daemonlib.sh library that provides functions to start/stop/query daemons as well as variables that store correct daemon name.
start_daemon() |
Starts daemon, name is passed as first parameter. |
stop_daemon() |
Stops daemon, name is passed as first parameter. |
restart_daemon() |
Restarts daemon, name is passed as first parameter. |
status_daemon() |
Detect daemon status (exit code: 0: running, 1: not running). |
CROND_DAEMON |
Cron daemon name (cron, crond). |
SYSLOG_DAEMON |
Syslog daemon name (syslog, syslog-ng, rsyslog). |
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Cron daemon restart example
TCID=cron01
TST_COUNT=1
. test.sh
. daemonlib.sh
...
restart_daemon $CROND_DAEMON
...
tst_exit
The shell library provides an implementation of the checkpoint interface compatible with the C version. All TST_CHECKPOINT_* functions are available.
In order to initialize checkpoints $TST_NEEDS_CHECKPOINTS must be set to 1 before the inclusion of tst_test.sh:
#!/bin/sh
TST_NEEDS_CHECKPOINTS=1
. tst_test.sh
Since both the implementations are compatible, it’s also possible to start a child binary process from a shell test and synchronize with it. This process must have checkpoints initialized by calling tst_reinit().