Test Writing Guidelines

Cyril Hrubis edited this page Mar 2, 2017 · 51 revisions

LTP Test Writing Guidelines

This document describes LTP guidelines and LTP test interface and is intended for anybody who want to write or modify a LTP testcase. It’s not a definitive guide and it’s not, by any means, a substitute for common sense.

1. General Rules

1.1 Simplicity

For all it’s worth keep the testcases simple or better as simple as possible. The kernel and libc are tricky beasts and the complexity imposed by their interfaces is quite high. Concentrate on the interface you want to test and follow the UNIX philosophy. It’s a good idea to make the test as self-contained as possible too (it should not depend on tools or libraries that are not widely available).

Do not reinvent the wheel!

  • Use LTP standard interface

  • Do not add custom PASS/FAIL reporting functions

  • Do not write Makefiles from scratch, use LTP build system instead, etc.

  • …​

1.2 Code duplication

Copy & paste is a good servant but very poor master. If you are about to copy a large part of the code from one testcase to another, think what would happen if you find bug in the code that has been copied all around the tree. What about moving it to a library instead?

The same goes for short but complicated parts, whenever you are about to copy & paste a syscall wrapper that packs arguments accordingly to machine architecture or similarly complicated code, put it into a header instead.

1.3 Coding style

1.3.1 C coding style

LTP adopted Linux kernel coding style. If you aren’t familiar with its rules locate linux/Documentation/CodingStyle in the kernel sources and read it, it’s a well written introduction.

There is also a checkpatch (see linux/scripts/checkpatch.pl) script that can be used to check your patches before the submission.

Note
If checkpatch does not report any problems, the code still may be wrong as the tool only looks for common mistakes.

1.3.2 Shell coding style

When writing testcases in shell write in portable shell only, it’s a good idea to try to run the test using alternative shell (alternative to bash, for example dash) too.

Portable shell means Shell Command Language as defined by POSIX with a exception of few widely used extensions, namely local keyword used inside of functions and -o and -a test parameters (that are marked as obsolete in POSIX).

You can either try to run the testcases on Debian which has /bin/sh pointing to dash by default or install dash on your favorite distribution and use it to run the tests. If your distribution lacks dash package you can always compile it from source.

Debian also has nice devscript checkbashism.pl that can be used to check for non-portable shell code.

Here are some common sense style rules for shell

  • Keep lines under 80 chars

  • Use tabs for indentation

  • Keep things simple, avoid unnecessary subshells

  • Don’t do confusing things (i.e. don’t name your functions like common shell commands, etc.)

  • Quote variables

  • Be consistent

1.4 Commenting code

Comments can sometimes save you day but they can easily do more harm than good. There has been several cases where comments and actual implementation were drifting slowly apart which yielded into API misuses and hard to find bugs. Remember there is only one thing worse than no documentation, wrong documentation.

Generally everybody should write code that is obvious (which unfortunately isn’t always possible). If there is a code that needs to be commented keep it short and to the point. Never ever comment the obvious.

In case of LTP testcases it’s customary to add a paragraph with highlevel test description somewhere at the beginning of the file (usually right under the GPL header). This helps other people to understand the overall goal of the test before they dive into the technical details.

1.5 Backwards compatibility

LTP test should be as backward compatible as possible. Think of an enterprise distributions with long term support (more than five years since the initial release) or of an embedded platform that needs to use several years old toolchain supplied by the manufacturer.

Therefore LTP test for more current features should be able to cope with older systems. It should at least compile fine and if it’s not appropriate for the configuration it should return TCONF (see test interface description below).

There are several types of checks we use:

The configure script is usually used to detect availability of a function declarations in system headers. It’s used to disable tests at compile time.

We also have runtime kernel version detection that can be used to disable tests at runtime.

Checking the errno value is another type of runtime check. Most of the syscalls returns either EINVAL or ENOSYS when syscall was not implemented or was disabled upon kernel compilation.

Sometimes it also makes sense to define a few macros instead of creating configure test. One example are Linux specific POSIX clock ids in include/lapi/posix_clocks.h.

1.6 Dealing with messed up legacy code

LTP contains a lot of old and messy code and we are cleaning it up as fast as we can but despite the efforts there is still a lot. If you start modifying old or a messed up testcase and your changes are more complicated than simple typo fixes you should do a cleanup first (in a separate patch). It’s easier to review the changes if you separate the formatting fixes from the changes that affects the test behavior.

The same goes for moving files. If you need a rename or move file do it in a separate patch.

1.7 License

Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or any later version).

2. Writing a testcase

2.1 LTP Structure

The structure of LTP is quite simple. Each test is a binary written either in portable shell or C. The test gets a configuration via environment variables and/or command line parameters, it prints additional information into the stdout and reports overall success/failure via the exit value.

Tests are generally placed under the testcases/ directory. Everything that is a syscall or (slightly confusingly) libc syscall wrapper goes under testcases/kernel/syscalls/. Then there is testcases/open_posix_testsuite which is a well maintained fork of the upstream project that has been dead since 2005 and also a number of directories with tests for more specific features.

2.1.1 Runtest Files

The list of tests to be executed is stored in runtest files under the runtest/ directory. The default set of runtest files to be executed is stored in scenario_groups/default. When you add a test you should add corresponding entries into some runtest file(s) as well.

For syscall tests (these placed under testcases/kernel/syscalls/) use runtest/syscalls file, for kernel related tests for memory management we have runtest/mm, etc.

Important
The runtest files should have one entry per a test. Creating a wrapper that runs all your tests and adding it as a single test into runtest file is strongly discouraged.

2.1.2 Datafiles

If your test needs datafiles to work, these should be put into a subdirectory named datafiles and installed into the testcases/data/$TCID directory (to do that you have to add INSTALL_DIR := testcases/data/TCID into the datafiles/Makefile).

You can obtain path to datafiles via $TST_DATAROOT provided by test.sh $TST_DATAROOT/…​ or via C function tst_dataroot() provided by libltp:

const char *dataroot = tst_dataroot();

Datafiles can also be accessed as $LTPROOT/testcases/data/$TCID/…​, but $TST_DATAROOT and tst_dataroot() are preferred as these can be used when running testcases directly in git tree as well as from install location.

The path is constructed according to these rules:

  1. if $LTPROOT is set, return $LTPROOT/testcases/data/$TCID

  2. else if tst_tmpdir() was called return $STARTWD/datafiles (where $STARTWD is initial working directory as recorded by tst_tmdir())

  3. else return $CWD/datafiles

See testcases/commands/file/ for example.

2.1.3 Subexecutables

If you test needs to execute a binary, place it in the same directory as the testcase and name the file starting with testname_ (.tid see below). Once the test is executed by the framework, the path to the directory with all LTP binaries is added to the $PATH and you can execute it just by its name.

Tip
If you need to execute such test from the LTP tree, you can add path to current directory to $PATH manually with: PATH="$PATH:$PWD" ./foo01.

2.2 Writing a test in C

2.2.1 Basic test structure

Let’s start with an example, following code is a simple test for a getenv().

/*
 * This is test for basic functionality of getenv().
 *
 *  - create an env variable and verify that getenv() can get get it
 *  - call getenv() with nonexisting variable name, check that it returns NULL
 */

#include "tst_test.h"

#define ENV1 "LTP_TEST_ENV"
#define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST"
#define ENV_VAL "val"

static void setup(void)
{
	if (setenv(ENV1, ENV_VAL, 1))
		tst_brk(TBROK | TERRNO, "setenv() failed");
}

static void test(void)
{
	char *ret;

	ret = getenv(ENV1);

	if (!ret) {
		tst_res(TFAIL, "getenv(" ENV1 ") = NULL");
		goto next;
	}

	if (!strcmp(ret, ENV_VAL)) {
		tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'");
	} else {
		tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '"
		               ENV_VAL "'", ret);
	}

next:
	ret = getenv(ENV2);

	if (ret)
		tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret);
	else
		tst_res(TPASS, "getenv(" ENV2 ") = NULL");
}

static struct tst_test test = {
	.tid = "getenv01",
	.test_all = test,
	.setup = setup,
};

Each test includes the tst_test.h header and must define the struct tst_test test structure.

The .tid defines test name (usually syscall/libcall name + number). The name is used in the test results as well as a base for temporary directory name if temporary directory is needed. In most of the cases it’s the same as test filename without the extension.

The overall test initialization is done in the setup() function.

The overall cleanup is done in a cleanup() function. Here cleanup() is omitted as the test does not have anything to clean up. If cleanup is set in the test structure it’s called on test exit just before the test library cleanup. That especially means that cleanup can be called at any point in a test execution. For example even when a test setup step has failed, therefore the cleanup() function must be able to cope with unfinished initialization, and so on.

The test itself is done in the test() function. The test function must work fine if called in a loop.

There are two types of a test function pointers in the test structure. The first one is a .test_all pointer that is used when test is implemented as a single function. Then there is a .test function along with the number of tests .tcnt that allows for more detailed result reporting. If the .test pointer is set the function is called .tcnt times with an integer parameter in range of [0, .tcnt - 1].

Important
Only one of .test and .test_all can be set at a time.

Each test has a default timeout set to 300s. The default timeout can be overriden by setting .timeout in the test structure or by calling tst_set_timeout() in the test setup().

A word about the cleanup() callback

There are a few rules that needs to be followed in order to write correct cleanup() callback.

  1. Free only resources that were initialized. Keep in mind that callback can be executed at any point in the test run.

  2. Make sure to free resources in the reverse order they were initialized. (Some of the steps may not depend on others and everything will work if there were swapped but let’s keep it in order.)

The first rule may seem complicated at first however, on the contrary, it’s quite easy. All you have to do is to keep track of what was already initialized. For example file descriptors needs to be closed only if they were assigned a valid file descriptor. For most of the things you need to create extra flag that is set right after successful initialization though. Consider, for example, test setup below.

static int fd0, fd1, mount_flag;

#define MNTPOINT "mntpoint"
#define FILE1 "mntpoint/file1"
#define FILE2 "mntpoint/file2"

static void setup(void)
{
	SAFE_MKDIR(MNTPOINT, 0777);
	SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
	SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0);
	mount_flag = 1;

	fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666);
	fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666);
}

In this case the cleanup() function may be invoked when any of the SAFE_* macros has failed and therefore must be able to work with unfinished initialization as well. Since global variables are initialized to zero we can just check that fd > 0 before we attempt to close it. The mount function requires extra flag to be set after device was successfully mounted.

static void cleanup(void)
{
	if (fd1 > 0)
		SAFE_CLOSE(fd1);

	if (fd0 > 0)
		SAFE_CLOSE(fd0);

	if (mount_flag && tst_umouont(MNTPOINT))
		tst_res(TWARN | TERRNO, "umount(%s)", MNTPOINT);
}
Important
SAFE_MACROS() used in cleanup do not exit the test. Failure only produces a warning and the cleanup() carries on. This is intentional as we want to execute as much cleanup() as possible.
Warning
Calling tst_brk() in test cleanup() does not exit the test as well and TBROK is converted to TWARN.
Note
Creation and removal of the test temporary directory is handled in the test library and the directory is removed recursively. Therefore we do not have to remove files and directories in the test cleanup.

2.2.2 Basic test interface

void tst_res(int ttype, char *arg_fmt, ...);

Printf-like function to report test result, it’s mostly used with ttype:

TPASS

Test has passed.

TFAIL

Test has failed.

TINFO

General message.

The ttype can be combined bitwise with TERRNO or TTERRNO to print errno, TEST_ERRNO respectively.

void tst_brk(int ttype, char *arg_fmt, ...);

Printf-like function to report error and exit the test, it can be used with ttype:

TBROK

Something has failed in test preparation phase.

TCONF

Test is not appropriate for current configuration (syscall not implemented, unsupported arch, …​)

The ttype can be combined bitwise with TERRNO or TTERRNO to print errno, TEST_ERRNO respectively.

const char *tst_strsig(int sig);

Return the given signal number’s corresponding string.

const char *tst_strerrno(int err);

Return the given errno number’s corresponding string. Using this function to translate errno values to strings is preferred. You should not use the strerror() function in the testcases.

void tst_set_timeout(unsigned int timeout);

Allows for setting timeout per test iteration dymanically in the test setup(), the timeout is specified in seconds.

2.2.3 Test temporary directory

If .needs_tmpdir is set to 1 in the struct tst_test unique test temporary is created and it’s set as the test working directory. Tests MUST NOT create temporary files outside that directory.

Important
Close all file descriptors (that point to files in test temporary directory, even the unlinked ones) either in the test() function or in the test cleanup() otherwise the test may break temporary directory removal on NFS (look for "NFS silly rename").

2.2.4 Safe macros

Safe macros aim to simplify error checking in test preparation. Instead of calling system API functions, checking for their return value and aborting the test if the operation has failed, you just use corresponding safe macro.

Use them whenever it’s possible.

Instead of writing:

	fd = open("/dev/null", O_RDONLY);
	if (fd < 0)
		tst_brk(TBROK | TERRNO, "opening /dev/null failed");

You write just:

	fd = SAFE_OPEN("/dev/null", O_RDONLY);
Important
The SAFE_CLOSE() function also sets the passed file descriptor to -1 after it’s successfully closed.

They can also simplify reading and writing of sysfs files, you can, for example, do:

	SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max);

See include/tst_safe_macros.h, include/tst_safe_stdio.h and include/tst_safe_file_ops.h and include/tst_safe_net.h for a complete list.

2.2.5 Test specific command line options

struct tst_option {
        char *optstr;
        char **arg;
        char *help;
};

Test specific command line parameters can be passed with the NULL-terminated array of struct tst_option. The optstr is the command line option i.e. "o" or "o:" if option has a parameter. Only short options are supported. The arg is where optarg is stored upon match. If option has no parameter it’s set to non-NULL value if option was present. The help is a short help string.

Note
The test parameters must not collide with common test parameters defined in the library the currently used ones are -i, -I, -C, and -h.
int tst_parse_int(const char *str, int *val, int min, int max);
int tst_parse_float(const char *str, float *val, float min, float max);

Helpers for parsing the the strings returned in the struct tst_option.

Both return zero on success and errno, mostly EINVAL or ERANGE, on failure.

Both functions are no-op if str is NULL.

The valid range for result includes both min and max.

Example Usage
#include <limits.h>
#include "tst_test.h"

static char *str_threads;
static int threads = 10;

static struct tst_option options[] = {
	{"t:", &str_threads, "Number of threads (default 10)"},
	...
	{NULL, NULL, NULL}
};

static void setup(void)
{
	if (tst_parse_int(str_threads, &threads, 1, INT_MAX))
		tst_brk(TBROK, "Invalid number of threads '%s'", str_threads);

	...
}

static void test_threads(void)
{
	...

	for (i = 0; i < threads; i++) {
		...
	}

	...
}

static struct tst_test test = {
	...
	.options = options,
	...
};

2.2.6 Runtime kernel version detection

Testcases for newly added kernel functionality require kernel newer than a certain version to run. All you need to skip a test on older kernels is to set the .min_kver string in the struct tst_test to a minimal required kernel version, e.g. .min_kver = "2.6.30".

For more complicated operations such as skipping a test for a certain range of kernel versions, following functions could be used:

int tst_kvercmp(int r1, int r2, int r3);

struct tst_kern_exv {
        char *dist_name;
        char *extra_ver;
};

int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers);

These two functions are intended for runtime kernel version detection. They parse the output from uname() and compare it to the passed values.

The return value is similar to the strcmp() function, i.e. zero means equal, negative value means that the kernel is older than than the expected value and positive means that it’s newer.

The second function tst_kvercmp2() allows for specifying per-vendor table of kernel versions as vendors typically backport fixes to their kernels and the test may be relevant even if the kernel version does not suggests so. See testcases/kernel/syscalls/inotify/inotify04.c for example usage.

Warning
The shell tst_kvercmp maps the result into unsigned integer - the process exit value.

2.2.7 Fork()-ing

Be wary that if the test forks and there were messages printed by the tst_*() interfaces, the data may still be in libc/kernel buffers and these ARE NOT flushed automatically.

This happens when stdout gets redirected to a file. In this case, the stdout is not line buffered, but block buffered. Hence after a fork content of the buffers will be printed by the parent and each of the children.

To avoid that you should use SAFE_FORK().

Important
You have to set the .forks_child flag in the test structure if your testcase forks.

2.2.8 Doing the test in the child process

Results reported by tst_res() are propagated to the parent test process via block of shared memory.

Calling tst_brk() causes child process to exit with non-zero exit value. Which means that it’s safe to use SAFE_*() macros in the child processes as well.

Children that outlive the test() function execution are waited for in the test library. Unclean child exit (killed by signal, non-zero exit value, etc.) will cause the main test process to exit with tst_brk(), which especially means that TBROK propagated from a child process will cause the whole test to exit with TBROK.

If a test needs a child that segfaults or does anything else that cause it to exit uncleanly all you need to do is to wait for such children from the test() function so that it’s reaped before the main test exits the test() function.

#include "tst_test.h"

void tst_reap_children(void);

The tst_reap_children() function makes the process wait for all of its children and exits with tst_brk(TBROK, …​) if any of them returned a non zero exit code.

2.2.9 Fork() and Parent-child synchronization

As LTP tests are written for Linux, most of the tests involve fork()-ing and parent-child process synchronization. LTP includes a checkpoint library that provides wait/wake futex based functions.

In order to use checkpoints the .needs_checkpoints flag in the struct tst_test must be set to 1, this causes the test library to initialize checkpoints before the test() function is called.

#include "tst_test.h"

TST_CHECKPOINT_WAIT(id)

TST_CHECKPOINT_WAIT2(id, msec_timeout)

TST_CHECKPOINT_WAKE(id)

TST_CHECKPOINT_WAKE2(id, nr_wake)

TST_CHECKPOINT_WAKE_AND_WAIT(id)

The checkpoint interface provides pair of wake and wait functions. The id is unsigned integer which specifies checkpoint to wake/wait for. As a matter of fact it’s an index to an array stored in a shared memory, so it starts on 0 and there should be enough room for at least of hundred of them.

The TST_CHECKPOINT_WAIT() and TST_CHECKPOINT_WAIT2() suspends process execution until it’s woken up or until timeout is reached.

The TST_CHECKPOINT_WAKE() wakes one process waiting on the checkpoint. If no process is waiting the function retries until it success or until timeout is reached.

If timeout has been reached process exits with appropriate error message (uses tst_brk()).

The TST_CHECKPOINT_WAKE2() does the same as TST_CHECKPOINT_WAKE() but can be used to wake precisely nr_wake processes.

The TST_CHECKPOINT_WAKE_AND_WAIT() is a shorthand for doing wake and then immediately waiting on the same checkpoint.

Child processes created via SAFE_FORK() are ready to use the checkpoint synchronization functions, as they inherited the mapped page automatically.

Child processes started via exec(), or any other processes not forked from the test process must initialize the checkpoint by calling tst_reinit().

For the details of the interface, look into the include/tst_checkpoint.h.

#include "tst_test.h"

/*
 * Waits for process state change.
 *
 * The state is one of the following:
 *
 * R - process is running
 * S - process is sleeping
 * D - process sleeping uninterruptibly
 * Z - zombie process
 * T - process is traced
 */
TST_PROCESS_STATE_WAIT(pid, state)

The TST_PROCESS_STATE_WAIT() waits until process pid is in requested state. The call polls /proc/pid/stat to get this information.

It’s mostly used with state S which means that process is sleeping in kernel for example in pause() or any other blocking syscall.

2.2.10 Signal handlers

If you need to use signal handlers, keep the code short and simple. Don’t forget that the signal handler is called asynchronously and can interrupt the code execution at any place.

This means that problems arise when global state is changed both from the test code and signal handler, which will occasionally lead to:

  • Data corruption (data gets into inconsistent state), this may happen, for example, for any operations on FILE objects.

  • Deadlock, this happens, for example, if you call malloc(2), free(2), etc. from both the test code and the signal handler at the same time since malloc has global lock for it’s internal data structures. (Be wary that malloc(2) is used by the libc functions internally too.)

  • Any other unreproducible and unexpected behavior.

Quite common mistake is to call exit(3) from a signal handler. Note that this function is not signal-async-safe as it flushes buffers, etc. If you need to exit a test immediately from a signal handler use _exit(2) instead.

Tip
See man 7 signal for the list of signal-async-safe functions.

If a signal handler sets a variable, its declaration must be volatile, otherwise compiler may misoptimize the code. This is because the variable may not be changed in the compiler code flow analysis. There is sig_atomic_t type defined in C99 but this one DOES NOT imply volatile (it’s just a typedef to int). So the correct type for a flag that is changed from a signal handler is either volatile int or volatile sig_atomic_t.

2.2.11 Kernel Modules

There are certain cases where the test needs a kernel part and userspace part, happily, LTP can build a kernel module and then insert it to the kernel on test start for you. See testcases/kernel/device-drivers/block for details.

2.2.11 Useful macros

ARRAY_SIZE(arr)

Returns the size of statically defined array, i.e. (sizeof(arr) / sizeof(*arr))

LTP_ALIGN(x, a)

Aligns the x to be next multiple of a. The a must be power of 2.

2.2.12 Filesystem type detection

Some tests are known to fail on certain filesytems (you cannot swap on TMPFS, there are unimplemented fcntl() etc.).

If your test needs to be skipped on certain filesystems, use the interface below:

#include "tst_test.h"

	/*
	 * Unsupported only on NFS.
	 */
	if (tst_fs_type(".") == TST_NFS_MAGIC)
		tst_brk(TCONF, "Test not supported on NFS filesystem");


	/*
	 * Unsupported on NFS, TMPFS and RAMFS
	 */
	long type;

	switch ((type = tst_fs_type("."))) {
	case TST_NFS_MAGIC:
	case TST_TMPFS_MAGIC:
	case TST_RAMFS_MAGIC:
		tst_brk(TCONF, "Test not supported on %s filesystem",
		        tst_fs_type_name(type));
	break;
	}

2.2.13 Thread-safety in the LTP library

It is safe to use library tst_res() function in multi-threaded tests.

Only the main thread must return from the test() function to the test library and that must be done only after all threads that may call any library function has been terminated. That especially means that threads that may call tst_brk() must terminate before the execution of the test() function returns to the library. This is usually done by the main thread joining all worker threads at the end of the test() function. Note that the main thread will never get to the library code in a case that tst_brk() was called from one of the threads since it will sleep at least in pthread_join() on the thread that called the tst_brk() till exit() is called by tst_brk().

The test-supplied cleanup function runs concurrently to the rest of the threads in a case that cleanup was entered from tst_brk(). Subsequent threads entering tst_brk() must be suspended or terminated at the start of the the user supplied cleanup function. It may be necessary to stop or exit the rest of the threads before the test cleans up as well. For example threads that create new files should be stopped before temporary directory is be removed.

Following code example shows thread safe cleanup function example using atomic increment as a guard. The library calls its cleanup after the execution returns from the user supplied cleanup and expects that only one thread returns from the user supplied cleanup to the test library.

#include "tst_test.h"

static void cleanup(void)
{
	static int flag;

	if (tst_atomic_inc(&flag) != 1)
		pthread_exit(NULL);

	/* if needed stop the rest of the threads here */

	...

	/* then do cleanup work */

	...

	/* only one thread returns to the library */
}

2.2.14 Testing with a block device

Some tests needs a block device (inotify tests, syscall EROFS failures, etc.). LTP library contains a code to prepare a testing device.

If .needs_device flag in the struct tst_test is set the the tst_device structure is initialized with a path to a test device and default filesystem to be used.

You can also request minimal device size in megabytes by setting .dev_min_size the device is guaranteed to have at least the requested size then.

If .format_device flag is set the device is formatted with a filesystem as well. You can use .dev_fs_type to override the default filesystem type if needed and pass additional options to mkfs via .dev_fs_opts and .dev_extra_opt pointers.

If .mount_device is set, the device is mounted at .mntpoint which is used to pass a directory name that will be created and used as mount destination. You can pass additional flags and data to the mount command via .mnt_flags and .mnt_data pointers.

#include "tst_test.h"

struct tst_device {
	const char *dev;
	const char *fs_type;
};

extern struct tst_device *tst_device;

int tst_umount(const char *path);

In case that LTP_DEV is passed to the test in an environment, the library checks that the file exists and that it’s a block device, if .device_min_size is set the device size is checked as well. If LTP_DEV wasn’t set or if size requirements were not met a temporary file is created and attached to a free loop device.

If there is no usable device and loop device couldn’t be initialized the test exits with TCONF.

The tst_umount() function works exactly as umount(2) but retries several times on EBUSY. This is because various desktop daemons (gvfsd-trash is known for that) may be stupid enough to probe all newly mounted filesystem which results in umount(2) failing with EBUSY.

Important
All testcases should use tst_umount() instead of umount(2) to umount filesystems.

2.2.15 Formatting a device with a filesystem

#include "tst_test.h"

static void setup(void)
{
	...
	SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
	...
}

This function takes a path to a device, filesystem type and an array of extra options passed to mkfs.

The fs options fs_opts should either be NULL if there are none, or a NULL terminated array of strings such as: const char *const opts[] = {"-b", "1024", NULL}.

The extra option extra_opt should either be NULL if there is none, or a string such as "102400"; extra_opt will be passed after device name. e.g: mkfs -t ext4 -b 1024 /dev/sda1 102400 in this case.

2.2.16 Verifying a filesystem’s free space

Some tests have size requirements for the filesystem’s free space. If these requirements are not satisfied, the tests should be skipped.

#include "tst_test.h"

int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult);

The tst_fs_has_free() function returns 1 if there is enough space and 0 if there is not.

The path is the pathname of any directory/file within a filesystem.

The mult is a multiplier, one of TST_BYTES, TST_KB, TST_MB or TST_GB.

The required free space is calculated by size * mult, e.g. tst_fs_has_free("/tmp/testfile", 64, TST_MB) will return 1 if the filesystem, which "/tmp/testfile" is in, has 64MB free space at least, and 0 if not.

2.2.17 Files, directories and fs limits

Some tests need to know the maximum count of links to a regular file or directory, such as rename(2) or linkat(2) to test EMLINK error.

#include "tst_test.h"

int tst_fs_fill_hardlinks(const char *dir);

Try to get maximum count of hard links to a regular file inside the dir.

Note
This number depends on the filesystem dir is on.

This function uses link(2) to create hard links to a single file until it gets EMLINK or creates 65535 links. If the limit is hit, the maximum number of hardlinks is returned and the dir is filled with hardlinks in format "testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if link(2) failed with ENOSPC or EDQUOT, zero is returned and previously created files are removed.

#include "tst_test.h"

int tst_fs_fill_subdirs(const char *dir);

Try to get maximum number of subdirectories in directory.

Note
This number depends on the filesystem dir is on. For current kernel, subdir limit is not available for all filesystems (available for ext2, ext3, minix, sysv and more). If the test runs on some other filesystems, like ramfs, tmpfs, it will not even try to reach the limit and return 0.

This function uses mkdir(2) to create directories in dir until it gets EMLINK or creates 65535 directories. If the limit is hit, the maximum number of subdirectories is returned and the dir is filled with subdirectories in format "testdir%i", where i belongs to [0, limit - 2) interval (because each newly created dir has two links already - the . and the link from parent dir). If no limit is hit or if mkdir(2) failed with ENOSPC or EDQUOT, zero is returned and previously created directories are removed.

#include "tst_test.h"

int tst_dir_is_empty(const char *dir, int verbose);

Returns non-zero if directory is empty and zero otherwise.

Directory is considered empty if it contains only . and ...

2.2.18 Getting an unused PID number

Some tests require a PID, which is not used by the OS (does not belong to any process within it). For example, kill(2) should set errno to ESRCH if it’s passed such PID.

#include "tst_test.h"

pid_t tst_get_unused_pid(void);

Return a PID value not used by the OS or any process within it.

#include "tst_test.h"

int tst_get_free_pids(void);

Returns number of unused pids in the system. Note that this number may be different once the call returns and should be used only for rough estimates.

2.2.20 Running executables

#include "tst_test.h"

int tst_run_cmd(const char *const argv[],
	        const char *stdout_path,
	        const char *stderr_path,
	        int pass_exit_val);

tst_run_cmd is a wrapper for vfork() + execvp() which provides a way to execute an external program.

argv[] is a NULL-terminated array of strings starting with the program name which is followed by optional arguments.

A non-zero pass_exit_val makes tst_run_cmd return the program exit code to the caller. A zero for pass_exit_val makes tst_run_cmd exit the tests on failure.

In case that execvp() has failed and the pass_exit_val flag was set, the return value is 255 if execvp() failed with ENOENT and 254 otherwise.

stdout_path and stderr_path determine where to redirect the program stdout and stderr I/O streams.

Example
#include "tst_test.h"

const char *const cmd[] = { "ls", "-l", NULL };

...
	/* Store output of 'ls -l' into log.txt */
	tst_run_cmd(cmd, "log.txt", NULL, 0);
...

2.2.21 Measuring elapsed time and helper functions

#include "tst_test.h"

void tst_timer_check(clockid_t clk_id);

void tst_timer_start(clockid_t clk_id);

void tst_timer_stop(void);

struct timespec tst_timer_elapsed(void);

long long tst_timer_elapsed_ms(void);

long long tst_timer_elapsed_us(void);

The tst_timer_check() function checks if specified clk_id is suppored and exits the test with TCONF otherwise. It’s expected to be used in test setup() before any resources that needs to be cleaned up are initialized, hence it does not include a cleanup function parameter.

The tst_timer_start() marks start time and stores the clk_id for further use.

The tst_timer_stop() marks the stop time using the same clk_id as last call to tst_timer_start().

The tst_timer_elapsed*() returns time difference between the timer start and last timer stop in several formats and units.

Important
The timer functions use clock_gettime() internally which needs to be linked with -lrt on older glibc. Please do not forget to add LDLIBS+=-lrt in Makefile.
long long tst_timespec_to_us(struct timespec t);
long long tst_timespec_to_ms(struct timespec t);

struct timeval tst_us_to_timeval(long long us);
struct timeval tst_ms_to_timeval(long long ms);

int tst_timespec_lt(struct timespec t1, struct timespec t2);

struct timespec tst_timespec_add_us(struct timespec t, long long us);

struct timespec tst_timespec_diff(struct timespec t1, struct timespec t2);
long long tst_timespec_diff_us(struct timespec t1, struct timespec t2);
long long tst_timespec_diff_ms(struct timespec t1, struct timespec t2);

struct timespec tst_timespec_abs_diff(struct timespec t1, struct timespec t2);
long long tst_timespec_abs_diff_us(struct timespec t1, struct timespec t2);
long long tst_timespec_abs_diff_ms(struct timespec t1, struct timespec t2);

The first four functions are simple inline conversion functions.

The tst_timespec_lt() function returns non-zero if t1 is earlier than t2.

The tst_timespec_add_us() function adds us microseconds to the timespec t. The us is expected to be positive.

The tst_timespec_diff*() functions returns difference between two times, the t1 is expected to be later than t2.

The tst_timespec_abs_diff*() functions returns absolute value of difference between two times.

Note
All conversions to ms and us rounds the value.

2.2.22 Datafiles

#include "tst_test.h"

static const char *const res_files[] = {
	"foo",
	"bar",
	NULL
};

static struct tst_test test = {
	...
	.resource_files = res_files,
	...
}

If the test needs additional files to be copied to the test temporary directory all you need to do is to list their filenames in the NULL-terminated array .resource_files in the tst_test structure.

When resource files is set test temporary directory is created automatically, there is need to set .needs_tmpdir as well.

The test library looks for datafiles first, these are either stored in a directory called datafiles in the $PWD at the start of the test or in $LTPROOT/testcases/data/$tid. If the file is not found the library looks into $LTPROOT/testcases/bin/ and to $PWD at the start of the test. This ensures that the testcases can copy the file(s) effortlessly both when test is started from the directory it was compiled in as well as when LTP was installed.

The file(s) are copied to the newly created test temporary directory which is set as the test working directory when the test() functions is executed.

2.2.23 Code path tracing

tst_res is a macro, so on when you define a function in one file:

int do_action(int arg)
{
	...

	if (ok) {
		tst_res(TPASS, "check passed");
		return 0;
	} else {
		tst_res(TFAIL, "check failed");
		return -1;
	}
}

and call it from another file, the file and line reported by tst_res in this function will be from the former file.

TST_TRACE can make the analysis of such situations easier. It’s a macro which inserts a call to tst_res(TINFO, …​) in case its argument evaluates to non-zero. In this call to tst_res(TINFO, …​) the file and line will be expanded using the actual location of TST_TRACE.

For example, if this another file contains:

#include "tst_test.h"

if (TST_TRACE(do_action(arg))) {
	...
}

the generated output may look similar to:

common.h:9: FAIL: check failed
test.c:8: INFO: do_action(arg) failed

2.3 Writing a testcase in shell

LTP supports testcases to be written in a portable shell too.

There is a shell library modeled closely to the C interface at testcases/lib/tst_test.sh.

Warning
All identifiers starting with TST_ or tst_ are reserved for the test library.

2.3.1 Basic test interface

#!/bin/sh
#
# This is a basic test for true shell buildin
#

TST_ID="true01"
TST_TESTFUNC=do_test
. tst_test.sh

do_test()
{
	true
	ret=$?

	if [ $ret -eq 0 ]; then
		tst_res TPASS "true returned 0"
	else
		tst_res TFAIL "true returned $ret"
	fi
}

tst_run
Tip
To execute this test the tst_test.sh library must be in $PATH. If you are executing the test from a git checkout you can run it as PATH="$PATH:../../lib" ./foo01.sh

The shell library expects test setup, cleanup and the test function executing the test in the $TST_SETUP, $TST_CLEANUP and $TST_TESTFUNC variables.

Both $TST_SETUP and $TST_CLEANUP are optional.

The $TST_TESTFUNC may be called several times if more than one test iteration was requested by passing right command line options to the test.

The $TST_CLEANUP may be called even in the middle of the setup and must be able to clean up correctly even in this situation. The easiest solution for this is to keep track of what was initialized and act accordingly in the cleanup.

Notice also the tst_run function called at the end of the test that actually starts the test.

#!/bin/sh
#
# Example test with tests in separate functions
#

TST_ID="example01"
TST_TESTFUNC=test
TST_CNT=2
. tst_test.sh

test1()
{
	tst_res TPASS "Test 1 passed"
}

test2()
{
	tst_res TPASS "Test 2 passed"
}

tst_run

If $TST_CNT is set, the test library looks if there are functions named ${TST_TESTFUNC}1, …​, ${TST_TESTFUNC}${TST_CNT} and if these are found they are executed one by one.

#!/bin/sh
#
# Example test with tests in a single function
#

TST_ID="example02"
TST_TESTFUNC=do_test
TST_CNT=2
. tst_test.sh

do_test()
{
	case $1 in
	1) tst_res TPASS "Test 1 passed";;
	2) tst_res TPASS "Test 2 passed";;
	esac
}

tst_run

Otherwise, if $TST_CNT is set but there is no ${TST_TESTFUNC}1, etc., the $TST_TESTFUNC is executed $TST_CNT times and the test number is passed to it in the $1.

2.3.2 Library variables

Similarily to the C library various checks and preparations can be requested simply by setting right $TST_NEEDS_FOO.

Variable name Action done

TST_NEEDS_ROOT

Exit the test with TCONF unless executed under root

TST_NEEDS_TMPDIR

Create test temporary directory and cd into it.

TST_NEEDS_DEVICE

Prepare test temporary device, the path to testing device is stored in $TST_DEVICE variable.

TST_NEEDS_CMDS

String with command names that has to be present for the test (see below).

TST_NEEDS_MODULE

Test module name needed for the test (see below).

Checking for presence of commands
#!/bin/sh

...

TST_NEEDS_CMDS="modinfo modprobe"
. tst_test.sh

...

Setting $TST_NEEDS_CMDS to a string listing required commands will check for existence each of them and exits the test with TCONF on first misssing.

Alternatively the tst_check_cmds() function can be used to do the same on runtime, since sometimes we need to the check at runtime too.

Locating kernel modules

The LTP build system can build kernel modules as well, setting $TST_NEEDS_MODULE to module name will cause to library to look for the module in a few possible paths.

If module was found the path to it will be stored into $TST_MODPATH variable, if module wasn’t found the test will exit with TCONF.

2.3.3 Optional command line parameters

#!/bin/sh
#
# Optional test command line parameters
#

TST_ID="example03"
TST_OPTS="af:"
TST_USAGE=usage
TST_PARSE_ARGS=parse_args
TST_TESTFUNC=do_test

. tst_test.sh

ALTERNATIVE=0
MODE="foo"

usage()
{
	cat << EOF
usage: $0 [-a] [-f <foo|bar>]

OPTIONS
-a     Enable support for alternative foo
-f     Specify foo or bar mode
EOF
}

parse_args()
{
	case $1 in
	a) ALTERNATIVE=1
	f) MODE="$2"
	esac
}

do_test()
{
	...
}

tst_run

The getopts string for optional parameters is passed in the $TST_OPTS variable. There are a few default parameters that cannot be used by a test, these can be listed with passing help -h option to any test.

The function that prints the usage is passed in $TST_USAGE, the help for the options implemented in the library is appended when usage is printed.

Lastly the fucntion $PARSE_ARGS is called with the option name in $1 and, if option has argument, its value in $2.

#!/bin/sh
#
# Optional test positional paramters
#

TST_ID="example04"
TST_POS_ARGS=3
TST_USAGE=usage
TST_TESTFUNC=do_test

. tst_test.sh

usage()
{
	cat << EOF
usage: $0 [min] [max] [size]

EOF
}

min="$1"
max="$2"
size="$3"

do_test()
{
	...
}

tst_run

You can also request a number of positional parameters by setting the $TST_POS_ARGS variable. If you do, these will be available as they were passed directly to the script in $1, $2, …​, $n.

2.3.4 Usefull library functions

Sleeping for subsecond intervals

Albeit there is a sleep command available basically everywhere not all implementations can support sleeping for less than one second. And most of the time sleeping for a second is too much. Therefore LTP includes tst_sleep that can sleep for defined amount of seconds, milliseconds or microseconds.

# sleep for 100 milliseconds
tst_sleep 100ms
Checking for integers
# returns zero if passed an integer parameter, non-zero otherwise
tst_is_int "$FOO"
Obtaining random numbers

There is no $RANDOM in portable shell, use tst_random instead.

# get random integer between 0 and 1000 (including 0 and 1000)
tst_random 0 1000
Formatting device with a filesystem

The tst_mkfs helper will format device with the filesystem.

# format test device with ext2
tst_mkfs ext2 $TST_DEVICE
Umounting filesystems

The tst_umount helper is a safe way to umount a filesystem.

If the path passed to the function is not mounted (present in /proc/mounts) it’s noop.

Otherwise it retries to umount the filesystem a few times on a failure, which is a workaround since there are a daemons dumb enough to probe all newly mounted filesystems, which prevents them from umounting shortly after they were mounted.

Running commands as different user with su

While some distributions retain paths added to $PATH when doing su user -c "command" this does not work at least in Debian. If you want to run LTP binaries as a different user you must use tst_su instead which sets up $PATH and the runs the command.

Run test child binary as a test user
#!/bin/sh
TCID=foo01
. test.sh

tst_su testusr foo01_child
if [ $? -ne 0 ]; then
	tst_resm TFAIL "foo failed"
else
	tst_resm TPASS "foo passed"
fi
ROD and ROD_SILENT

These functions supply the SAFE_MACROS used in C although they work and are named differently.

ROD_SILENT command arg1 arg2 ...

# is shorthand for:

command arg1 arg2 ... > /dev/null 2>&1
if [ $? -ne 0 ]; then
        tst_brkm TBROK "..."
fi


ROD command arg1 arg2 ...

# is shorthand for:

ROD arg1 arg2 ...
if [ $? -ne 0 ]; then
        tst_brkm TBROK "..."
fi
Warning
Keep in mind that output redirection (to a file) happens in the caller rather than in the ROD function and cannot be checked for write errors by the ROD function.

As a matter of a fact doing ROD echo a > /proc/cpuinfo would work just fine since the ROD function will only get the echo a part that will run just fine.

# Redirect output to a file with ROD
ROD echo foo \> bar

Note the > is escaped with \, this causes that the > and filename are passed to the ROD function as parameters and the ROD function contains code to split $@ on > and redirects the output to the file.

EXPECT_PASS and EXPECT_FAIL
EXPECT_PASS command arg1 arg2 ... [ \> file ]
EXPECT_FAIL command arg1 arg2 ... [ \> file ]

EXPECT_PASS calls tst_resm TPASS if the command exited with 0 exit code, and tst_resm TFAIL otherwise. EXPECT_FAIL does vice versa.

Output redirection rules are the same as for the ROD function. In addition to that, EXPECT_FAIL always redirects the command’s stderr to /dev/null.

tst_kvcmp

This command compares the currently running kernel version given conditions with syntax similar to the shell test command.

# Exit the test if kernel version is older or equal to 2.6.8
if tst_kvcmp -le 2.6.8; then
	tst_brk TCONF "Kernel newer than 2.6.8 is needed"
fi

# Exit the test if kernel is newer than 3.8 and older than 4.0.1
if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then
	tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1"
fi
expression description

-eq kver

Returns true if kernel version is equal

-ne kver

Returns true if kernel version is not equal

-gt kver

Returns true if kernel version is greater

-ge kver

Returns true if kernel version is greater or equal

-lt kver

Returns true if kernel version is lesser

-le kver

Returns true if kernel version is lesser or equal

-a

Does logical and between two expressions

-o

Does logical or between two expressions

The format for kernel version has to either be with one dot e.g. 2.6 or with two dots e.g. 4.8.1.

tst_fs_has_free
#!/bin/sh

...

# whether current directory has 100MB free space at least.
if ! tst_fs_has_free . 100MB; then
	tst_brkm TCONF "Not enough free space"
fi

...

The tst_fs_has_free shell interface returns 0 if the specified free space is satisfied, 1 if not, and 2 on error.

The second argument supports suffixes kB, MB and GB, the default unit is Byte.

tst_retry
#!/bin/sh

...

# Retry ping command three times
tst_retry "ping -c 1 127.0.0.1"

if [ $? -ne 0 ]; then
	tst_resm TFAIL "Failed to ping 127.0.0.1"
else
	tst_resm TPASS "Successfully pinged 127.0.0.1"
fi

...

The tst_retry function allows you to retry a command after waiting small amount of time until it succeeds or until given amount of retries has been reached (default is three attempts).

2.3.5 Restarting daemons

Restarting system daemons is a complicated task for two reasons.

  • There are different init systems (SysV init, systemd, etc…​)

  • Daemon names are not unified between distributions (apache vs httpd, cron vs crond, various syslog variations)

To solve these problems LTP has testcases/lib/daemonlib.sh library that provides functions to start/stop/query daemons as well as variables that store correct daemon name.

Table 1. Supported operations

start_daemon()

Starts daemon, name is passed as first parameter.

stop_daemon()

Stops daemon, name is passed as first parameter.

restart_daemon()

Restarts daemon, name is passed as first parameter.

status_daemon()

Returns daemon status, TODO: what is return value?

Table 2. Variables with detected names

CROND_DAEMON

Cron daemon name (cron, crond).

SYSLOG_DAEMON

Syslog daemon name (syslog, syslog-ng, rsyslog).

Cron daemon restart example
#!/bin/sh
#
# Cron daemon restart example
#
TCID=cron01
TST_COUNT=1
. test.sh
. daemonlib.sh

...

restart_daemon $CROND_DAEMON

...

tst_exit

2.3.6 Access to the checkpoint interface

The shell library provides an implementation of the checkpoint interface compatible with the C version. All TST_CHECKPOINT_* functions are available.

In order to initialize checkpoints $TST_NEEDS_CHECKPOINTS must be set to 1 before the inclusion of test.sh:

#!/bin/sh

TST_NEEDS_CHECKPOINTS=1
. test.sh

Since both the implementations are compatible, it’s also possible to start a child binary process from a shell test and synchronize with it. This process must have checkpoints initialized by calling tst_reinit()'.

3. Common problems

This chapter describes common problems/misuses and less obvious design patters (quirks) in UNIX interfaces. Read it carefully :)

3.1 umask()

I’ve been hit by this one several times already…​ When you create files with open() or creat() etc, the mode specified as the last parameter is not the mode the file is created with. The mode depends on current umask() settings which may clear some of the bits. If your test depends on specific file permissions you need either to change umask to 0 or chmod() the file afterwards or use SAFE_TOUCH() that does the chmod() for you.

3.2 access()

If access(some_file, W_OK) is executed by root, it will return success even if the file doesn’t have write permission bits set (the same holds for R_OK too). For sysfs files you can use open() as a workaround to check file read/write permissions. It might not work for other filesystems, for these you have to use stat(), lstat() or fstat().

3.3 umount() EBUSY

Various desktop daemons (gvfsd-trash is known for that) may be stupid enough to probe all newly mounted filesystem which results in umount(2) failing with EBUSY; use tst_umount() described in 2.2.19 that retries in this case instead of plain umount(2).

3.4 FILE buffers and fork()

Be vary that if a process calls fork(2) the child process inherits open descriptors as well as copy of the parent memory so especially if there are any open FILE buffers with a data in them they may be written both by the parent and children resulting in corrupted/duplicated data in the resulting files.

Also open FILE streams are flushed and closed at exit(3) so if your program works with FILE streams, does fork(2), and the child may end up calling exit(3) you will likely end up with corrupted files.

The solution to this problem is either simply call fflush(NULL) that flushes all open output FILE streams just before doing fork(2). You may also use _exit(2) in child processes which does not flush FILE buffers and also skips atexit(3) callbacks.

4. Test Contribution Checklist

  1. Test compiles and runs fine (check with -i 10 too)

  2. Checkpatch does not report any errors

  3. The runtest entires are in place

  4. Test files are added into corresponding .gitignore files

  5. Patches apply over the latest git

4.1 About .gitignore files

There are numerous .gitignore files in the LTP tree. Usually there is a .gitignore file per a group of tests. The reason for this setup is simple. It’s easier to maintain a .gitignore file per directory with tests, rather than having single file in the project root directory. This way, we don’t have to update all the gitignore files when moving directories, and they get deleted automatically when a directory with tests is removed.