-
Notifications
You must be signed in to change notification settings - Fork 7
Test Writing Guidelines
This document describes LTP guidelines and is intended for anybody who want to write or modify a LTP testcase. It’s not a definitive guide and it’s not, by any means, a substitute for common sense.
Note
|
See also C Test API, Shell Test API, LTP Library API Writing Guidelines. |
Rules and recommendations which are "machine checkable" should be
tagged with an ID like LTP-XXX
. There will be a corresponding entry
in
doc/rules.tsv. When
you run make check or make check-test it will display these IDs as
a reference.
For testcases it’s required that the source code is as easy to follow as possible. When a test starts to fail the failure has to be analyzed, clean test codebase makes this task much easier and quicker.
Here are some hints on how to write clean and understandable code, a few of these points are further discussed below:
-
First of all Keep things simple
-
Keep function and variable names short but descriptive
-
Keep functions reasonably short and focused on a single task
-
Do not overcomment
-
Be consistent
-
Avoid deep nesting
-
DRY
For all it’s worth keep the testcases simple or better as simple as possible.
The kernel and libc are tricky beasts and the complexity imposed by their interfaces is quite high. Concentrate on the interface you want to test and follow the UNIX philosophy.
It’s a good idea to make the test as self-contained as possible too, ideally tests should not depend on tools or libraries that are not widely available.
Do not reinvent the wheel!
-
Use LTP standard interface
-
Do not add custom PASS/FAIL reporting functions
-
Do not write Makefiles from scratch, use LTP build system instead
-
Etc.
Choosing a good name for an API functions or even variables is a difficult task do not underestimate it.
There are a couple of customary names for different things that help people to understand code, for example:
-
For loop variables are usually named with a single letter i, j, …
-
File descriptors fd or fd_foo.
-
Number of bytes stored in file are usually named as size or len
-
Etc.
Comments can sometimes save you day but they can easily do more harm than good. There has been several cases where comments and actual implementation drifted slowly apart which yielded into API misuses and hard to find bugs. Remember there is only one thing worse than no documentation, wrong documentation.
Ideally everybody should write code that is obvious, which unfortunately isn’t always possible. If there is a code that requires to be commented keep it short and to the point. These comments should explain why and not how things are done.
Never ever comment the obvious.
In case of LTP testcases it’s customary to add an asciidoc formatted comment paragraph with highlevel test description at the beginning of the file right under the GPL SPDX header. This helps other people to understand the overall goal of the test before they dive into the technical details. It’s also exported into generated documentation hence it should mostly explain what is tested.
Copy & paste is a good servant but very poor master. If you are about to copy a large part of the code from one testcase to another, think what would happen if you find bug in the code that has been copied all around the tree. What about moving it to a library instead?
The same goes for short but complicated parts, whenever you are about to copy & paste a syscall wrapper that packs arguments accordingly to machine architecture or similarly complicated code, put it into a header instead.
LTP adopted Linux kernel coding style. If you aren’t familiar with its rules locate linux/Documentation/CodingStyle in the kernel sources and read it, it’s a well written introduction.
Run make check
in the test’s directory and/or use make check-$TCID
,
it uses (among other checks) our vendored version of
checkpatch.pl
script from kernel git tree.
Note
|
If make check does not report any problems, the code still may be wrong
as all tools used for checking only look for common mistakes.
|
Test executables should not export symbols unnecessarily. This means that all top-level variables and functions should be marked with the static keyword. The only visible symbols should be those included from shared object files.
When writing testcases in shell write in portable shell only, it’s a good idea to try to run the test using alternative shell (alternative to bash, for example dash) too.
Portable shell means Shell Command Language as defined by POSIX with a exception of few widely used extensions, namely local keyword used inside of functions and -o and -a test parameters (that are marked as obsolete in POSIX).
You can either try to run the testcases on Debian which has /bin/sh pointing to dash by default or install dash on your favorite distribution and use it to run the tests. If your distribution lacks dash package you can always compile it from source.
Run make check
in the test’s directory and/or use make check-$TCID.sh
,
it uses (among other checks) our vendored version of
checkbashism.pl
from Debian, that is used to check for non-portable shell code.
Note
|
If make check does not report any problems, the code still may be wrong
as checkbashisms.pl used for checking only looks for common mistakes.
|
Here are some common sense style rules for shell
-
Keep lines under 80 chars
-
Use tabs for indentation
-
Keep things simple, avoid unnecessary subshells
-
Don’t do confusing things (i.e. don’t name your functions like common shell commands, etc.)
-
Quote variables
-
Be consistent
LTP test should be as backward compatible as possible. Think of an enterprise distributions with long term support (more than five years since the initial release) or of an embedded platform that needs to use several years old toolchain supplied by the manufacturer.
Therefore LTP test for more current features should be able to cope with older systems. It should at least compile fine and if it’s not appropriate for the configuration it should return TCONF.
There are several types of checks we use:
The configure script is usually used to detect availability of a function declarations in system headers. It’s used to disable tests at compile time or to enable fallback definitions.
Checking the errno value is another type of runtime check. Most of the syscalls returns either EINVAL or ENOSYS when syscall was not implemented or was disabled upon kernel compilation.
LTP has kernel version detection that can be used to disable tests at runtime, unfortunately kernel version does not always corresponds to a well defined feature set as distributions tend to backport hundreds of patches while the kernel version stays the same. Use with caution.
Lately we added kernel .config parser, a test can define a boolean expression of kernel config variables that has to be satisfied in order for a test to run. This is mostly used for kernel namespaces at the moment.
Sometimes it also makes sense to define a few macros instead of creating configure test. One example are Linux specific POSIX clock ids in include/lapi/posix_clocks.h.
LTP still contains a lot of old and messy code and we are cleaning it up as fast as we can but despite the decade of efforts there is still a lot. If you start modifying old or a messy testcase and your changes are more complicated than simple typo fixes you should convert the test into a new library first.
It’s also much easier to review the changes if you split them into a smaller logical groups. The same goes for moving files. If you need a rename or move file do it in a separate patch.
Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or any later version).
Use SPDX-License-Identifier: GPL-2.0-or-later
The structure of LTP is quite simple. Each test is a binary written either in portable shell or C. The test gets a configuration via environment variables and/or command line parameters, it prints additional information into the stdout and reports overall success/failure via the exit value.
Tests are generally placed under the testcases/ directory. Everything that is a syscall or (slightly confusingly) libc syscall wrapper goes under testcases/kernel/syscalls/.
Then there is testcases/open_posix_testsuite/ which is a well maintained fork of the upstream project that has been dead since 2005 and also a number of directories with tests for more specific features.
The list of tests to be executed is stored in runtest files under the runtest/ directory. The default set of runtest files to be executed is stored in scenario_groups/default. When you add a test you should add corresponding entries into some runtest file(s) as well.
For syscall tests (these placed under testcases/kernel/syscalls/) use runtest/syscalls file, for kernel related tests for memory management we have runtest/mm, etc.
Important
|
The runtest files should have one entry per a test. Creating a wrapper that runs all your tests and adding it as a single test into runtest file is strongly discouraged. |
If your test needs datafiles to work, these should be put into a subdirectory named datafiles and installed into the testcases/data/$TCID directory (to do that you have to add INSTALL_DIR := testcases/data/TCID into the datafiles/Makefile).
You can obtain path to datafiles via $TST_DATAROOT provided by test.sh $TST_DATAROOT/… or via C function tst_dataroot() provided by libltp:
const char *dataroot = tst_dataroot();
Datafiles can also be accessed as $LTPROOT/testcases/data/$TCID/…, but $TST_DATAROOT and tst_dataroot() are preferred as these can be used when running testcases directly in git tree as well as from install location.
The path is constructed according to these rules:
-
if $LTPROOT is set, return $LTPROOT/testcases/data/$TCID
-
else if tst_tmpdir() was called return $STARTWD/datafiles (where $STARTWD is initial working directory as recorded by tst_tmpdir())
-
else return $CWD/datafiles
See testcases/commands/file/ for example.
If you test needs to execute a binary, place it in the same directory as the testcase and name the file starting with ${test_binary_name}_. Once the test is executed by the framework, the path to the directory with all LTP binaries is added to the $PATH and you can execute it just by its name.
Tip
|
If you need to execute such test from the LTP tree, you can add path to current directory to $PATH manually with: PATH="$PATH:$PWD" ./foo01. |
Note
|
See also Maintainer Patch Review Checklist. |
-
Test compiles and runs fine (check with
-i 10
too) -
make check
does not emit any warnings for the test you are working on (hint: run it in the test’s directory and/or usemake check-$TCID
) -
The runtest entries are in place
-
Test binaries are added into corresponding .gitignore files
-
Patches apply over the latest git
There are numerous .gitignore files in the LTP tree. Usually there is a .gitignore file per a group of tests. The reason for this setup is simple. It’s easier to maintain a .gitignore file per directory with tests, rather than having single file in the project root directory. This way, we don’t have to update all the gitignore files when moving directories, and they get deleted automatically when a directory with tests is removed.