forked from linux-test-project/ltp
-
Notifications
You must be signed in to change notification settings - Fork 1
/
test-writing-guidelines.txt
1903 lines (1416 loc) · 64.4 KB
/
test-writing-guidelines.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
LTP Test Writing Guidelines
===========================
This document describes LTP guidelines and LTP test interface and is intended
for anybody who want to write or modify a LTP testcase. It's not a definitive
guide and it's not, by any means, a substitute for common sense.
1. General Rules
----------------
1.1 Simplicity
~~~~~~~~~~~~~~
For all it's worth keep the testcases simple or better as simple as possible.
The kernel and libc are tricky beasts and the complexity imposed by their
interfaces is quite high. Concentrate on the interface you want to test and
follow the UNIX philosophy. It's a good idea to make the test as
self-contained as possible too (it should not depend on tools or libraries
that are not widely available).
Do not reinvent the wheel!
* Use LTP standard interface
* Do not add custom PASS/FAIL reporting functions
* Do not write Makefiles from scratch,
use LTP build system instead, etc.
* ...
1.2 Code duplication
~~~~~~~~~~~~~~~~~~~~
Copy & paste is a good servant but very poor master. If you are about to copy a
large part of the code from one testcase to another, think what would happen if
you find bug in the code that has been copied all around the tree. What about
moving it to a library instead?
The same goes for short but complicated parts, whenever you are about to copy &
paste a syscall wrapper that packs arguments accordingly to machine
architecture or similarly complicated code, put it into a header instead.
1.3 Coding style
~~~~~~~~~~~~~~~~
1.3.1 C coding style
^^^^^^^^^^^^^^^^^^^^
LTP adopted Linux kernel coding style. If you aren't familiar with its rules
locate 'linux/Documentation/CodingStyle' in the kernel sources and read it,
it's a well written introduction.
There is also a checkpatch (see 'linux/scripts/checkpatch.pl') script that can
be used to check your patches before the submission.
NOTE: If checkpatch does not report any problems, the code still may be wrong
as the tool only looks for common mistakes.
1.3.2 Shell coding style
^^^^^^^^^^^^^^^^^^^^^^^^
When writing testcases in shell write in *portable shell* only, it's a good
idea to try to run the test using alternative shell (alternative to bash, for
example dash) too.
*Portable shell* means Shell Command Language as defined by POSIX with a
exception of few widely used extensions, namely 'local' keyword used inside of
functions and '-o' and '-a' test parameters (that are marked as obsolete in
POSIX).
You can either try to run the testcases on Debian which has '/bin/sh' pointing
to 'dash' by default or install 'dash' on your favorite distribution and use
it to run the tests. If your distribution lacks 'dash' package you can always
compile it from http://gondor.apana.org.au/~herbert/dash/files/[source].
Debian also has nice devscript
https://anonscm.debian.org/cgit/collab-maint/devscripts.git/tree/scripts/checkbashisms.pl[checkbashism.pl]
that can be used to check for non-portable shell code.
Here are some common sense style rules for shell
* Keep lines under 80 chars
* Use tabs for indentation
* Keep things simple, avoid unnecessary subshells
* Don't do confusing things (i.e. don't name your functions like common shell
commands, etc.)
* Quote variables
* Be consistent
1.4 Commenting code
~~~~~~~~~~~~~~~~~~~
Comments can sometimes save you day but they can easily do more harm than
good. There has been several cases where comments and actual implementation
were drifting slowly apart which yielded into API misuses and hard to find
bugs. Remember there is only one thing worse than no documentation, wrong
documentation.
Generally everybody should write code that is obvious (which unfortunately
isn't always possible). If there is a code that needs to be commented keep it
short and to the point. Never ever comment the obvious.
In case of LTP testcases it's customary to add a paragraph with highlevel test
description somewhere at the beginning of the file (usually right under the GPL
header). This helps other people to understand the overall goal of the test
before they dive into the technical details.
1.5 Backwards compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~
LTP test should be as backward compatible as possible. Think of an enterprise
distributions with long term support (more than five years since the initial
release) or of an embedded platform that needs to use several years old
toolchain supplied by the manufacturer.
Therefore LTP test for more current features should be able to cope with older
systems. It should at least compile fine and if it's not appropriate for the
configuration it should return 'TCONF' (see test interface description below).
There are several types of checks we use:
The *configure script* is usually used to detect availability of a function
declarations in system headers. It's used to disable tests at compile time.
We also have runtime kernel version detection that can be used to disable
tests at runtime.
Checking the *errno* value is another type of runtime check. Most of the
syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
or was disabled upon kernel compilation.
Sometimes it also makes sense to define a few macros instead of creating
configure test. One example are Linux specific POSIX clock ids in
'include/lapi/posix_clocks.h'.
1.6 Dealing with messed up legacy code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LTP contains a lot of old and messy code and we are cleaning it up as fast as
we can but despite the efforts there is still a lot. If you start modifying
old or a messed up testcase and your changes are more complicated than simple
typo fixes you should do a cleanup first (in a separate patch). It's easier to
review the changes if you separate the formatting fixes from the changes that
affects the test behavior.
The same goes for moving files. If you need a rename or move file do it in a
separate patch.
1.7 License
~~~~~~~~~~~
Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
any later version).
2. Writing a testcase
---------------------
2.1 LTP Structure
~~~~~~~~~~~~~~~~~
The structure of LTP is quite simple. Each test is a binary written either in
portable shell or C. The test gets a configuration via environment variables
and/or command line parameters, it prints additional information into the
stdout and reports overall success/failure via the exit value.
Tests are generally placed under the 'testcases/' directory. Everything that
is a syscall or (slightly confusingly) libc syscall wrapper goes under
'testcases/kernel/syscalls/'. Then there is 'testcases/open_posix_testsuite'
which is a well maintained fork of the upstream project that has been dead
since 2005 and also a number of directories with tests for more specific
features.
2.1.1 Runtest Files
^^^^^^^^^^^^^^^^^^^
The list of tests to be executed is stored in runtest files under the
'runtest/' directory. The default set of runtest files to be executed is
stored in 'scenario_groups/default'. When you add a test you should add
corresponding entries into some runtest file(s) as well.
For syscall tests (these placed under 'testcases/kernel/syscalls/') use
'runtest/syscalls' file, for kernel related tests for memory management we
have 'runtest/mm', etc.
IMPORTANT: The runtest files should have one entry per a test. Creating a
wrapper that runs all your tests and adding it as a single test
into runtest file is strongly discouraged.
2.1.2 Datafiles
^^^^^^^^^^^^^^^
If your test needs datafiles to work, these should be put into a subdirectory
named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
'datafiles/Makefile').
You can obtain path to datafiles via $TST_DATAROOT provided by test.sh
'$TST_DATAROOT/...'
or via C function 'tst_dataroot()' provided by libltp:
[source,c]
-------------------------------------------------------------------------------
const char *dataroot = tst_dataroot();
-------------------------------------------------------------------------------
Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used
when running testcases directly in git tree as well as from install
location.
The path is constructed according to these rules:
1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
(where '$STARTWD' is initial working directory as recorded by 'tst_tmdir()')
3. else return '$CWD/datafiles'
See 'testcases/commands/file/' for example.
2.1.3 Subexecutables
^^^^^^^^^^^^^^^^^^^^
If you test needs to execute a binary, place it in the same directory as the
testcase and name the file starting with testname_ ('.tid' see below).
Once the test is executed by the framework, the path to the directory with all
LTP binaries is added to the '$PATH' and you can execute it just by its name.
TIP: If you need to execute such test from the LTP tree, you can add path to
current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.
2.2 Writing a test in C
~~~~~~~~~~~~~~~~~~~~~~~
2.2.1 Basic test structure
^^^^^^^^^^^^^^^^^^^^^^^^^^
Let's start with an example, following code is a simple test for a 'getenv()'.
[source,c]
-------------------------------------------------------------------------------
/*
* This is test for basic functionality of getenv().
*
* - create an env variable and verify that getenv() can get get it
* - call getenv() with nonexisting variable name, check that it returns NULL
*/
#include "tst_test.h"
#define ENV1 "LTP_TEST_ENV"
#define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST"
#define ENV_VAL "val"
static void setup(void)
{
if (setenv(ENV1, ENV_VAL, 1))
tst_brk(TBROK | TERRNO, "setenv() failed");
}
static void test(void)
{
char *ret;
ret = getenv(ENV1);
if (!ret) {
tst_res(TFAIL, "getenv(" ENV1 ") = NULL");
goto next;
}
if (!strcmp(ret, ENV_VAL)) {
tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'");
} else {
tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '"
ENV_VAL "'", ret);
}
next:
ret = getenv(ENV2);
if (ret)
tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret);
else
tst_res(TPASS, "getenv(" ENV2 ") = NULL");
}
static struct tst_test test = {
.tid = "getenv01",
.test_all = test,
.setup = setup,
};
-------------------------------------------------------------------------------
Each test includes the 'tst_test.h' header and must define the 'struct
tst_test test' structure.
The '.tid' defines test name (usually syscall/libcall name + number). The name
is used in the test results as well as a base for temporary directory name if
temporary directory is needed. In most of the cases it's the same as test
filename without the extension.
The overall test initialization is done in the 'setup()' function.
The overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is
omitted as the test does not have anything to clean up. If cleanup is set in
the test structure it's called on test exit just before the test library
cleanup. That especially means that cleanup can be called at any point in a
test execution. For example even when a test setup step has failed, therefore
the 'cleanup()' function must be able to cope with unfinished initialization,
and so on.
The test itself is done in the 'test()' function. The test function must work
fine if called in a loop.
There are two types of a test function pointers in the test structure. The
first one is a '.test_all' pointer that is used when test is implemented as a
single function. Then there is a '.test' function along with the number of
tests '.tcnt' that allows for more detailed result reporting. If the '.test'
pointer is set the function is called '.tcnt' times with an integer parameter
in range of [0, '.tcnt' - 1].
IMPORTANT: Only one of '.test' and '.test_all' can be set at a time.
Each test has a default timeout set to 300s. The default timeout can be
overriden by setting '.timeout' in the test structure or by calling
'tst_set_timeout()' in the test 'setup()'.
A word about the cleanup() callback
+++++++++++++++++++++++++++++++++++
There are a few rules that needs to be followed in order to write correct
cleanup() callback.
1. Free only resources that were initialized. Keep in mind that callback can
be executed at any point in the test run.
2. Make sure to free resources in the reverse order they were
initialized. (Some of the steps may not depend on others and everything
will work if there were swapped but let's keep it in order.)
3. Avoid using SAFE_MACROS() in cleanup if you want the cleanup to carry on
when a cleanup step has failed.
The first rule may seem complicated at first however, on the contrary, it's
quite easy. All you have to do is to keep track of what was already
initialized. For example file descriptors needs to be closed only if they were
assigned a valid file descriptor. For most of the things you need to create
extra flag that is set right after successful initialization though. Consider,
for example, test setup below.
[source,c]
-------------------------------------------------------------------------------
static int fd0, fd1, mount_flag;
#define MNTPOINT "mntpoint"
#define FILE1 "mntpoint/file1"
#define FILE2 "mntpoint/file2"
static void setup(void)
{
SAFE_MKDIR(MNTPOINT, 0777);
SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0);
mount_flag = 1;
fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666);
fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666);
}
-------------------------------------------------------------------------------
In this case the 'cleanup()' function may be invoked when any of the 'SAFE_*'
macros has failed and therefore must be able to work with unfinished
initialization as well. Since global variables are initialized to zero we can
just check that fd > 0 before we attempt to close it. The mount function
requires extra flag to be set after device was successfully mounted.
[source,c]
-------------------------------------------------------------------------------
static void cleanup(void)
{
if (fd1 > 0 && close(fd1))
tst_res(TWARN | TERRNO, "close(fd1)");
if (fd0 > 0 && close(fd0))
tst_res(TWARN | TERRNO, "close(fd0)");
if (mount_flag && tst_umouont(MNTPOINT))
tst_res(TBROK | TERRNO, "umount(%s)", MNTPOINT);
}
-------------------------------------------------------------------------------
NOTE: Creation and removal of the test temporary directory is handled in
the test library and the directory is removed recursively. Therefore
we do not have to remove files and directories in the test cleanup.
2.2.2 Basic test interface
^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
void tst_res(int ttype, char *arg_fmt, ...);
-------------------------------------------------------------------------------
Printf-like function to report test result, it's mostly used with ttype:
|==============================
| 'TPASS' | Test has passed.
| 'TFAIL' | Test has failed.
| 'TINFO' | General message.
|==============================
The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TEST_ERRNO' respectively.
[source,c]
-------------------------------------------------------------------------------
void tst_brk(int ttype, char *arg_fmt, ...);
-------------------------------------------------------------------------------
Printf-like function to report error and exit the test, it can be used with ttype:
|============================================================
| 'TBROK' | Something has failed in test preparation phase.
| 'TCONF' | Test is not appropriate for current configuration
(syscall not implemented, unsupported arch, ...)
|============================================================
The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TEST_ERRNO' respectively.
[source,c]
-------------------------------------------------------------------------------
const char *tst_strsig(int sig);
-------------------------------------------------------------------------------
Return the given signal number's corresponding string.
[source,c]
-------------------------------------------------------------------------------
const char *tst_strerrno(int err);
-------------------------------------------------------------------------------
Return the given errno number's corresponding string. Using this function to
translate 'errno' values to strings is preferred. You should not use the
'strerror()' function in the testcases.
[source,c]
-------------------------------------------------------------------------------
void tst_set_timeout(unsigned int timeout);
-------------------------------------------------------------------------------
Allows for setting timeout per test iteration dymanically in the test setup(),
the timeout is specified in seconds.
2.2.3 Test temporary directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If '.needs_tmpdir' is set to '1' in the 'struct tst_test' unique test
temporary is created and it's set as the test working directory. Tests *MUST
NOT* create temporary files outside that directory.
IMPORTANT: Close all file descriptors (that point to files in test temporary
directory, even the unlinked ones) either in the 'test()' function
or in the test 'cleanup()' otherwise the test may break temporary
directory removal on NFS (look for "NFS silly rename").
[[2.2.4]]
2.2.4 Safe macros
^^^^^^^^^^^^^^^^^
Safe macros aim to simplify error checking in test preparation. Instead of
calling system API functions, checking for their return value and aborting the
test if the operation has failed, you just use corresponding safe macro.
Use them whenever it's possible.
Instead of writing:
[source,c]
-------------------------------------------------------------------------------
fd = open("/dev/null", O_RDONLY);
if (fd < 0)
tst_brk(TBROK | TERRNO, "opening /dev/null failed");
-------------------------------------------------------------------------------
You write just:
[source,c]
-------------------------------------------------------------------------------
fd = SAFE_OPEN("/dev/null", O_RDONLY);
-------------------------------------------------------------------------------
IMPORTANT: The SAFE_CLOSE() function also sets the passed file descriptor to -1
after it's successfully closed.
They can also simplify reading and writing of sysfs files, you can, for
example, do:
[source,c]
-------------------------------------------------------------------------------
SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max);
-------------------------------------------------------------------------------
See 'include/tst_safe_macros.h', 'include/tst_safe_stdio.h' and
'include/tst_safe_file_ops.h' and 'include/tst_safe_net.h' for a complete list.
2.2.5 Test specific command line options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
struct tst_option {
char *optstr;
char **arg;
char *help;
};
-------------------------------------------------------------------------------
Test specific command line parameters can be passed with the 'NULL'-terminated
array of 'struct tst_option'. The 'optstr' is the command line option i.e. "o"
or "o:" if option has a parameter. Only short options are supported. The 'arg'
is where 'optarg' is stored upon match. If option has no parameter it's set to
non-'NULL' value if option was present. The 'help' is a short help string.
NOTE: The test parameters must not collide with common test parameters defined
in the library the currently used ones are +-i+, +-I+, +-C+, and +-h+.
[source,c]
-------------------------------------------------------------------------------
int tst_parse_int(const char *str, int *val, int min, int max);
int tst_parse_float(const char *str, float *val, float min, float max);
-------------------------------------------------------------------------------
Helpers for parsing the the strings returned in the 'struct tst_option'.
Both return zero on success and 'errno', mostly 'EINVAL' or 'ERANGE', on
failure.
Both functions are no-op if 'str' is 'NULL'.
The valid range for result includes both 'min' and 'max'.
.Example Usage
[source,c]
-------------------------------------------------------------------------------
#include <limits.h>
#include "tst_test.h"
static char *str_threads;
static int threads = 10;
static struct tst_option options[] = {
{"t:", &str_threads, "Number of threads (default 10)"},
...
{NULL, NULL, NULL}
};
static void setup(void)
{
if (tst_parse_int(str_threads, &threads, 1, INT_MAX))
tst_brk(TBROK, "Invalid number of threads '%s'", str_threads);
...
}
static void test_threads(void)
{
...
for (i = 0; i < threads; i++) {
...
}
...
}
static struct tst_test test = {
...
.options = options,
...
};
-------------------------------------------------------------------------------
2.2.6 Runtime kernel version detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Testcases for newly added kernel functionality require kernel newer than a
certain version to run. All you need to skip a test on older kernels is to
set the '.min_kver' string in the 'struct tst_test' to a minimal required
kernel version, e.g. '.min_kver = "2.6.30"'.
For more complicated operations such as skipping a test for a certain range
of kernel versions, following functions could be used:
[source,c]
-------------------------------------------------------------------------------
int tst_kvercmp(int r1, int r2, int r3);
struct tst_kern_exv {
char *dist_name;
char *extra_ver;
};
int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers);
-------------------------------------------------------------------------------
These two functions are intended for runtime kernel version detection. They
parse the output from 'uname()' and compare it to the passed values.
The return value is similar to the 'strcmp()' function, i.e. zero means equal,
negative value means that the kernel is older than than the expected value and
positive means that it's newer.
The second function 'tst_kvercmp2()' allows for specifying per-vendor table of
kernel versions as vendors typically backport fixes to their kernels and the
test may be relevant even if the kernel version does not suggests so. See
'testcases/kernel/syscalls/inotify/inotify04.c' for example usage.
WARNING: The shell 'tst_kvercmp' maps the result into unsigned integer - the
process exit value.
2.2.7 Fork()-ing
^^^^^^^^^^^^^^^^
Be wary that if the test forks and there were messages printed by the
'tst_*()' interfaces, the data may still be in libc/kernel buffers and these
*ARE NOT* flushed automatically.
This happens when 'stdout' gets redirected to a file. In this case, the
'stdout' is not line buffered, but block buffered. Hence after a fork content
of the buffers will be printed by the parent and each of the children.
To avoid that you should use 'SAFE_FORK()'.
IMPORTANT: You have to set the '.forks_child' flag in the test structure
if your testcase forks.
[[2.2.8]]
2.2.8 Doing the test in the child process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Results reported by 'tst_res()' are propagated to the parent test process via
block of shared memory.
Calling 'tst_brk()' causes child process to exit with non-zero exit value.
Which means that it's safe to use 'SAFE_*()' macros in the child processes as
well.
Children that outlive the 'test()' function execution are waited for in the
test library. Unclean child exit (killed by signal, non-zero exit value, etc.)
will cause the main test process to exit with 'tst_brk()', which especially
means that 'TBROK' propagated from a child process will cause the whole test
to exit with 'TBROK'.
If a test needs a child that segfaults or does anything else that cause it to
exit uncleanly all you need to do is to wait for such children from the
'test()' function so that it's reaped before the main test exits the 'test()'
function.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
void tst_reap_children(void);
-------------------------------------------------------------------------------
The 'tst_reap_children()' function makes the process wait for all of its
children and exits with 'tst_brk(TBROK, ...)' if any of them returned
a non zero exit code.
2.2.9 Fork() and Parent-child synchronization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As LTP tests are written for Linux, most of the tests involve fork()-ing and
parent-child process synchronization. LTP includes a checkpoint library that
provides wait/wake futex based functions.
In order to use checkpoints the '.needs_checkpoints' flag in the 'struct
tst_test' must be set to '1', this causes the test library to initialize
checkpoints before the 'test()' function is called.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
TST_CHECKPOINT_WAIT(id)
TST_CHECKPOINT_WAKE(id)
TST_CHECKPOINT_WAKE2(id, nr_wake)
TST_CHECKPOINT_WAKE_AND_WAIT(id)
-------------------------------------------------------------------------------
The checkpoint interface provides pair of wake and wait functions. The 'id' is
unsigned integer which specifies checkpoint to wake/wait for. As a matter of
fact it's an index to an array stored in a shared memory, so it starts on
'0' and there should be enough room for at least of hundred of them.
The 'TST_CHECKPOINT_WAIT()' suspends process execution until it's woken
up or until timeout is reached.
The 'TST_CHECKPOINT_WAKE()' wakes one process waiting on the checkpoint.
If no process is waiting the function retries until it success or until
timeout is reached.
If timeout has been reached process exits with appropriate error message (uses
'tst_brk()').
The 'TST_CHECKPOINT_WAKE2()' does the same as 'TST_CHECKPOINT_WAKE()' but can
be used to wake precisely 'nr_wake' processes.
The 'TST_CHECKPOINT_WAKE_AND_WAIT()' is a shorthand for doing wake and then
immediately waiting on the same checkpoint.
Child processes created via 'SAFE_FORK()' are ready to use the checkpoint
synchronization functions, as they inherited the mapped page automatically.
Child processes started via 'exec()', or any other processes not forked from
the test process must initialize the checkpoint by calling 'tst_reinit()'.
For the details of the interface, look into the 'include/tst_checkpoint.h'.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
/*
* Waits for process state change.
*
* The state is one of the following:
*
* R - process is running
* S - process is sleeping
* D - process sleeping uninterruptibly
* Z - zombie process
* T - process is traced
*/
TST_PROCESS_STATE_WAIT(pid, state)
-------------------------------------------------------------------------------
The 'TST_PROCESS_STATE_WAIT()' waits until process 'pid' is in requested
'state'. The call polls +/proc/pid/stat+ to get this information.
It's mostly used with state 'S' which means that process is sleeping in kernel
for example in 'pause()' or any other blocking syscall.
2.2.10 Signal handlers
^^^^^^^^^^^^^^^^^^^^^^
If you need to use signal handlers, keep the code short and simple. Don't
forget that the signal handler is called asynchronously and can interrupt the
code execution at any place.
This means that problems arise when global state is changed both from the test
code and signal handler, which will occasionally lead to:
* Data corruption (data gets into inconsistent state), this may happen, for
example, for any operations on 'FILE' objects.
* Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)',
etc. from both the test code and the signal handler at the same time since
'malloc' has global lock for it's internal data structures. (Be wary that
'malloc(2)' is used by the libc functions internally too.)
* Any other unreproducible and unexpected behavior.
Quite common mistake is to call 'exit(3)' from a signal handler. Note that this
function is not signal-async-safe as it flushes buffers, etc. If you need to
exit a test immediately from a signal handler use '_exit(2)' instead.
TIP: See 'man 7 signal' for the list of signal-async-safe functions.
If a signal handler sets a variable, its declaration must be 'volatile',
otherwise compiler may misoptimize the code. This is because the variable may
not be changed in the compiler code flow analysis. There is 'sig_atomic_t'
type defined in C99 but this one *DOES NOT* imply 'volatile' (it's just a
'typedef' to 'int'). So the correct type for a flag that is changed from a
signal handler is either 'volatile int' or 'volatile sig_atomic_t'.
2.2.11 Kernel Modules
^^^^^^^^^^^^^^^^^^^^^
There are certain cases where the test needs a kernel part and userspace part,
happily, LTP can build a kernel module and then insert it to the kernel on test
start for you. See 'testcases/kernel/device-drivers/block' for details.
2.2.11 Useful macros
^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
ARRAY_SIZE(arr)
-------------------------------------------------------------------------------
Returns the size of statically defined array, i.e.
'(sizeof(arr) / sizeof(*arr))'
[source,c]
-------------------------------------------------------------------------------
LTP_ALIGN(x, a)
-------------------------------------------------------------------------------
Aligns the x to be next multiple of a. The a must be power of 2.
2.2.12 Filesystem type detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests are known to fail on certain filesytems (you cannot swap on TMPFS,
there are unimplemented 'fcntl()' etc.).
If your test needs to be skipped on certain filesystems, use the interface
below:
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
/*
* Unsupported only on NFS.
*/
if (tst_fs_type(".") == TST_NFS_MAGIC)
tst_brk(TCONF, "Test not supported on NFS filesystem");
/*
* Unsupported on NFS, TMPFS and RAMFS
*/
long type;
switch ((type = tst_fs_type("."))) {
case TST_NFS_MAGIC:
case TST_TMPFS_MAGIC:
case TST_RAMFS_MAGIC:
tst_brk(TCONF, "Test not supported on %s filesystem",
tst_fs_type_name(type));
break;
}
-------------------------------------------------------------------------------
2.2.13 Thread-safety in the LTP library
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is safe to use library 'tst_res()' function in multi-threaded tests.
Only the main thread must return from the 'test()' function to the test
library and that must be done only after all threads that may call any library
function has been terminated. That especially means that threads that may call
'tst_brk()' must terminate before the execution of the 'test()' function
returns to the library. This is usually done by the main thread joining all
worker threads at the end of the 'test()' function. Note that the main thread
will never get to the library code in a case that 'tst_brk()' was called from
one of the threads since it will sleep at least in 'pthread_join()' on the
thread that called the 'tst_brk()' till 'exit()' is called by 'tst_brk()'.
The test-supplied cleanup function runs *concurrently* to the rest of the
threads in a case that cleanup was entered from 'tst_brk()'. Subsequent
threads entering 'tst_brk()' must be suspended or terminated at the start of
the the user supplied cleanup function. It may be necessary to stop or exit
the rest of the threads before the test cleans up as well. For example threads
that create new files should be stopped before temporary directory is be
removed.
Following code example shows thread safe cleanup function example using atomic
increment as a guard. The library calls its cleanup after the execution returns
from the user supplied cleanup and expects that only one thread returns from
the user supplied cleanup to the test library.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
static void cleanup(void)
{
static int flag;
if (tst_atomic_inc(&flag) != 1)
pthread_exit(NULL);
/* if needed stop the rest of the threads here */
...
/* then do cleanup work */
...
/* only one thread returns to the library */
}
-------------------------------------------------------------------------------
2.2.14 Testing with a block device
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests needs a block device (inotify tests, syscall 'EROFS' failures,
etc.). LTP library contains a code to prepare a testing device.
If '.needs_device' flag in the 'struct tst_test' is set the the 'tst_device'
structure is initialized with a path to a test device and default filesystem
to be used.
You can also request minimal device size in megabytes by setting
'.device_min_size' in the 'struct tst_test' structure. The device is
guaranteed to have at least the requested size then.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
struct tst_device {
const char *dev;
const char *fs_type;
};
extern struct tst_device *tst_device;
int tst_umount(const char *path);
-------------------------------------------------------------------------------
In case that 'LTP_DEV' is passed to the test in an environment, the library
checks that the file exists and that it's a block device, if
'.device_min_size' is set the device size is checked as well. If 'LTP_DEV'
wasn't set or if size requirements were not met a temporary file is created
and attached to a free loop device.
If there is no usable device and loop device couldn't be initialized the test
exits with 'TCONF'.
The 'tst_umount()' function works exactly as 'umount(2)' but retries several
times on 'EBUSY'. This is because various desktop daemons (gvfsd-trash is known
for that) may be stupid enough to probe all newly mounted filesystem which
results in 'umount(2)' failing with 'EBUSY'.
IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to
umount filesystems.
2.2.15 Formatting a device with a filesystem
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
static void setup(void)
{
...
SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL);
...
}
-------------------------------------------------------------------------------
This function takes a path to a device, filesystem type and an array of extra
options passed to mkfs.
The fs options 'fs_opts' should either be 'NULL' if there are none, or a
'NULL' terminated array of strings such as:
+const char *const opts[] = {"-b", "1024", NULL}+.
The extra option 'extra_opt' should either be 'NULL' if there is none, or a
string such as '"102400"'; 'extra_opt' will be passed after device name. e.g:
+mkfs -t ext4 -b 1024 /dev/sda1 102400+ in this case.
2.2.16 Verifying a filesystem's free space
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests have size requirements for the filesystem's free space. If these
requirements are not satisfied, the tests should be skipped.
[source,c]
-------------------------------------------------------------------------------
#include "tst_test.h"
int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult);
-------------------------------------------------------------------------------
The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if
there is not.
The 'path' is the pathname of any directory/file within a filesystem.
The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'.
The required free space is calculated by 'size * mult', e.g.
'tst_fs_has_free("/tmp/testfile", 64, TST_MB)' will return 1 if the
filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0
if not.
2.2.17 Files, directories and fs limits
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests need to know the maximum count of links to a regular file or
directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error.
[source,c]
-------------------------------------------------------------------------------