README.test

The MiNTLib now comes with a validation suite.  To perform the tests
cd into the top-level directory and type "make -k check".  You should
give the option "-k" (resp. "--keep-going") to "make" because some
tests are known to fail.

Most Makefiles in subdirectories also know about the target "check".
If you want to run only the tests for stdio simply cd into subdirectory
"stdio" and type "make check" there.

By default only the results of the tests are presented on standard
output.  Possible error messages printed to standard error also 
appear in the output (and mess it up).  If you want to see what is
really going on you have to set the environment variable "TEST_VERBOSE" 
to a non-empty value.  This will print the commands that are actually executed.
This allows you to run individual tests without the help of the Makefile
(sometimes it is desirable to do only one test from a larger suite
in a subdirectory).

Note that all tests check the functionality of the standard library
(i. e. 32 bit integers, no FPU code, plain m68000).  If you are mean
enough to check the other libraries you should also be smart enough
to change the file "checkrules".  This file is a Makefile fragment
that gets included for tests.  In order to avoid inconsistent 
compiles/links it is usually inevitable to run "make checkclean" whenever
you edit "checkrules".  The makefile target "checkclean" removes
all object files that are linked into the test programs plus the
test programs themselves.

To understand how the tests internally work you should follow an example.
Suppose there is a new string function "strfoo()" which should be
tested.  You write a little test program for it, name the source file
"test-strfoo.c".  The name of the test will simply be "strfoo".  Now
edit the Makefile in the subdirectory "string", look for a line starting 
with "TESTS =" and add your test ("strfoo") to it.  Note that the name
of the test (as displayed when running the suite) is "strfoo" whereas
the corresponding files are all named "test-strfoo.*"; all filenames 
beginning with the string "test-" should be considered special, they are
used for the tests.

Your program should exit with a zero exit code in case of success and
non-zero in case of failure.  You can write as much stuff as you want
to standard output but you should only write error messages to 
standard error.

If your program needs user input create another file "test-strfoo.input"
and write whatever you program should receive as standard input into
it.  If it should be started with options or arguments put it into
another file "test-strfoo.args".  The contents of it will be put on
the command line after "test-strfoo".

If your program produces something important on standard output you 
need another file "test-strfoo.expect".  If it exists the test suite
will compare the output of your program with the file "test-strfoo.expect".
In this case the test will fail either if the program itself exited with
non-zero or if the program's output and "test-strfoo.expect" differ.

If this is not enough you can also write a shell script "test-strfoo.sh".
If this file exists, instead of running the program file "test-strfoo" the
test suite will run the shell script.  Apart from that everything is the
same, i. e. you can still use the files "test-strfoo.args", "test-strfoo.input",
or "test-strfoo.expect".

Even if you use arguments, input, expected output or a wrapper shell
script you only have to add then name of your test to the variable
"TESTS" in the Makefile.  But if you think your test is important
you should also add the name of all needed files to the file
"EXTRAFILES", so that they will be included in a distribution.  In
fact the Makefile only calls the shell script "do_test" in the 
top-level directory with the names of the tests (the leading "test-" 
omitted) and the script is smart enough to figure out what you want
based on the existing "test-strfoo.*" files it finds.

When writing tests you may have a look at the source file "test-skeleton.c"
in the top-level directory.  Several tests include this file via

	#include "../test-skeleton.c"
	
so that they only have to define the bare test routine "do_test()".

If you find a bug in the MiNTLib it is probably a good idea to think
of some test case that will reveal the bug and send the test to me.
If the bug cannot be fixed the failed test will remind of it, if
it can be fixed it is assured that it will remain fixed in future
versions.

Saarbruecken/Germany, March 7, 2000

Guido Flohr <guido@freemint.de>