README.ParameterVariation

This document "briefly" describes what the parameter variation
facility does.

*How this works*

     * -C <filename> with common CLI loads configuration file.  Common
     * CLI runs test cases with each permutation of parameters as
       defined in the configuration.

*What the test module needs to implement:

-- In argument handler

   * Example: test module declares that test "Test X" takes 3 parameters
(integer, floating point and string). Order will be preserved (and
same as in the config file). The second parameter must be one of the
parameters defined in the configuration file. The third parameter in
each pair is a default, used when config file is not supplied by user.

    int ret;
    ret = blts_config_test_declare_params("Test X",
        test_x_arg_handler,
        CONFIG_PARAM_INT, "cats", 1,
        CONFIG_PARAM_FLOAT, "dogs", 42.0,
        CONFIG_PARAM_STRING, "word", "foo",
        CONFIG_PARAM_NONE); /* <-- ends variable argument list */
    if (ret)
        BLTS_ERROR("Internal error.\n");

   * From this point on, any tests with the given name can get
parameters as defined above. If the user didn't provide a
configuration file, the defaults are used.

   * Tests that aren't defined in the configuration file still get their
parameter values from the file when appropriately named parameters are
found. Default values are used for missing parameters. Note that
the parameters declared in [test] sections are positional, and if a test
is declared that way, all parameters must be listed.

   * Parameter generation can be done in user function; use the
*generate* keyword in parameter definition. You'll need to define the
handler for these:

    ret = blts_config_register_generating_func("func_name_in_config", function);

   * The registered function will get called when needed with parameters
generated from those named in the configuration. It must return a
boxed value (see below) using the parameters. Typical use includes
eg. generating strings with content varying according to several
parameters (see the example).

-- Passing parameters to test case

* Test module implements a function which gets called for each
iteration with the varying parameters. The function should return the
user data structure that gets passed to the test case, using the
parameters supplied. Example:

    void *my_test_arg_handler(struct boxed_value *args, void *user_data)
    {
        /* this is the structure we receive as a parameter in test cases */
        struct my_data_struct *data = (struct my_data_struct *) user_data;

        data->cats = blts_config_boxed_value_get_int(args);

        /* ... */
        return data;
    }



*Boxed values*

   * The variation engine mostly uses "boxed" values for passing around
data. These are type-agnostic values that also have a link field for
easy list building.

* Available types: int, long, float, double, char* (nul-terminated
* string) , boolean (== int)

Boxing and unboxing:

   *  blts_config_boxed_value_new_* ()
   *  blts_config_boxed_value_get_* ()

Utilities:

   * blts_config_boxed_value_free ()
   * blts_config_boxed_value_dup ()

Note: strdup() a string from ..get_string() if you wish to modify it
      if you got the value from elsewhere.

Note 2: ..new_string() makes a copy of the passed string,
      ..new_string_take() assumes it doesn't need to. 

Note 3: Values are type-checked on unboxing, with assertions on
      mismatches.

   * Lists are formed with the value->next pointer, with null
terminator.

List handling:

   * blts_config_boxed_value_list_reverse ()
   * blts_config_boxed_value_list_dup ()

Note: .._free() returns a pointer to the next list element, or null if
      list tail was reached.




*More Complex Example*

Configuration file
----------------------------------------------------------------------------
# comment
[globals]
	glob1 3
[end_globals]

[parameter]
	name foo
	const 1 2 glob1
[end_parameter]

[parameter]
	name bar
	range double 4.0 6.0
[end_parameter]

[parameter]
	name baz
	const "kissa"
[end_parameter]

[parameter]
	name zzz
	generate stringgen baz foo
[end_parameter]

[pgroup]
	name quux
	bar baz
[end_pgroup]

[test]
	name "Test 1"
	params foo
	fix bar 99.0
	pgroup quux
[end_test]

[test]
	name "Test 2"
	params zzz
[end_test]
----------------------------------------------------------------------------

* (Exercise: Can you figure out what parameter variations each test
  gets without looking at the code?)


Test code
----------------------------------------------------------------------------
#include <blts_params.h>

#include <blts_cli_frontend.h>
#include <blts_log.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

struct my_test_data {
	int foo;
	double bar;
	char* baz;
};

static void my_help(const char* base)
{
	fprintf(stdout, base, "helpful text","");
}

struct boxed_value *stringgen(struct boxed_value *arg_list)
{
	char *seed = blts_config_boxed_value_get_string(arg_list);
	unsigned segment_len = strlen(seed);
	arg_list = arg_list->next;
	unsigned len = blts_config_boxed_value_get_int(arg_list) * strlen(seed) + 1;
	char *str = malloc(len);
	for (unsigned i = 0; i < len-1; ++i) {
		str[i] = seed[i % segment_len];
	}
	str[len-1] = 0;
	struct boxed_value *ret = blts_config_boxed_value_new_string(str);
	free(str);
	return ret;
}

static int testcase(void* user_ptr, int test_num)
{
	struct my_test_data *td;
	LOG("tst: in testcase\n");
	assert(user_ptr);
	td = (struct my_test_data *) user_ptr;

	LOG("tst: Params got: %d, %lf, %s\n",td->foo, td->bar, td->baz);

	return (td->foo > 2);
}

static int testcase2(void* user_ptr, int test_num)
{
	struct my_test_data *td;
	LOG("tst: in testcase\n");
	assert(user_ptr);
	td = (struct my_test_data *) user_ptr;

	LOG("tst: Params got: %s\n", td->baz);

	return 0;
}

static blts_cli_testcase my_cases[] =
{
	{ "Test 1", testcase, 2000 },
	{ "Test 2", testcase2, 2000 },
	BLTS_CLI_END_OF_LIST
};

static void *test_arg_handler(struct boxed_value *args, void* p)
{
	struct my_test_data *td;
	assert(p);
	td = (struct my_test_data *) p;
	td->foo = blts_config_boxed_value_get_int(args);
	args = args->next;
	td->bar = blts_config_boxed_value_get_double(args);
	args = args->next;
	td->baz = strdup(blts_config_boxed_value_get_string(args));

	return td;
}

static void *test2_arg_handler(struct boxed_value *args, void* p)
{
	struct my_test_data *td;
	assert(p);
	td = (struct my_test_data *) p;
	td->baz = strdup(blts_config_boxed_value_get_string(args));

	return td;
}

static void *my_arg_handler(int argc, char **argv)
{
	int ret;
	struct my_test_data *td;
	struct variant_list *param_variants;
	LOG("tst: Arg handler\n");

	ret = blts_config_declare_variable_test("Test 1", test_arg_handler,
		CONFIG_PARAM_INT, "foo", 1,
		CONFIG_PARAM_DOUBLE, "bar", 42.0,
		CONFIG_PARAM_STRING, "baz", "kissa",
		CONFIG_PARAM_NONE);

	ret |= blts_config_declare_variable_test("Test 2", test2_arg_handler,
		CONFIG_PARAM_STRING, "zzz", "default",
		CONFIG_PARAM_NONE);

	ret |= blts_config_register_generating_func("stringgen", stringgen);
	if (ret) {
		BLTS_ERROR("!! Test declaration failed\n");
	}
	td = malloc(sizeof *td);
	memset(td, 0, sizeof *td);
	return td;
}

static blts_cli my_cli =
{
	.test_cases = my_cases,
	.log_file = "test.txt",
	.blts_cli_help = my_help,
	.blts_cli_process_arguments = my_arg_handler,
	.blts_cli_teardown = 0
};

int main(int argc, char **argv)
{
 	int ret;
	ret = blts_cli_main(&my_cli, argc, argv);
	if (ret)
		printf("tst: All passed\n");
	else
		printf("tst: Some failed\n");
	return ret;
}
----------------------------------------------------------------------------

h2. *Config file syntax:*

   * Note that some functionality mentioned here is not yet implemented

----------------------------------------------------------------------------
# <- comment is # to end of line
# line-oriented format
# [section] .. [end_section]
# 	    whitespace   is  ignored
# ID, <ident> == C identifier
# <str> == quoted string
# <something> == mandatory parameter
# <val> == C int / float; boolean (true/false/on/off/..?); quoted string; global variable


# Globals contain static values; referred later with given identifier.
# The "include" directive reads in a file with similar definitions (format??).
[globals]
	include <filename>         # global settings read from here
	ID <val>                   # global variable with constant value
	ID <val>
	# ...
[end_globals]


# Parameter definition.
[parameter]
	name <ident>				# parameter id, unique

	# one of
	const <val> <val> <val> #...		# list of possible values

	range <type> <num> <num>         	# arbitrary numeric range,
	      	     	   			# type="int"/"float"

	# boundary <type> <num> <num> # ...       # ? generalised range, values between endpoints
	  	   	  	      		  #   specify boundaries

	generate <gen> <param> <param>	#...	# call function registered by test
		  				# module as <gen>, with arguments
						# generated from given parameters
	# more? ...
[end_parameter]


# Parameter grouping; sugar for "params <list>" in test definitions (is this needed?)
[pgroup]		# "parameter group"
	name <ident>    # unique
	ID ID #...  	# parameter ids
[end_pgroup]
[pgroup]
	#...
[end_pgroup]
#...


# Test definitions; each produces 1-n test case variants.
# 0-n:
[test]
	name <str>			# unique, ref to definition in code
	# 0 or more of:
	params <id-list> 		# all parameters test knows about (optional)

	pgroup <pgroup-id>   		# use parameters from named group (optional)

	fix <param-id> <val>		# fix value of this parameter as <val> (optional)

	# more? (many extensions possible here) ...
[end_test]
[test]
	# ...
[end_test]
[test]
	# ...
[end_test]
# ...


# !! Testsets are not implemented yet: !!
# Combinations of tests. Parameter set is union of sets from each test
# (restrictions -> first wins?).
# 0-n:
[testset]
	name <str>                      # unique

	# 0-n:
	fix <param-id> <val>            # as with [test], but overrides all in set

	# one of :
	serial <test> <test> #... 	# run given tests in order

	parallel <test> <test> # ...    # run given tests in parallel (different processes)

	threads <test> <test> # ...     # parallel, but threaded in same process

	# more? (many extensions possible here) ...
[end_testset]
[testset]
	# ...
[end_testset]
# ...