README.config

Configuring your tests

You will set up a top level config.py containing a python
dict 'opts'  which can contain an entry for each phase of testing.

You may also specifiy job-level configuration in a file config.py
containing a dict jobopts that in all other respects has the
same structure as the top level configuration.

JOB PHASES

These two phases are outside of any test suite job.

prep    -- run once on hosts before any other setup, before
           the test suite is executed
cleanup -- run on hosts after the entire test suite is complete

These phases are done once in each test suite job.

data    -- data files to be put into place on hosts before
           each test job in the suite
main    -- the scripts or programs to be tested
collect -- collect test output from the hosts
           after each test job; this stanza contains default
           attributes for the per job collct scripts
verify  -- verify the output from the hosts
           after each test job; this stanza contains default
           attributes for the per job verify scripts
mod     -- scripts that modify the data or files on the hosts,
           run before a given job; this stanza contains default
           attributes for the per job mod scripts

The following two phases must be present in the top level config
and must not be present in the per job config, since they are run
before and after the entire suite of jobs has run: prep, cleanup

The following phases may have stanzas in the top level config or
in the job level config or both: data, mod, main, collect, verify

For each section, the script or filename will be specified,
and a set of attributes provided for it.

Here is an example section:

jobopts = {
     ...
    'main': {
        'rsync-dump.py': {
            'remote': '/usr/local/bin/',
            'hosts': [ 'dataset2', 'dataset1001' ],
            'execute': 'python'
        }
    },
    ...
}

This tells us that in the 'main' phase for the specific job,
there will be one script tested, 'rsync-dump.py'.
It will be copied into /usr/local/bin/ on the hosts
(containers) names dataset2 and dataset1001.
It will be executed, by invoking python <scriptname>.

Additional attributes:
'untar' -- True if a file to be copied to remote host needs
	   to be untarred after copying

FIXME there are more attributes, document them here.

DEFAULTS

All files listed in a stanza for a job-level configuration
or in the top level configuration will be used (run or
installed or whatever else needs to be done with them), if
they are present in the appropriate subdirectory for
that stanza in tests/jobs.

If an attribute is not specified for a file in th job level
directory, it will be taken from the top level config,
if a declaration is present for the file there; if neither
of these holds but there is a wildcard entry, i.e. an entry
for '*' in the stanza in the top level file, the attribute
value will be taken from there if given.

mod, data, main, collect and verify are done per job, so the
entry for e.g. 'mod' in the top level config.py contains defaults
that apply to all mods. Below is an example of a wild card stanza
and a top level attribute definition for a specific script.

    'mod': {
        '*' {
            'remote': '/root/'
        },
        'mod-1.sh': {
            'execute': 'bash'
        }
    }

This says, for all mods where these attributes are not specified
in a job-specific config file, copy them into the /root directory
on hosts (containers) dataset2 and dataset1001.  And for every job
that runs mod-1.sh, use the bash interpreter to run it.

Addtionally you can add a section 'default' and put attributes there
which will apply to all items in all stanzas if they are
not otherwise specified.

A good example of an attribute that could go into 'default'
is 'hosts', which might be the same set of hosts for copying
data, for mods, for running the script to be tested, etc.

If you have specified an attribute value for an item in the top level
config and you don't want that attribute present at all in the job
level config, you can set the value of the attribute to None in the
job level config, and it will be treated as absent.

GLOBALS

A section 'globals' should be defined in the top level config
which will contain entries for the following:
'password' -- the root password to all the docker containers
'tmp' -- directory on the remote hosts where files can be
    temporarily copied; for example, data files before they are untarred

EXAMPLES

See the file config.py.sample for an example top-level config, and
job.py.sample for an example per job config.