Combinatrix is a utility library to produce a full set of variations of a set of parameters, according to a provided set of rules for the values of those parameters, any inter-constraints/conditions on them.
It can take input in the form of a structured CSV or as a suitable JSON document. It can also be integrated directly with your Python
code by providing a dict with the appropriate configuration.
For example, if you are looking to generate an exhaustive set of paramters for parameterised testing, you can feed the sets of parameters and their constraints into Combinatrix, and it will output all parameter sets that should be fed to your tests.
For now, combinatrix is not on pypi, but you can install it in your requirements.txt as follows:
-e git+https://github.com/CottageLabs/combinatrix.git#egg=combinatrix
To use from the command line
usage: combinatrix [-h] [-c] [-j] [-m MATRIX] [-s SETTINGS] input
positional arguments:
input Source input file; can be either CSV or JSON, and
should end with either .csv or .json
optional arguments:
-h, --help show this help message and exit
-c, --csv if the input file does not end with .csv but is a CSV,
use this flag to force that format
-j, --json if the input file does not end with .csv but is a CSV,
use this flag to force that format
-m MATRIX, --matrix MATRIX
output file path for matrix. If omitted, the matrix
will not be generated
-s SETTINGS, --settings SETTINGS
output file path for JSON settings, if this is a
conversion from a CSV settings file
for example
combinatrix -c -m combos.csv -s myjson.json myinput.settings.csv
You can try this yourself straight away with some real data from this code repository, like this:
combinatrix -c -m my-first-combination.matrix.csv -j my-first-combination.settings.json docs/examples/example.csv
Fields are ordered in Combinatrix, primarily for the purposes of output display. When you provide the fields for combination, you
provide them in order. If you interact with the Combinatrix directly via the python API, and do not write your results out to CSV, they
you will be returned a list of dict objects which are not ordered.
For example, when provided with the following input
{
"parameters" : [
{
"name" : "field1"
},
{
"name" : "field2"
}
]
}The resulting CSV output would be ordered the same way:
field1,field2
...,...
There are three field types, which behave in different ways during combnination:
Fields of index type are not included in combinations. Instead they result in a unique number (within the context of the combination set)
which defines an arbitrary order for the output. The output will be provided in this order, whether it is the CSV output or the list of
dict results from the python API.
For example, when provided with the following input:
{
"paramters" : [
{
"name" : "id",
"type" : "index"
},
{
"name" : "field1"
}
]
}The CSV output would be
id,field1
1,...
2,...
3,etc.
Fields of generated type are included in the combinations. Each value for this type will be used in the combination, and its constraints
will be taken into account (see below for more information on constraints). This is the default field type. If a type parameter is omitted
from the settings, it will default to this.
For example, when provided with the following input:
{
"paramters" : [
{
"name" : "id",
"type" : "index"
},
{
"name" : "field1"
"type" : "generated",
"values" : {
"one" : {},
"two" : {}
}
}
]
}Will result in the following CSV output
id,field1
1,one
2,two
Fields of type conditional will not be included in the combinations initially, but will have a suitable value attached to each combination for which its supplied conditions (see below for more information) are satisfied.
This could be used, for example, to specify the expected output of a test, depending on the supplied arguments. A conditional field
may also supply a default value, for use when none of its conditions apply.
For example, given the following input:
{
"paramters" : [
{
"name" : "id",
"type" : "index"
},
{
"name" : "field1"
"type" : "generated",
"values" : {
"one" : {},
"two" : {}
}
},
{
"name" : "result",
"type" : "conditional",
"default" : "success"
"values" : {
"error" : {
"conditions" : [
{"field1" : ["one"]}
]
}
}
}
]
}The output CSV would be
id,field1,result
1,one,error
2,two,success
For generated and conditional types, you must specify the possible values.
For a generated type, this tells Combinatrix which values to use to build the full set of combinations.
For a conditional type, this tells Combinatrix which values are available, and can be used if their conditions are met.
For example, this document specifies a generated and a conditional field, which have the allowed values "one", "two" and "error"
respectively. Note that the conditional field can also have a default value:
{
"paramters" : [
{
"name" : "field1"
"type" : "generated",
"values" : {
"one" : {},
"two" : {}
}
},
{
"name" : "result",
"type" : "conditional",
"default" : "success"
"values" : {
"error" : {}
}
}
]
}Note also that each value is itself a key to a dictionary; within that dictionary we can place additional configuration for that value in that field, see below.
Without any constraints, Combinatrix will generate a full set of combinations for all values in all generated fields. So, for example, the following document provides two unconstrained fields:
{
"paramters" : [
{
"name" : "field1"
"type" : "generated",
"values" : {
"one" : {},
"two" : {}
}
},
{
"name" : "field2",
"type" : "generated",
"values" : {
"red" : {},
"blue" : {},
"green" : {}
}
}
]
}The resulting CSV would be:
field1,field2
one,red
one,blue
one,green
two,red
two,blue
two,green
If you wanted to assert that "one" and "red" can never appear together in a combination, then this can be done by supplying a constraint:
In the configuration for the field, express that when field1 has the value "one", that "field2" must have the values "blue" or "green":
{
"name" : "field1"
"type" : "generated",
"values" : {
"one" : {
"contstraints" : {
"field2" : ["blue", "green"]
}
},
"two" : {}
}
}This would produce the following CSV, where the previous combination of "one" and "red" is no longer present:
field1,field2
one,blue
one,green
two,red
two,blue
two,green
This feature can be used to eliminate unnecessary or unwanted parameters from a set of test parameters, for example, where combinations of certain parameters would make no sense.
You may wish to specify the value of certain fields depending on the values of other fields, independently of the combinations process. This can be done with a conditional field, which specifies the value of the field against each combination after they have been generated.
In the configuration for the field, for each value specify the conditions under which that value will be attached to a combination.
There can be multiple conditions for each value, each one specified as a separate object in the list of conditions. Each condition may refer to multiple fields in the combination, and each field may be specified with one or more possible values.
For example, to add the value "error" to a combination when both "field1" and "field2" have the value "none", you can specify the condition:
{
"field1" : ["none"],
"field2" : ["none"]
}If you had other values that may cause an error, for example, if field1 contained the value "oops", this could be added thus:
{
"field1" : ["none", "oops"],
"field2" : ["none"]
}The following example shows:
- ArgumentException if "field1" OR "field2" contain the value "none"
- AuthoriseException if "account" contains the value "unauthorised" or "unauthenticated"
- An empty string if none of the above conditions are met
{
"name" : "result",
"type" : "conditional",
"default" : "",
"values" : {
"ArgumentException" : {
"conditions" : [
{ "field1" : ["none"] },
{ "field2" : ["none"] }
]
},
"AuthoriseException" : {
"conditions" : [
{ "account" : ["unauthorised", "unauthenticated"] }
]
}
}
}This feature is useful to specifying the outcomes of tests based on the input parameters.
The following example document from a real-world use case shows how arguments can be specified for the Combinatrix, either as a JSON file
or as a python dict.
This document defines the parameters for a set of unit tests, which take the fields application and account with a variety of values
and inter-constraints, and which may raise certain kinds of exceptions conditional on the supplied parameters.
{
"parameters" : [
{
"name" : "test_id",
"type" : "index"
},
{
"name" : "application",
"type" : "generated",
"values" : {
"none" : {
"constraints" : {
"application_status" : ["-"],
"current_journal" : ["-"],
"save" : ["success"]
}
},
"exists" : {}
}
},
{
"name" : "account",
"type" : "generated",
"values" : {
"none" : {},
"publisher" : {},
"admin" : {}
}
},
{
"name" : "save",
"type" : "generated",
"values" : {
"success" : {},
"fail" : {
"constraints" : {
"application" : ["exists"]
}
}
}
},
{
"name" : "raises",
"type" : "conditional",
"default" : "",
"values" : {
"ArgumentException" : {
"conditions" : [
{ "application" : ["none"] },
{ "account" : ["none"] }
]
},
"AuthoriseException" : {
"conditions" : [
{
"application" : ["exists"],
"account" : ["unauthorised"]
}
]
},
"SaveException" : {
"conditions" : [
{ "save" : ["fail"] }
]
}
}
}
]
}The CSV format is isomorphic with the JSON format, and allows you to manage your test parameters in a more convenient visual way.
The general format of the CSV is as follows:
- In the first column is a directive which tells us what kind of row follows
- The first row MUST be the field name
- All subsequent rows provide the appropriate setting for the field name in each column
The first row must therefore be something like
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|
The first column tells us this is the field row, and then each subsequent column is the name of that field
We can then set the type of a field thus:
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
And the default value for columns where this is appropriate
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
| default | yes | no |
Note also that blank lines in the CSV are ignored, which can allow you to space the content for readability.
Further note that other than the first row, ordering of all subsequent rows is arbitrary.
We can then optionally (though recommended for readability) set the values which will appear in each field. If you do not specify the values, then these will be inferred from the constraints and conditions (see below for more information).
The first column contains the text value then each subsequent cell in the row contains a value that the corresponding field may take. If the cell is empty, the value is ignored, which means you cannot set the empty string as an allowed value in your combination.
For example, suppose field1 has two possible values, and field2 has three possible values:
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
| default | yes | no | |||
| values | a | 1 | yes | yes | |
| values | b | 2 | no | no | |
| values | 3 |
To add field constraints you provide a first column directive of the form constraint [field name]. So to apply a constraint on
values in field you would have the first column contain constraint field1:
For example, the final row in the following table places a constraint on the value a in field1 such that when a is the value in
a combination, then field2 must have the value 1:
For example:
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
| default | yes | no | |||
| values | a | 1 | yes | yes | |
| values | b | 2 | no | no | |
| values | 3 | ||||
| constraint field1 | a | 1 |
You may provide the constraint in both directions, or leave it only in one direction. For example, the following reflects the constraint
on field1 for field2 and also supplies an additional constraint on field2 when its value is 3:
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
| default | yes | no | |||
| values | a | 1 | yes | yes | |
| values | b | 2 | no | no | |
| values | 3 | ||||
| constraint field1 | a | 1 | |||
| constraint field2 | a | 1 | |||
| constraint field2 | b | 3 |
In this simple example, the constraints for field1 and field2 are equivalent, but the need to be able to specify which field
constrains which other fields is more obvious when there are 3 or more fields, and multiple co-dependent constraints.
To add conditionals, you provide a first column directive of the form conditional [field name]. So to apply a conditional value
for condition1, you would have the first column contain conditional condition1.
For example, the final row in the following table adds a condition such that condition1 is yes whenever field1 is b:
| field | test_id | field1 | field2 | condition1 | condition2 |
|---|---|---|---|---|---|
| type | index | generated | generated | conditional | conditional |
| default | yes | no | |||
| values | a | 1 | yes | yes | |
| values | b | 2 | no | no | |
| values | 3 | ||||
| constraint field1 | a | 1 | |||
| constraint field2 | a | 1 | |||
| constraint field2 | b | 3 | |||
| conditional condition1 | a | yes |
TODO
TODO
Combinatrix is designed to allow you to plug combinations directly into parameterised tests, and it does this through the concept of a test "bundle".
Each bundle consists of one or both of a settings CSV and settings JSON, and the resulting test matrix itself.
To execute a test based on a test bundle, we can use parameterized as follows:
import unittest
from parameterized import parameterized
from combinatrix.testintegration import load_parameter_sets, rel2abs
def load_cases():
return load_parameter_sets(rel2abs(__file__, "..", "resources", "bundles", "combine"), "combine", "test_id", {"test_id" : []})
class TestCombine(unittest.TestCase):
@parameterized.expand(load_cases)
def test_combine(self, name, kwargs):
passHere we have defined a test class called TestCombine which extends the unittest.TestCase. In turn this contains a single test function
called test_combine which takes two arguments:
- name - the name of the test
- kwargs - a dictionary of arguments for the test
This function is annotated with @parameterized.expand(load_cases). This is a feature of parameterized which will load the test cases
from the supplied function, and inject them into the test class at run-time.
We therefore have also defined a function called load_cases which uses Combinatrix to deliver the test cases.
There is a function provided which can load parameter sets and pass them to parameterized:
from combinatrix.testintegration import load_parameter_sets, rel2abs
def load_cases():
return load_parameter_sets(rel2abs(__file__, "..", "resources", "bundles", "combine"), "combine", "test_id")load_parameter_sets is invoked with the following arguments:
- The path to the test "bundle" for this test
- The name of the test "bundle"
- The field in the test matrix which will define the name of the test
- An optional filter which will allow us to limit the parameter sets to a subset of the total
Note also we provide a convenience function rel2abs which allows us to convert relative paths to absolue paths easily.
The above example loads a bundle called combine from the directory ../resources/bundles/combine. It tells us that the field test_id is the field that will uniquely name each test, and it applies a filter
TODO