68 changes: 54 additions & 14 deletions doc/test_organization/parametric_test_case_generation.qbk
View file
Expand Up @@ -122,8 +122,8 @@ The size of the tuple will be by definition the *arity* of the sample itself.
A *dataset* is a /collection of samples/, that

* is forward iterable,
* can be queried for its `size`, which in turn can be infinite,
* has an arity, which is the arity of the samples it contains.
* can be queried for its `size` which in turn can be infinite,
* has an arity which is the arity of the samples it contains.

Hence the dataset implements the notion of /sequence/.

Expand All @@ -134,30 +134,67 @@ The descriptive power of the datasets in __UTF__ comes from
* their interface with other type of collections (`stl` containers, `C` arrays)
* the available built-in [link boost_test.tests_organization.test_cases.test_case_generation.generators /dataset generators/]

[tip Only "monomorphic" datasets are supported, which means that all samples in a dataset have the same type and same arity
[tip Only "monomorphic" datasets are supported, which means that all samples within a single dataset have the same type and same arity
[footnote polymorphic datasets will be considered in the future. Their need is mainly driven by the replacement of the
[link boost_test.tests_organization.test_cases.test_organization_templates typed parametrized test cases] by the dataset-like API.]
.
. However, dataset of different sample types may be combined together with zip and cartesian product.
]

As we will see in the next sections, datasets representing collections of different types may be combined together (e.g.. /zip/ or /grid/).
These operations result in new datasets, in which the samples are of an augmented type.

[h4 Dataset interface]
[/ ###################################################################### ]

[section Dataset interface]
The interface of the /dataset/ should implement the two following functions/fields:

* `iterator begin()` where /iterator/ is a forward iterator,
* `boost::unit_test::data::size_t size() const` indicates the size of the dataset. The returned type is a dedicated
class [classref boost::unit_test::data::size_t size_t] that can indicate an /infinite/ dataset size.
* an enum called `arity` indicating the arity of the samples returned by the dataset

Once a dataset class `D` is declared, it should be registered to the framework by specializing the class ``boost::unit_test::data::monomorphic::is_dataset``
Once a dataset class `D` is declared, it should be registered to the framework by specializing the template class
``boost::unit_test::data::monomorphic::is_dataset``
with the condition that ``boost::unit_test::data::monomorphic::is_dataset<D>::value`` evaluates to `true`.

The following example implements a custom dataset generating a Fibonacci sequence.

[bt_example dataset_example68..Example of custom dataset..run-fail]

[endsect]

[/ ###################################################################### ]

[section Dataset creation and delayed creation]
Datasets as defined above are constructed before even the test module starts its execution as global objects. This makes impossible to access,
from within the dataset generator and during their iteration, elements like `argc` / `argv`, the
[link boost_test.tests_organization.test_tree.master_test_suite master test suite] (and the preprocessed `argc` / `argv`), or any other object
that has been instanciated after the `main` of the test module entry.

To overcome this, a [*delayed] dataset instanciation interface has been introduced. This effectively wraps the dataset inside another one,
which [*lazyly] instanciates the dataset.

To instanciate a delayed dataset, the [funcref boost::unit_test::data::monomorphic::make_delayed] function should be used in the
__BOOST_DATA_TEST_CASE__ call. The following snippet:

```
BOOST_DATA_TEST_CASE(dataset_test_case,
boost::unit_test::data::make_delayed<custom_dataset>(arg1, ... ), ...)
{
}
```

creates a delayed dataset test case with a generator of type `custom_dataset`. The generator is ['lazily] constructed
with `arg1`, `...`.

[tip A detailed example of delayed creation is given in the section about [link boost_test.runtime_config.custom_command_line_arguments custom command line]
arguments.]

[tip See the class [classref boost::unit_test::data::monomorphic::delayed_dataset `monomorphic::delayed_dataset`] for more details on the
wrapping object.]

[endsect]

[endsect] [/ datasets]


Expand Down Expand Up @@ -219,19 +256,22 @@ More precisely, what
does is the following:

* it registers a *test suite* named "`test_case_name`",
* it registers as many test cases as they are in "`dataset`", each of which with the name corresponding to the index of the sample in the database prefixed by `_` and
starting at index `0` ("`_0`", "`_1`", ... "`_(N-1)`" where `N` is the size of the dataset)
* it registers as many test cases as they are in "`dataset`", each of which with the name corresponding to the index of the sample
in the database prefixed by `_` and starting at index `0` ("`_0`", "`_1`", ... "`_(N-1)`" where `N` is the size of the dataset)

This make it easy to:

* identify which sample is failing (say "`test_case_name/_3`")
* allows a replay of one or several samples (or the full dataset) from the command line using the [link boost_test.runtime_config.test_unit_filtering test filtering facility] provided by the __UTF__
* identify which sample is failing (say "`test_case_name/_3`"),
* replay the test for one or several samples (or the full dataset) from the command line using the [link boost_test.runtime_config.test_unit_filtering test filtering features] provided by the __UTF__,
* apply a [link boost_test.tests_organization.decorators.explicit_decorator_declaration decorator] to each individual test cases of the
dataset, as the decorator would apply to the test suite.

Exactly as regular test cases, each test case (associated to a specific sample) is executed within the test body in a /guarded manner/:
Exactly as regular test cases, each test case (associated to a specific sample) is executed in /monitored manner/:

* the test execution are independent: if an error occurs for one sample, the remaining samples execution is not affected
* in case of error, the [link boost_test.test_output.test_tools_support_for_logging.contexts context] within which the error occurred is reported in the [link boost_test.test_output log] along with
the failing sample index. This context contains the sample for which the test failed, which would ease the debugging.
* the test execution are independent: if an error occurs for one sample, the remaining samples execution is not affected,
* in case of error, the [link boost_test.test_output.test_tools_support_for_logging.contexts context] along with the index of the sample
within which the error occurred is reported in the [link boost_test.test_output log].
This context contains the sample names and values for which the test failed, which would ease the debugging.

[endsect]

Expand Down
4 changes: 4 additions & 0 deletions doc/test_organization/test_enabling_disabling.qbk
View file
Expand Up @@ -64,4 +64,8 @@ specified tests passed. (It assumes that the tests are registered in the specifi
* test case `test4` has a precondition that either `test2` or `test3` passed. Since they both failed, the
precondition is not satisfied, therefore `test4` is skipped.

[note A __decorator_precondition__ that evaluates to `false` does not yield an error and does not fail the attached unit test. However
the __UTF__ returns an error if the test tree is empty (see [link boost_test.tests_organization.test_tree.test_tree_content this section]
for more details). ]

[endsect]
22 changes: 20 additions & 2 deletions doc/test_organization/test_naming.qbk
View file
Expand Up @@ -7,9 +7,11 @@


[section:test_naming Tests naming]
The __UTF__ is flexible on how naming the test units (test cases/suites) and

[h2 Acceptable characters for names]
The __UTF__ is flexible on the naming the test units (test cases/suites) and
it is possible to use a wide variety of characters. However there are scenarios
where the name of a test unit may collide with other and remote functionalities
where the name of a test unit may collide with other features
of the __UTF__. For instance:

* a character may collide with specific command line filters for selecting a
Expand All @@ -29,4 +31,20 @@ conflicting character is replaced by an underscore `'_'`.
[note The sanitization of the test unit names has been introduced in [link ref_CHANGE_LOG_3_7 Boost 1.67].
]

[h2 Constraints on test names]
The __UTF__ considers an error if two sibling test cases have the same name. This consistency check is performed
at runtime and is not sensitive to test filtering.

For test-cases registered automatically, and within the same compilation unit, a compilation error should be raised by
the compiler in case of name conflict.
For manual registration of test cases and in case of conflicts, the too __BOOST_TEST_CASE_NAME__ may be used.

The data driven tests are automatically created in an isolated test-suite and with a name that is indexed on the
sample.


[note This constraints does not apply to test-suites: Opening/closing a test-suite declaration
adds the test-cases to an existing or newly created test-suite of that name.]

[endsect] [/ test test_naming]

14 changes: 7 additions & 7 deletions doc/test_organization/test_organization.qbk
View file
Expand Up @@ -31,10 +31,12 @@ These units are the *test cases*, the *test suites* and the *fixtures*.
]
[
[Organization]
[The __UTF__ provides facilities to group several test cases into [link boost_test.tests_organization.test_suite test suites].
The test suites can be nested, and the set of test suites and test cases defines the /test tree/, where the leaves are the test cases.
Besides hierarchical structure the __UTF__ allows you to organize the test tree using /logical grouping/ and /dependencies/
and provides you with controls to utilize the defined test tree organization the way you want.
[The __UTF__ provides facilities to group several test cases into [link boost_test.tests_organization.test_tree.test_suite test suites].
The test suites can be nested, and the set of test suites and test cases defines the [link boost_test.tests_organization.test_tree test tree],
where the leaves are the test cases.
Besides hierarchical structure the __UTF__ allows you to organize the test tree using [link boost_test.tests_organization.tests_grouping logical grouping]
and [link boost_test.tests_organization.tests_dependencies dependencies]
and provides you with controls to utilize the defined test tree organization the way you want (eg. from command line).
]
]
[
Expand All @@ -53,8 +55,7 @@ These units are the *test cases*, the *test suites* and the *fixtures*.
]

[include test_cases.qbk]
[include test_suites.qbk]
[include test_naming.qbk]
[include test_tree.qbk]
[include decorators.qbk]
[include fixtures.qbk]
[include managing_tests_dependencies.qbk]
Expand All @@ -63,7 +64,6 @@ These units are the *test cases*, the *test suites* and the *fixtures*.
[include semantic.qbk]
[include testorg_summary.qbk]


[endsect] [/ test organization]

[/EOF]
85 changes: 3 additions & 82 deletions doc/test_organization/test_suites.qbk
View file
Expand Up @@ -16,7 +16,7 @@ test suite creation and registration facilities:
# Manually registered test suite

In addition the __UTF__ presents a notion of the
[link boost_test.tests_organization.test_suite.master_test_suite Master Test Suite].
[link boost_test.tests_organization.test_tree.master_test_suite Master test suite].
The most important reason to learn about this component is that it provides an ability to access
command line arguments supplied to a test module.

Expand Down Expand Up @@ -87,8 +87,7 @@ default no errors are expected.
The third optional parameter - `timeout` - defines the timeout value for the test unit. As of now the __UTF__
isn't able to set a timeout for the test suite execution, so this parameter makes sense only for test case
registration. By default no timeout is set. See the method
[memberref boost::execution_monitor::execute] for more details about the timeout value. [warning is the reference
good? It looks to me that [memberref boost::unit_test::test_suite::add] is better]
[memberref boost::unit_test::test_suite::add] for more details about the timeout value.

To register group of test units in one function call, the [classref boost::unit_test::test_suite `test_suite`] class provides another
[memberref boost::unit_test::test_suite::add `add`] interface covered in the advanced section of this documentation.
Expand All @@ -111,7 +110,7 @@ constructed instance. Alternatively you can create an instance of class `boost::
to create one on the stack.
]

Newly created test suite has to be registered in a parent one using add interface. Both test suite creation and
Newly created test suite has to be registered in a parent one using the `add` interface. Both test suite creation and
registration is performed in the test module initialization function.

The example below creates a test tree, which can be represented by the following hierarchy:
Expand All @@ -120,84 +119,6 @@ The example below creates a test tree, which can be represented by the following

[bt_example example11..Manually registered test suites..run]

[section:master_test_suite Master Test Suite]


As defined in introduction section the master test suite is the *root* node of the test tree. Each test module built
with the __UTF__ always has the (unique) master test suite defined. The __UTF__ maintain the master test suite instance
internally. All other test units are registered as direct or indirect children of the master test suite.

``
namespace boost {
namespace unit_test {
class master_test_suite_t : public test_suite
{
/// implementation details
public:
int argc;
char** argv;
};

} // namespace unit_test
} // namespace boost
``


To access single instance of the master test suite use the following interface:

``
namespace boost {
namespace unit_test {
namespace framework {

master_test_suite_t& master_test_suite();

} // namespace framework
} // namespace unit_test
} // namespace boost
``

[h4 Command line arguments access interface]

Master test suite implemented as an extension to the regular test suite, since it maintains references to the
command line arguments passed to the test module. To access the command line arguments use

``
boost::unit_test::framework::master_test_suite().argc
boost::unit_test::framework::master_test_suite().argv
``

In below example references to the command line arguments are accessible either as an initialization function
parameters or as members of the master test suite. Both references point to the same values. A test module that
uses the alternative initialization function specification can only access command line arguments through the
master test suite.


Returning to the free function example, let's modify initialization function to check for absence of any
test module arguments.

[bt_example example13..Command line access in initialization function..run]

[#ref_BOOST_TEST_MODULE][h4 Naming the ['Master test suite]]

The master test suite is created with default name ['Master Test Suite]. There are two methods two
reset the name to a different value: using the macro __BOOST_TEST_MODULE__
and from within the test module initialization function. Former is used for test modules that don't have the
manually implemented initialization function. Following examples illustrate these methods.

[bt_example example14..Naming master test suite using the macro __BOOST_TEST_MODULE__..run]

If the macro __BOOST_TEST_MODULE__ is defined, the test module initialization
function is [*automatically generated] and the
macro value becomes the name of the master test suite. The name may include spaces.

[bt_example example15..Naming master test suite explicitly in the test module initialization function..run]

Without the __BOOST_TEST_MAIN__ and the __BOOST_TEST_MODULE__ flags defined, the test module initialization
function has to be manually implemented. The master test suite name can be reset at any point within this function.

[endsect] [/ command line interface]

[endsect] [/ test suite]

[/EOF]
41 changes: 41 additions & 0 deletions doc/test_organization/test_tree.qbk
View file
@@ -0,0 +1,41 @@
[/
/ Copyright (c) 2019 Raffi Enficiaud
/
/ Distributed under the Boost Software License, Version 1.0. (See accompanying
/ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
/]

[section:test_tree Test tree]

The test tree is the hierarchy of test cases and test suites, along with all the fixtures (global, case or
suite level), and the respective dependencies within all those elements.

A test tree is composed with:

* Test cases: those are the elements in the tree that contain the body of the tests, and they constitute
the [*leaves] of the tree.
* Test suites: those are the are internal nodes of the tree. These elements that do not have any body or executable code themselves,
but fixtures that execute code and tests can be attached to them.
* The master test suite: this is the root of the tree, and is by definition a test suite. Fixtures attached to the master test suite
are [*global] fixtures.
* fixtures: those are units of code that are executed before and/or after the test units above.

The following hierarchy represents a test tree (further detailed in the [link boost_test.tests_organization.test_tree.test_suite test-suite]
section) [*without] any fixture:

[$images/class-hier.jpg]

[link boost_test.tests_organization.decorators Decoration] can be added to test suites and cases except for the master test suite. Those
decoration may modify the way the __UTF__
handles the tree. For instance, there is no defined order in the execution of test cases enforced by the tree itself, except for the fixtures
and the elements they relate to (suite, case); decoration may be used to instruct a specific order among the elements of the tree.

[note The test-tree by itself does not give any particular order in the execution of the test-cases. The only implicit order
is given by the fixtures. To indicate a particular order, specific decorators should be used.]

[include test_suites.qbk]
[include master_test_suite.qbk]
[include test_naming.qbk]
[include test_tree_content.qbk]

[endsect] [/ test test_tree]
20 changes: 20 additions & 0 deletions doc/test_organization/test_tree_content.qbk
View file
@@ -0,0 +1,20 @@
[/
/ Copyright (c) 2019 Raffi Enficiaud
/
/ Distributed under the Boost Software License, Version 1.0. (See accompanying
/ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
/]

[section:test_tree_content Test tree content]

The test module, and hence the test tree, should have at least one test-case scheduled for execution, otherwise the __UTF__
returns an error.

This check is performed *after* the filtering of the tests has occured. The sections about
[link boost_test.tests_organization.enabling tests enabling/disabling], [link boost_test.tests_organization.tests_dependencies tests dependencies]
and [link boost_test.runtime_config.test_unit_filtering runtime filtering]
explain various methods that modify the tests that are being run.

[note A [link boost_test.tests_organization.fixtures.global global fixture] is not considered as a test case.]

[endsect] [/ test test_tree_content]
2 changes: 1 addition & 1 deletion doc/test_organization/testorg_reference.qbk
View file
Expand Up @@ -172,7 +172,7 @@ This macro is deprecated in favor of __BOOST_TEST_GLOBAL_FIXTURE__ and __BOOST_T
[/-----------------------------------------------------------------]
[section:test_org_boost_test_global_fixture `BOOST_TEST_GLOBAL_FIXTURE`]
Declares and registers a global fixture. The global fixture acts exactly as a suite fixture attached to the
[link boost_test.tests_organization.test_suite.master_test_suite master test suite],
[link boost_test.tests_organization.test_tree.master_test_suite master test suite],
and is called before any of the test case in the test tree is executed.

The class implementing the fixture should have the appropriate [link boost_test.tests_organization.fixtures.models interface].
Expand Down
2 changes: 1 addition & 1 deletion doc/test_output/log_format.qbk
View file
Expand Up @@ -215,7 +215,7 @@ active log level threshold.
The [@http://junit.org/ JUNIT format] is log format supported by a wide range of Continuous Build/Integration tools.

This format defaults its log level to [link test_log_output_table `General information`] and its default stream to a file named after
[link boost_test.tests_organization.test_suite.master_test_suite master test suite].
[link boost_test.tests_organization.test_tree.master_test_suite master test suite].
The logger will attempt to not overwrite any existing output file, which is also usually understood by Continuous Build tools.

This format is in fact both a log and a report format: most of the Continuous Build tools will summarize
Expand Down
26 changes: 16 additions & 10 deletions doc/testing_tools/boost_test_collection_comparison.qbk
View file
Expand Up @@ -181,20 +181,26 @@ through the macro `BOOST_TEST_SPECIALIZED_COLLECTION_COMPARE` that might be used
[#what_is_a_collection][h3 What is a sequence?]
A /sequence/ is given by the iteration over a /forward iterable/ container. A forward iterable container is:

* either a C-array, where the underlying type is neither `char` or `wchar_t`
* or a `class`/`struct` that implements the member functions `begin` and `end` and for which `value_type` is neither
`char` or `wchar_t`
* either a C-array,
* or a `class`/`struct` that implements the member functions `begin` and `end`.

When the stored type `char` or `wchar_t`, then the container is considered as a string and is dispatched on
[link boost_test.testing_tools.extended_comparison.strings string comparison] instead.
For collection comparisons, both sequences are also required to be different than `string` sequences. In that case, the sequences are
dispatched to string [link boost_test.testing_tools.extended_comparison.strings comparison instead].

[note Since [link ref_CHANGE_LOG_3_6 Boost.Test 3.6] (Boost 1.65) the requirements for the collection concepts have been relaxed to include C-arrays as well]
[note Since [link ref_CHANGE_LOG_3_7 Boost.Test 3.7] (Boost 1.67) the definition of `const_iterator` in the container type is not required anymore (for
the compilers properly supporting `decltype`). ]
[warning `string` (or `wstring`) meets the sequence concept by definition, but their handling with __BOOST_TEST__ is done differently.
See [link boost_test.testing_tools.extended_comparison.strings Strings and C-strings comparison] for more details.]

[tip If the behaviour of __BOOST_TEST__ is not the one you expect, you can always use raw comparison. See [link boost_test_statement_limitations this section]
for details.]

[note Since [link ref_CHANGE_LOG_3_6 Boost.Test 3.6] (Boost 1.65) the requirements for the collection concepts have been relaxed to
include C-arrays as well]
[note Since [link ref_CHANGE_LOG_3_7 Boost.Test 3.7] (Boost 1.67) the definition of `const_iterator` and `value_type` in the collection
type is not required anymore (for the compilers properly supporting `decltype`). ]

The detection of the types that meet these requirements containers is delegated to the class [classref boost::unit_test::is_forward_iterable],
which for C++11 detects the required member functions and fields. However for C++03, the types providing the sequences should be explicitly indicated
to the __UTF__ by a specialization of [classref boost::unit_test::is_forward_iterable]
which for C++11 detects the required member functions and fields. However for C++03, the types providing the sequences should be explicitly
indicated to the __UTF__ by a specialization of [classref boost::unit_test::is_forward_iterable]
[footnote Standard containers of the `STL` are recognized as /forward iterable/ container.].

[endsect] [/ sequences]
2 changes: 2 additions & 0 deletions include/boost/test/data/monomorphic/delayed.hpp
View file
Expand Up @@ -100,6 +100,8 @@ struct is_dataset< delayed_dataset<dataset_t, Args...> > : boost::mpl::true_ {};

} // namespace monomorphic


//! Delayed dataset instanciation
template<class dataset_t, class ...Args>
inline typename std::enable_if<
monomorphic::is_dataset< dataset_t >::value,
Expand Down