Permalink
Browse files

Update documentation and examples for generators

  • Loading branch information...
horenmar committed Jan 29, 2019
1 parent 5929d95 commit 061f1f836af4fcbec103e55f5c375260f0028937
@@ -1,50 +1,125 @@
<a id="top"></a>
# Data Generators

_Generators are currently considered an experimental feature and their
API can change between versions freely._

Data generators (also known as _data driven/parametrized test cases_)
let you reuse the same set of assertions across different input values.
In Catch2, this means that they respect the ordering and nesting
of the `TEST_CASE` and `SECTION` macros.

How does combining generators and test cases work might be better
explained by an example:
of the `TEST_CASE` and `SECTION` macros, and their nested sections
are run once per each value in a generator.

This is best explained with an example:
```cpp
TEST_CASE("Generators") {
auto i = GENERATE( range(1, 11) );
auto i = GENERATE(1, 2, 3);
SECTION("one") {
auto j = GENERATE( -3, -2, -1 );
REQUIRE(j < i);
}
}
```

The assertion in this test case will be run 9 times, because there
are 3 possible values for `i` (1, 2, and 3) and there are 3 possible
values for `j` (-3, -2, and -1).


There are 2 parts to generators in Catch2, the `GENERATE` macro together
with the already provided generators, and the `IGenerator<T>` interface
that allows users to implement their own generators.

## Provided generators

Catch2's provided generator functionality consists of three parts,

* `GENERATE` macro, that serves to integrate generator expression with
a test case,
* 2 fundamental generators
* `ValueGenerator<T>` -- contains only single element
* `ValuesGenerator<T>` -- contains multiple elements
* 4 generic generators that modify other generators
* `FilterGenerator<T, Predicate>` -- filters out elements from a generator
for which the predicate returns "false"
* `TakeGenerator<T>` -- takes first `n` elements from a generator
* `RepeatGenerator<T>` -- repeats output from a generator `n` times
* `MapGenerator<T, U, Func>` -- returns the result of applying `Func`
on elements from a different generator

The generators also have associated helper functions that infer their
type, making their usage much nicer. These are

* `value(T&&)` for `ValueGenerator<T>`
* `values(std::initializer_list<T>)` for `ValuesGenerator<T>`
* `filter(predicate, GeneratorWrapper<T>&&)` for `FilterGenerator<T, Predicate>`
* `take(count, GeneratorWrapper<T>&&)` for `TakeGenerator<T>`
* `repeat(repeats, GeneratorWrapper<T>&&)` for `RepeatGenerator<T>`
* `map(func, GeneratorWrapper<T>&&)` for `MapGenerator<T, T, Func>` (map `T` to `T`)
* `map<T>(func, GeneratorWrapper<U>&&)` for `MapGenerator<T, U, Func>` (map `U` to `T`)

SECTION( "Some section" ) {
auto j = GENERATE( range( 11, 21 ) );
REQUIRE(i < j);
And can be used as shown in the example below to create a generator
that returns 100 odd random number:

```cpp
TEST_CASE("Generating random ints", "[example][generator]") {
SECTION("Deducing functions") {
auto i = GENERATE(take(100, filter([](int i) { return i % 2 == 1; }, random(-100, 100))));
REQUIRE(i > -100);
REQUIRE(i < 100);
REQUIRE(i % 2 == 1);
}
}
```

the assertion will be checked 100 times, because there are 10 possible
values for `i` (1, 2, ..., 10) and for each of them, there are 10 possible
values for `j` (11, 12, ..., 20).
_Note that `random` is currently not a part of the first-party generators_.


Apart from registering generators with Catch2, the `GENERATE` macro has
one more purpose, and that is to provide simple way of generating trivial
generators, as seen in the first example on this page, where we used it
as `auto i = GENERATE(1, 2, 3);`. This usage converted each of the three
literals into a single `ValueGenerator<int>` and then placed them all in
a special generator that concatenates other generators. It can also be
used with other generators as arguments, such as `auto i = GENERATE(0, 2,
take(100, random(300, 3000)));`. This is useful e.g. if you know that
specific inputs are problematic and want to test them separately/first.

**For safety reasons, you cannot use variables inside the `GENERATE` macro.**

You can also override the inferred type by using `as<type>` as the first
argument to the macro. This can be useful when dealing with string literals,
if you want them to come out as `std::string`:

You can also combine multiple generators by concatenation:
```cpp
static int square(int x) { return x * x; }
TEST_CASE("Generators 2") {
auto i = GENERATE(0, 1, -1, range(-20, -10), range(10, 20));
CAPTURE(i);
REQUIRE(square(i) >= 0);
TEST_CASE("type conversion", "[generators]") {
auto str = GENERATE(as<std::string>{}, "a", "bb", "ccc");`
REQUIRE(str.size() > 0);
}
```

This will call `square` with arguments `0`, `1`, `-1`, `-20`, ..., `-11`,
`10`, ..., `19`.
## Generator interface

You can also implement your own generators, by deriving from the
`IGenerator<T>` interface:

```cpp
template<typename T>
struct IGenerator : GeneratorUntypedBase {
// via GeneratorUntypedBase:
// Attempts to move the generator to the next element.
// Returns true if successful (and thus has another element that can be read)
virtual bool next() = 0;
// Precondition:
// The generator is either freshly constructed or the last call to next() returned true
virtual T const& get() const = 0;
};
```

However, to be able to use your custom generator inside `GENERATE`, it
will need to be wrapped inside a `GeneratorWrapper<T>`.
`GeneratorWrapper<T>` is a value wrapper around a
`std::unique_ptr<IGenerator<T>>`.

----------
For full example of implementing your own generator, look into Catch2's
examples, specifically
[Generators: Create your own generator](../examples/300-Gen-OwnGenerator.cpp).

Because of the experimental nature of the current Generator implementation,
we won't list all of the first-party generators in Catch2. Instead you
should look at our current usage tests in
[projects/SelfTest/UsageTests/Generators.tests.cpp](/projects/SelfTest/UsageTests/Generators.tests.cpp).
For implementing your own generators, you can look at their implementation in
[include/internal/catch_generators.hpp](/include/internal/catch_generators.hpp).
@@ -14,6 +14,9 @@
- Report: [TeamCity reporter](../examples/207-Rpt-TeamCityReporter.cpp)
- Listener: [Listeners](../examples/210-Evt-EventListeners.cpp)
- Configuration: [Provide your own output streams](../examples/231-Cfg-OutputStreams.cpp)
- Generators: [Create your own generator](../examples/300-Gen-OwnGenerator.cpp)
- Generators: [Use variables in generator expressions](../examples/310-Gen-VariablesInGenerators.cpp)


## Planned

@@ -0,0 +1,59 @@
// 300-Gen-OwnGenerator.cpp
// Shows how to define a custom generator.

// Specifically we will implement a random number generator for integers
// It will have infinite capacity and settable lower/upper bound

#include <catch2/catch.hpp>

#include <random>

// This class shows how to implement a simple generator for Catch tests
class RandomIntGenerator : public Catch::Generators::IGenerator<int> {
std::minstd_rand m_rand;
std::uniform_int_distribution<> m_dist;
int current_number;
public:

RandomIntGenerator(int low, int high):
m_rand(std::random_device{}()),
m_dist(low, high)
{
static_cast<void>(next());
}

int const& get() const override;
bool next() override {
current_number = m_dist(m_rand);
return true;
}
};

// Avoids -Wweak-vtables
int const& RandomIntGenerator::get() const {
return current_number;
}

// This helper function provides a nicer UX when instantiating the generator
// Notice that it returns an instance of GeneratorWrapper<int>, which
// is a value-wrapper around std::unique_ptr<IGenerator<int>>.
Catch::Generators::GeneratorWrapper<int> random(int low, int high) {
return Catch::Generators::GeneratorWrapper<int>(std::unique_ptr<Catch::Generators::IGenerator<int>>(new RandomIntGenerator(low, high)));
}

// The two sections in this test case are equivalent, but the first one
// is much more readable/nicer to use
TEST_CASE("Generating random ints", "[example][generator]") {
SECTION("Nice UX") {
auto i = GENERATE(take(100, random(-100, 100)));
REQUIRE(i >= -100);
REQUIRE(i <= 100);
}
SECTION("Creating the random generator directly") {
auto i = GENERATE(take(100, GeneratorWrapper<int>(std::unique_ptr<IGenerator<int>>(new RandomIntGenerator(-100, 100)))));
REQUIRE(i >= -100);
REQUIRE(i <= 100);
}
}

// Compiling and running this file will result in 400 successful assertions
@@ -0,0 +1,72 @@
// 310-Gen-VariablesInGenerator.cpp
// Shows how to use variables when creating generators.

// Note that using variables inside generators is dangerous and should
// be done only if you know what you are doing, because the generators
// _WILL_ outlive the variables -- thus they should be either captured
// by value directly, or copied by the generators during construction.

#include <catch2/catch.hpp>

#include <random>

// Lets start by implementing a parametrizable double generator
class RandomDoubleGenerator : public Catch::Generators::IGenerator<double> {
std::minstd_rand m_rand;
std::uniform_real_distribution<> m_dist;
double current_number;
public:

RandomDoubleGenerator(double low, double high):
m_rand(std::random_device{}()),
m_dist(low, high)
{
static_cast<void>(next());
}

double const& get() const override;
bool next() override {
current_number = m_dist(m_rand);
return true;
}
};

// Avoids -Wweak-vtables
double const& RandomDoubleGenerator::get() const {
return current_number;
}


// Also provide a nice shortcut for creating the generator
Catch::Generators::GeneratorWrapper<double> random(double low, double high) {
return Catch::Generators::GeneratorWrapper<double>(std::unique_ptr<Catch::Generators::IGenerator<double>>(new RandomDoubleGenerator(low, high)));
}


TEST_CASE("Generate random doubles across different ranges",
"[generator][example][advanced]") {
// Workaround for old libstdc++
using record = std::tuple<double, double>;
// Set up 3 ranges to generate numbers from
auto r = GENERATE(table<double, double>({
record{3, 4},
record{-4, -3},
record{10, 1000}
}));

// This will not compile (intentionally), because it accesses a variable
// auto number = GENERATE(take(50, random(r.first, r.second)));

// We have to manually register the generators instead
// Notice that we are using value capture in the lambda, to avoid lifetime issues
auto number = Catch::Generators::generate( CATCH_INTERNAL_LINEINFO,
[=]{
using namespace Catch::Generators;
return makeGenerators(take(50, random(std::get<0>(r), std::get<1>(r))));
}
);
REQUIRE(std::abs(number) > 0);
}

// Compiling and running this file will result in 150 successful assertions

@@ -44,6 +44,8 @@ set( SOURCES_IDIOMATIC_TESTS
110-Fix-ClassFixture.cpp
120-Bdd-ScenarioGivenWhenThen.cpp
210-Evt-EventListeners.cpp
300-Gen-OwnGenerator.cpp
310-Gen-VariablesInGenerators.cpp
)

# main-s for reporter-specific test sources:

0 comments on commit 061f1f8

Please sign in to comment.