Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
353657d
Fixed issue with `lazy_value` when a single mark is passed
Jun 24, 2020
def5efd
`_parametrize_plus` minor refactoring for readability: created `_proc…
Jun 24, 2020
e91211c
Tests reorganization: old things moved to subfolder
Jun 24, 2020
40c0c44
Refactoring for V2 and first working prototypes
Jun 24, 2020
e8a143a
Added associated tests for new features
Jun 24, 2020
433f308
More readable error message when a lazy_value used in parametrize ret…
Jun 24, 2020
ed1ff08
Now correctly handling marks and case info put on a case function in …
Jun 24, 2020
b196e58
Fixed tests for pytest 2
Jun 24, 2020
b6191ed
New pytest goodie `assert_exception` that can be used as a context ma…
Jun 25, 2020
6c1f359
Minor test improvement
Jun 25, 2020
0f44859
Initiated changelog. I forgot to note it in the commit message but a …
Jun 25, 2020
3b0d8bd
A few deprecation warnings and explicit associated docstring
Jun 25, 2020
744f052
`parametrize_plus` now automatically detects when wrong `argnames` ar…
Jun 26, 2020
702fea3
`parametrize_plus` now provides an alternate way to pass argnames, ar…
Jun 28, 2020
14d96b3
New aliases for readibility: `fixture` for `fixture_plus`, and`parame…
Jun 28, 2020
950930b
Fixed `idgen` when a `pytest.param` (or mark in pytest 2) is present …
Jun 29, 2020
93094a9
Now `idgen` has a consistent default behaviour in `parametrize_plus` …
Jun 29, 2020
430527e
missing module from last commit
Jun 29, 2020
d0b2838
Added tagging and filtering capabilities to the new symbols. `target`…
Jun 29, 2020
5a8f908
Now a case can be parametrized using `@parametrize` : no need to use …
Jun 29, 2020
0776d00
Fixed marks for old pytest 2
Jun 29, 2020
e779dbf
Updated tests for the new cases, and extended them with parametrized …
Jun 29, 2020
9708ae7
Fixed a few old tests that were now failing
Jun 29, 2020
018a104
Big refactoring:
Jun 30, 2020
2ebe828
Fixed `_get_argnames_argvalues` in presence of an hybrid parametrizat…
Jun 30, 2020
7b50a96
Added/updated tests corresponding to all latest changes
Jun 30, 2020
9d7ce03
fixed tests
Jun 30, 2020
070c61c
fixed tests
Jun 30, 2020
9c80a4b
Fixed test for pytest 2 with python 3.6+
Jun 30, 2020
f31163c
Improved cartesian product
Jun 30, 2020
97b4d8f
Fixed test (2)
Jun 30, 2020
064c07f
Fixed test... last ? :)
Jul 1, 2020
506119e
Now case functions can require a fixture. In that case they will be t…
Jul 3, 2020
0578c09
Any error message associated to a `lazy_value` function call is not c…
Jul 3, 2020
5aefb68
2.0.0 changelog first draft
Jul 3, 2020
c666613
Updated changelog
Jul 3, 2020
c628e5d
Improved error message associated with default case module import
Jul 7, 2020
3bff0e0
Now using `mini_idval` in `@parametrize` when `idgen=AUTO` (default),…
Jul 7, 2020
87370af
New easier API for filtering in `@parametrize_with_cases`, through a …
Jul 7, 2020
fefb76c
Documentation improvements and associated tests
Jul 7, 2020
2fbb150
Minor: Docstring and type hints
Jul 9, 2020
935b872
Improved filtering capabilities: new `glob` argument, and `filter` ca…
Jul 9, 2020
fe44b64
Added/updated tests corresponding to all latest changes
Jul 9, 2020
9466f55
Updated doc and API reference
Jul 9, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
377 changes: 169 additions & 208 deletions docs/api_reference.md

Large diffs are not rendered by default.

62 changes: 62 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,67 @@
# Changelog

### 2.0.0 - Less boilerplate & full `pytest` alignment !

**Case functions**

- New `@case(id=None, tags=(), marks=())` decorator to replace `@case_name`, `@case_tags` and `@test_target` (all deprecated): a single simple way to customize all aspects of a case function. Also, `target` disappears from the picture as it was just a tag like others - this could be misleading.

- `@cases_generator` is now deprecated and replaced with `@parametrize` : now, cases can be parametrized exactly the same way than tests. This includes the ability to put marks on the whole or on some specific parameter values. `@parametrize_plus` has been renamed `@parametrize` for readability. It was also improved in order to support the alternate parametrization mode that was previously offered by `@cases_generator`. That way, users will be able to choose the style of their choice. Fixes [#57](https://github.com/smarie/python-pytest-cases/issues/57) and [#106](https://github.com/smarie/python-pytest-cases/issues/106).

- Since `@cases_generat`Marks can now

- Now case functions can require fixtures ! In that case they will be transformed into fixtures and injected as `fixture_ref` in the parametrization. Fixes [#56](https://github.com/smarie/python-pytest-cases/issues/56).


**Test functions**

New `@parametrize_with_cases(argnames, cases, ...)` decorator to replace `@cases_data` (deprecated):

- Aligned with `pytest` behaviour:

- now `argnames` can contain several names, and the cases are unpacked automatically. **Less boilerplate code**: no need to perform a `case.get()` in the test anymore !

@parametrize_with_cases("a,b")
def test_foo(a, b):
# use a and b directly !
...


- cases are unpacked at test *setup* time, so *the clock does not run while the case is created* - in case you use `pytest-harvest` to collect the timings.

- `@parametrize_with_cases` can be used on test functions *as well as fixture functions* (it was already the case in v1)


- **A single `cases` argument** to indicate the cases, wherever they come from:

- Default (`cases=AUTO`) *automatically looks for cases in the associated case module* named `test_xxx_cases.py`. Users can easily switch to alternate pattern `cases_xxx.py` with `cases=AUTO2`. Fixes [#91](https://github.com/smarie/python-pytest-cases/issues/91).

- *Cases can sit inside a class*, which makes it much more convenient to organize when they sit in the same file than the tests. Fixes [#93](https://github.com/smarie/python-pytest-cases/issues/93).

- an explicit sequence can be provided, *it can mix all kind of sources*: functions, classes, modules, and *module names as strings* (even relative ones!).

@parametrize_with_cases("a", cases=(CasesClass, '.my_extra_cases'))
def test_foo(a):
...

**Misc / pytest goodies**

- `parametrize_plus` now raises an explicit error message when the user makes a mistake with the argnames. Fixes [#105](https://github.com/smarie/python-pytest-cases/issues/105).

- `parametrize_plus` now provides an alternate way to pass argnames, argvalues and ids. Fixes [#106](https://github.com/smarie/python-pytest-cases/issues/106).

- New aliases for readability: `fixture` for `fixture_plus`, and`parametrize` for `parametrize_plus` (both aliases will coexist with the old names). Fixes [#107](https://github.com/smarie/python-pytest-cases/issues/107).

- New pytest goodie `assert_exception` that can be used as a context manager. Fixes [#104](https://github.com/smarie/python-pytest-cases/issues/104).

- More readable error messages when `lazy_value` does not return the same number of argvalues than expected by the `parametrize`.

- Any error message associated to a `lazy_value` function call is not caught and hidden anymore but is emitted to the user.

- Fixed issue with `lazy_value` when a single mark is passed in the constructor.

- `lazy_value` used as a tuple for several arguments now have a correct id generated even in old pytest version 2.

### 1.17.0 - `lazy_value` improvements + annoying warnings suppression

- `lazy_value` are now resolved at pytest `setup` stage, not pytest `call` stage. This is important for execution time recorded in the reports (see also `pytest-harvest` plugin). Fixes [#102](https://github.com/smarie/python-pytest-cases/issues/102)
Expand Down
Binary file added docs/imgs/1_files_overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/imgs/2_class_overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/imgs/source.pptx
Binary file not shown.
544 changes: 282 additions & 262 deletions docs/index.md

Large diffs are not rendered by default.

6 changes: 1 addition & 5 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,7 @@ docs_dir: .
site_dir: ../site
nav:
- Home: index.md
- Usage details:
- Overview: usage.md
- Basics: usage/basics.md
- Intermediate: usage/intermediate.md
- Advanced: usage/advanced.md
- pytest goodies: pytest_goodies.md
- API reference: api_reference.md
- Changelog: changelog.md
theme: material # readthedocs mkdocs
Expand Down
232 changes: 232 additions & 0 deletions docs/pytest_goodies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
# `pytest` Goodies

Many `pytest` features were missing to make `pytest_cases` work with such a "no-boilerplate" experience. Many of these can be of interest to the general `pytest` audience, so they are exposed in the public API.


## `@fixture`

`@fixture` is similar to `pytest.fixture` but without its `param` and `ids` arguments. Instead, it is able to pick the parametrization from `@pytest.mark.parametrize` marks applied on fixtures. This makes it very intuitive for users to parametrize both their tests and fixtures. As a bonus, its `name` argument works even in old versions of pytest (which is not the case for `fixture`).

Finally it now supports unpacking, see [unpacking feature](#unpack_fixture-unpack_into).

!!! note "`@fixture` deprecation if/when `@pytest.fixture` supports `@pytest.mark.parametrize`"
The ability for pytest fixtures to support the `@pytest.mark.parametrize` annotation is a feature that clearly belongs to `pytest` scope, and has been [requested already](https://github.com/pytest-dev/pytest/issues/3960). It is therefore expected that `@fixture` will be deprecated in favor of `@pytest_fixture` if/when the `pytest` team decides to add the proposed feature. As always, deprecation will happen slowly across versions (at least two minor, or one major version update) so as for users to have the time to update their code bases.

## `unpack_fixture` / `unpack_into`

In some cases fixtures return a tuple or a list of items. It is not easy to refer to a single of these items in a test or another fixture. With `unpack_fixture` you can easily do it:

```python
import pytest
from pytest_cases import unpack_fixture, fixture

@fixture
@pytest.mark.parametrize("o", ['hello', 'world'])
def c(o):
return o, o[0]

a, b = unpack_fixture("a,b", c)

def test_function(a, b):
assert a[0] == b
```

Note that you can also use the `unpack_into=` argument of `@fixture` to do the same thing:

```python
import pytest
from pytest_cases import fixture

@fixture(unpack_into="a,b")
@pytest.mark.parametrize("o", ['hello', 'world'])
def c(o):
return o, o[0]

def test_function(a, b):
assert a[0] == b
```

And it is also available in `fixture_union`:

```python
import pytest
from pytest_cases import fixture, fixture_union

@fixture
@pytest.mark.parametrize("o", ['hello', 'world'])
def c(o):
return o, o[0]

@fixture
@pytest.mark.parametrize("o", ['yeepee', 'yay'])
def d(o):
return o, o[0]

fixture_union("c_or_d", [c, d], unpack_into="a, b")

def test_function(a, b):
assert a[0] == b
```

## `param_fixture[s]`

If you wish to share some parameters across several fixtures and tests, it might be convenient to have a fixture representing this parameter. This is relatively easy for single parameters, but a bit harder for parameter tuples.

The two utilities functions `param_fixture` (for a single parameter name) and `param_fixtures` (for a tuple of parameter names) handle the difficulty for you:

```python
import pytest
from pytest_cases import param_fixtures, param_fixture

# create a single parameter fixture
my_parameter = param_fixture("my_parameter", [1, 2, 3, 4])

@pytest.fixture
def fixture_uses_param(my_parameter):
...

def test_uses_param(my_parameter, fixture_uses_param):
...

# -----
# create a 2-tuple parameter fixture
arg1, arg2 = param_fixtures("arg1, arg2", [(1, 2), (3, 4)])

@pytest.fixture
def fixture_uses_param2(arg2):
...

def test_uses_param2(arg1, arg2, fixture_uses_param2):
...
```

You can mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test).

## `fixture_union`

As of `pytest` 5, it is not possible to create a "union" fixture, i.e. a parametrized fixture that would first take all the possible values of fixture A, then all possible values of fixture B, etc. Indeed all fixture dependencies (a.k.a. "closure") of each test node are grouped together, and if they have parameters a big "cross-product" of the parameters is done by `pytest`.

The topic has been largely discussed in [pytest-dev#349](https://github.com/pytest-dev/pytest/issues/349) and a [request for proposal](https://docs.pytest.org/en/latest/proposals/parametrize_with_fixtures.html) has been finally made.

`fixture_union` is an implementation of this proposal. It is also used by `parametrize` to support `fixture_ref` in parameter values, see [below](#parametrize).

```python
from pytest_cases import fixture, fixture_union

@fixture
def first():
return 'hello'

@fixture(params=['a', 'b'])
def second(request):
return request.param

# c will first take all the values of 'first', then all of 'second'
c = fixture_union('c', [first, second])

def test_basic_union(c):
print(c)
```

yields

```
<...>::test_basic_union[c_is_first] hello PASSED
<...>::test_basic_union[c_is_second-a] a PASSED
<...>::test_basic_union[c_is_second-b] b PASSED
```

As you can see the ids of union fixtures are slightly different from standard ids, so that you can easily understand what is going on. You can change this feature with `ìdstyle`, see [API documentation](./api_reference.md#fixture_union) for details.

You can mark any of the alternatives with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test).

Fixture unions also support unpacking with the `unpack_into` argument, see [unpacking feature](#unpack_fixture-unpack_into).

Fixture unions are a **major change** in the internal pytest engine, as fixture closures (the ordered set of all fixtures required by a test node to run - directly or indirectly) now become trees where branches correspond to alternative paths taken in the "unions", and leafs are the alternative fixture closures. This feature has been tested in very complex cases (several union fixtures, fixtures that are not selected by a given union but that is requested by the test function, etc.). But if you find some strange behaviour don't hesitate to report it in the [issues](https://github.com/smarie/python-pytest-cases/issues) page !

**IMPORTANT** if you do not use `@fixture` but only `@pytest.fixture`, then you will see that your fixtures are called even when they are not used, with a parameter `NOT_USED`. This symbol is automatically ignored if you use `@fixture`, otherwise you have to handle it. Alternatively you can use `@ignore_unused` on your fixture function.

!!! note "fixture unions vs. cases"
If you're familiar with `pytest-cases` already, you might note that `@cases_data` is not so different than a fixture union: we do a union of all case functions. If one day union fixtures are directly supported by `pytest`, we will probably refactor this lib to align all the concepts.


## `@parametrize`

`@parametrize` is a replacement for `@pytest.mark.parametrize` with many additional features to make the most of parametrization. See [API reference](./api_reference.md#parametrize) for details about all the new features. In particular it allows you to include references to fixtures and to value-generating functions in the parameter values.

- Simply use `fixture_ref(<fixture>)` in the parameter values, where `<fixture>` can be the fixture name or fixture function.
- if you do not wish to create a fixture, you can also use `lazy_value(<function>)`
- Note that when parametrizing several argnames, both `fixture_ref` and `lazy_value` can be used *as* the tuple, or *in* the tuple. Several `fixture_ref` and/or `lazy_value` can be used in the same tuple, too.

For example, with a single argument:

```python
import pytest
from pytest_cases import parametrize, fixture, fixture_ref, lazy_value

@pytest.fixture
def world_str():
return 'world'

def whatfun():
return 'what'

@fixture
@parametrize('who', [fixture_ref(world_str),
'you'])
def greetings(who):
return 'hello ' + who

@parametrize('main_msg', ['nothing',
fixture_ref(world_str),
lazy_value(whatfun),
fixture_ref(greetings)])
@pytest.mark.parametrize('ending', ['?', '!'])
def test_prints(main_msg, ending):
print(main_msg + ending)
```

yields the following

```bash
> pytest -s -v
collected 10 items
test_prints[main_msg_is_nothing-?] PASSED [ 10%]nothing?
test_prints[main_msg_is_nothing-!] PASSED [ 20%]nothing!
test_prints[main_msg_is_world_str-?] PASSED [ 30%]world?
test_prints[main_msg_is_world_str-!] PASSED [ 40%]world!
test_prints[main_msg_is_whatfun-?] PASSED [ 50%]what?
test_prints[main_msg_is_whatfun-!] PASSED [ 60%]what!
test_prints[main_msg_is_greetings-who_is_world_str-?] PASSED [ 70%]hello world?
test_prints[main_msg_is_greetings-who_is_world_str-!] PASSED [ 80%]hello world!
test_prints[main_msg_is_greetings-who_is_you-?] PASSED [ 90%]hello you?
test_prints[main_msg_is_greetings-who_is_you-!] PASSED [100%]hello you!
```

You can also mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test).

As you can see in the example above, the default ids are a bit more explicit than usual when you use at least one `fixture_ref`. This is because the parameters need to be replaced with a fixture union that will "switch" between alternative groups of parameters, and the appropriate fixtures referenced. As opposed to `fixture_union`, the style of these ids is not configurable for now, but feel free to propose alternatives in the [issues page](https://github.com/smarie/python-pytest-cases/issues). Note that this does not happen if you only use `lazy_value`s, as they do not require to create a fixture union behind the scenes.

Another consequence of using `fixture_ref` is that the priority order of the parameters, relative to other standard `pytest.mark.parametrize` parameters that you would place on the same function, will get impacted. You may solve this by replacing your other `@pytest.mark.parametrize` calls with `param_fixture`s so that all the parameters are fixtures (see [above](#param_fixtures).)

## passing a `hook`

As per version `1.14`, all the above functions now support passing a `hook` argument. This argument should be a callable. It will be called everytime a fixture is about to be created by `pytest_cases` on your behalf. The fixture function is passed as the argument of the hook, and the hook should return it as the result.

You can use this fixture to better understand which fixtures are created behind the scenes, and also to decorate the fixture functions before they are created. For example you can use `hook=saved_fixture` (from [`pytest-harvest`](https://smarie.github.io/python-pytest-harvest/)) in order to save the created fixtures in the fixture store.

## `assert_exception`

`assert_exception` context manager is an alternative to `pytest.raises` to check exceptions in your tests. You can either check type, instance equality, repr string pattern, or use custom validation functions. See [API reference](./api_reference.md).

## `--with-reorder`

`pytest` postprocesses the order of the collected items in order to optimize setup/teardown of session, module and class fixtures. This optimization algorithm happens at the `pytest_collection_modifyitems` stage, and is still under improvement, as can be seen in [pytest#3551](https://github.com/pytest-dev/pytest/pull/3551), [pytest#3393](https://github.com/pytest-dev/pytest/issues/3393), [#2846](https://github.com/pytest-dev/pytest/issues/2846)...

Besides other plugins such as [pytest-reorder](https://github.com/not-raspberry/pytest_reorder) can modify the order as well.

This new commandline is a goodie to change the reordering:

* `--with-reorder normal` is the default behaviour: it lets pytest and all the plugins execute their reordering in each of their `pytest_collection_modifyitems` hooks, and simply does not interact

* `--with-reorder skip` allows you to restore the original order that was active before `pytest_collection_modifyitems` was initially called, thus not taking into account any reordering done by pytest or by any of its plugins.
10 changes: 0 additions & 10 deletions docs/usage.md

This file was deleted.

Loading