diff --git a/docs/api_reference.md b/docs/api_reference.md index 49b8ed51..8b3e82ac 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -2,252 +2,124 @@ In general, using `help(symbol)` is the recommended way to get the latest documentation. In addition, this page provides an overview of the various elements in this package. +## 1 - Case functions -## 1 - On case functions side +As explained in the [documentation](index.md), case functions have no requirement anymore, and starting from version 2.0.0 of `pytest_cases` they can be parametrized with the usual `@pytest.mark.parametrize` or its improvement [`@parametrize`](#parametrize). Therefore the only remaining decorator is the optional `@case` decorator: -### `CaseData` type hint - -A proposed standard type hint for the case functions outputs. `CaseData = Tuple[Given, ExpectedNormal, ExpectedError]` where +### `@case` ```python -Given = Any -"""The input(s) for the test. It can be anything""" - -ExpectedNormal = Optional[Any] -"""The expected test results in case success is expected, or None if this test - should fail""" - -ExpectedError = Optional[Union[Type[Exception], - Exception, - Callable[[Exception], Optional[bool]]]] -"""The expected error in case failure is expected, or None if the test should -succeed. It is proposed that expected error can be defined as an exception -type, an exception instance, or an exception validation function""" +@case(id=None, # type: str # noqa + tags=None, # type: Union[Any, Iterable[Any]] + marks=(), # type: Union[MarkDecorator, Iterable[MarkDecorator]] + ) ``` -### `@case_name` - -`@case_name(name: str)` - -Decorator to override the name of a case function. The new name will be used instead of the function name, in test names. +Optional decorator for case functions so as to customize some information. ```python -@case_name('simple_case') -def case_simple(): - ... +@case(id='hey') +def case_hi(): + return 1 ``` -### `@case_tags` - -`@case_tags(*tags: Any)` - -Decorator to tag a case function with a list of tags. These tags can then be used in the `@cases_data` test function decorator to filter cases within the selected module(s). - **Parameters:** - - `tags`: a list of tags that may be used to filter the case. Tags can be anything (string, objects, types, functions...) - - -### `@test_target` - -`@test_target(target: Any)` - -A simple decorator to declare that a case function is associated with a particular target. - -```python -@test_target(int) -def case_to_test_int(): - ... -``` - -This is actually an alias for `@case_tags(target)`, that some users may find a bit more readable. + - `id`: the custom pytest id that should be used when this case is active. Replaces the deprecated `@case_name` decorator from v1. If no id is provided, the id is generated from case functions by removing their prefix, see [`@parametrize_with_cases(prefix='case_')`](#parametrize_with_cases). + - `tags`: custom tags to be used for filtering in [`@parametrize_with_cases(has_tags)`](#parametrize_with_cases). Replaces the deprecated `@case_tags` and `@target` decorators. + + - `marks`: optional pytest marks to add on the case. Note that decorating the function directly with the mark also works, and if marks are provided in both places they are merged. -### `@cases_generator` -`@cases_generator(name_template: str, lru_cache: bool=False, **param_ranges)` +## 2 - Cases collection -Decorator to declare a case function as being a cases generator. `param_ranges` should be a named list of parameter ranges to explore to generate the cases. - -The decorator will use `itertools.product` to create a cartesian product of the named parameter ranges, and create a case for each combination. When the case function will be called for a given combination, the corresponding parameters will be passed to the decorated function. +### `@parametrize_with_cases` ```python -@cases_generator("test with i={i}", i=range(10)) -def case_10_times(i): - ''' Generates 10 cases ''' - ins = dict(a=i, b=i+1) - outs = i+1, i+2 - return ins, outs, None +@parametrize_with_cases(argnames: str, + cases: Union[Callable, Type, ModuleRef] = AUTO, + prefix: str = 'case_', + glob:str = None, + has_tag: Union[str, Iterable[str]]=None, + filter: Callable = None, + **kwargs + ) ``` -**Parameters:** - - - `name_template`: a name template, that will be transformed into the case name using `name_template.format(**params)` for each case, where params is the dictionary of parameter values for this generated case. - - `lru_cache`: a boolean (default False) indicating if the generated cases should be cached. This is identical to decorating the function with an additional `@lru_cache(maxsize=n)` where n is the total number of generated cases. - - `param_ranges`: named parameters and for each of them the list of values to be used to generate cases. For each combination of values (a cartesian product is made) the parameters will be passed to the underlying function so they should have names the underlying function can handle. - - -### `MultipleStepsCaseData` type hint - -You may wish to use this type hint instead of `CaseData` when your case functions may return dictionaries of given/expected_normal/expected_error. - -`MultipleStepsCaseData = Tuple[Union[Given, Dict[Any, Given]], - Union[ExpectedNormal, Dict[Any, ExpectedNormal]], - Union[ExpectedError, Dict[Any, ExpectedError]]]` - -## 2 - On test functions side - -### `@cases_fixture` - -`@cases_fixture(cases=None, module=None, case_data_argname='case_data', has_tag=None, filter=None)` - -Decorates a function so that it becomes a parametrized fixture. +A decorator for test functions or fixtures, to parametrize them based on test cases. It works similarly to [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html): argnames represent a coma-separated string of arguments to inject in the decorated test function or fixture. The argument values (`argvalues` in [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html)) are collected from the various case functions found according to `cases`, and injected as lazy values so that the case functions are called just before the test or fixture is executed. -The fixture will be automatically parametrized with all cases listed in module `module`, or with -all cases listed explicitly in `cases`. +By default (`cases=AUTO`) the list of test cases is automatically drawn from the python module file named `test__cases.py` where `test_` is the current module name. An alternate naming convention `cases_.py` can be used by setting `cases=AUTO2`. -Using it with a non-None `module` argument is equivalent to - * extracting all cases from `module` - * then decorating your function with @pytest.fixture(params=cases) with all the cases +Finally, the `cases` argument also accepts an explicit case function, cases-containing class, module or module name; or a list of such elements. Note that both absolute and relative module names are suported. -So +Note that `@parametrize_with_cases` collection and parameter creation steps are strictly equivalent to [`get_all_cases`](#get_all_cases) + [`get_parametrize_args`](#get_parametrize_args). This can be handy for debugging purposes. ```python -from pytest_cases import cases_fixture, CaseData +# Collect all cases +cases_funs = get_all_cases(f, cases=cases, prefix=prefix, + glob=glob, has_tag=has_tag, filter=filter) -# import the module containing the test cases -import test_foo_cases - -@cases_fixture(module=test_foo_cases) -def foo_fixture(case_data: CaseData): - ... -``` - -is equivalent to: - -```python -import pytest -from pytest_cases import get_all_cases, get_pytest_parametrize_args - -# import the module containing the test cases -import test_foo_cases - -# manually list the available cases -cases = get_all_cases(module=test_foo_cases) - -# transform into required arguments for pytest (applying the pytest marks if needed) -marked_cases, cases_ids = get_pytest_parametrize_args(cases) - -# parametrize the fixture manually -@pytest.fixture(params=marked_cases, ids=cases_ids) -def foo_fixture(request): - case_data = request.param # type: CaseData - ... +# Transform the various functions found +argvalues = get_parametrize_args(cases_funs) ``` **Parameters** - - `case_data_argname`: the optional name of the function parameter that should receive the `CaseDataGetter` object. Default is `case_data`. - - Other parameters (cases, module, has_tag, filter) can be used to perform explicit listing, or filtering, of cases to include. See `get_all_cases()` for details about them. + - `argnames`: same than in `@pytest.mark.parametrize` + + - `cases`: a case function, a class containing cases, a module object or a module name string (relative module names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module. `AUTO` (default) means that the module named `test__cases.py` will be loaded, where `test_.py` is the module file of the decorated function. `AUTO2` allows you to use the alternative naming scheme `case_.py`. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag` are selected, including those functions nested in classes following naming pattern `*Case*`. When classes are explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern. -### `@cases_data` + - `prefix`: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to denote different kind of cases, for example 'data_', 'algo_', 'user_', etc. -`@cases_data(cases=None, module=None, case_data_argname='case_data', has_tag=None, filter=None)` + - `glob`: an optional glob-like pattern for case ids, for example "*_success" or "*_failure". Note that this is applied on the case id, and therefore if it is customized through [`@case(id=...)`](#case) it should be taken into account. -Decorates a test function so as to automatically parametrize it with all cases listed in module `module`, or with all cases listed explicitly in `cases`. + - `has_tag`: a single tag or a tuple, set, list of tags that should be matched by the ones set with the [`@case`](#case) decorator on the case function(s) to be selected. -Using it with a non-None `module` argument is equivalent to + - `filter`: a callable receiving the case function and returning True or a truth value in case the function needs to be selected. - * extracting all cases from `module` - * then decorating your function with `@pytest.mark.parametrize` with all the cases -So +### `get_all_cases` ```python -from pytest_cases import cases_data - -# import the module containing the test cases -import test_foo_cases - -@cases_data(module=test_foo_cases) -def test_foo(case_data): - ... +def get_all_cases(parametrization_target: Callable, + cases: Union[Callable, Type, ModuleRef] = None, + prefix: str = 'case_', + glob: str = None, + has_tag: Union[str, Iterable[str]] = None, + filter: Callable[[Callable], bool] = None + ) -> List[Callable]: ``` - -is equivalent to: -```python -import pytest -from pytest_cases import get_all_cases, get_pytest_parametrize_args +Lists all desired cases for a given `parametrization_target` (a test function or a fixture). This function may be convenient for debugging purposes. See [`@parametrize_with_cases`](#parametrize_with_cases) for details on the parameters. -# import the module containing the test cases -import test_foo_cases -# manually list the available cases -cases = get_all_cases(module=test_foo_cases) +### `get_parametrize_args` -# transform into required arguments for pytest (applying the pytest marks if needed) -marked_cases, cases_ids = get_pytest_parametrize_args(cases) - -# parametrize the test function manually -@pytest.mark.parametrize('case_data', marked_cases, ids=str) -def test_foo(case_data): - ... +```python +def get_parametrize_args(cases_funs: List[Callable], + ) -> List[Union[lazy_value, fixture_ref]]: ``` -**Parameters** - - - `case_data_argname`: the optional name of the function parameter that should receive the `CaseDataGetter` object. Default is `case_data`. - - Other parameters (cases, module, has_tag, filter) can be used to perform explicit listing, or filtering, of cases to include. See `get_all_cases()` for details about them. - -### `CaseDataGetter` +Transforms a list of cases (obtained from [`get_all_cases`](#get_all_cases)) into a list of argvalues for [`@parametrize`](#parametrize). Each case function `case_fun` is transformed into one or several [`lazy_value`](#lazy_value)(s) or a [`fixture_ref`](#fixture_ref): -A proxy for a test case. Instances of this class are created by `@cases_data` or `get_all_cases`. It provides a single method: + - If `case_fun` requires at least on fixture, a fixture will be created if not yet present, and a `fixture_ref` will be returned. -`get(self, *args, **kwargs) -> Union[CaseData, Any]` - -This method calls the actual underlying case function with arguments propagation, and returns the result. The case functions can use the proposed standard `CaseData` type hint and return outputs matching this type hint, but this is not mandatory. - -### `unfold_expected_err` - -'Unfolds' the expected error `expected_e` to return a tuple of - - - expected error type - - expected error instance - - error validation callable - -If `expected_e` is an exception type, returns `expected_e, None, None`. -If `expected_e` is an exception instance, returns `type(expected_e), expected_e, None`. -If `expected_e` is an exception validation function, returns `Exception, None, expected_e`. - -**Parameters:** - - - `expected_e`: an `ExpectedError`, that is, either an exception type, an exception instance, or an exception validation function - - -### `get_all_cases` - -`get_all_cases(cases=None, module=None, this_module_object=None, has_tag=None, filter=None) -> List[CaseDataGetter]` - -Lists all desired cases for a given user query. This function may be convenient for debugging purposes. - - -**Parameters:** - - - `cases`: a single case or a hardcoded list of cases to use. Only one of `cases` and `module` should be set. - - `module`: a module or a hardcoded list of modules to use. You may use `THIS_MODULE` to indicate that the module is the current one. Only one of `cases` and `module` should be set. - - `this_module_object`: any variable defined in the module of interest, for example a function. It is used to find "this module", when `module` contains `THIS_MODULE`. - - `has_tag`: an optional tag used to filter the cases in the `module`. Only cases with the given tag will be selected. - - `filter`: an optional filtering function taking as an input a list of tags associated with a case, and returning a boolean indicating if the case should be selected. It will be used to filter the cases in the `module`. It both `has_tag` and `filter` are set, both will be applied in sequence. + - If `case_fun` is a parametrized case, one `lazy_value` with a partialized version will be created for each parameter combination. + - Otherwise, `case_fun` represents a single case: in that case a single `lazy_value` is returned. ## 3 - Pytest goodies -### `@fixture_plus` +### `@fixture` ```python -fixture_plus(scope="function", autouse=False, name=None, - unpack_into=None, hook=None, **kwargs) +@fixture(scope: str = "function", + autouse: bool = False, + name: str = None, + unpack_into: Iterable[str] = None, + hook: Callable = None, + **kwargs) ``` Identical to `@pytest.fixture` decorator, except that @@ -269,12 +141,32 @@ As a consequence it does not support the `params` and `ids` arguments anymore. ### `unpack_fixture` -`unpack_fixture(argnames, fixture, hook=None) -> Tuple[]` +```python +def unpack_fixture(argnames: str, + fixture: Union[str, Callable], + hook: Callable = None + ) -> Tuple[] +``` Creates several fixtures with names `argnames` from the source `fixture`. Created fixtures will correspond to elements unpacked from `fixture` in order. For example if `fixture` is a tuple of length 2, `argnames="a,b"` will create two fixtures containing the first and second element respectively. The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. `a, b = unpack_fixture('a,b', 'c')`. +```python +import pytest +from pytest_cases import unpack_fixture, fixture_plus + +@fixture_plus +@pytest.mark.parametrize("o", ['hello', 'world']) +def c(o): + return o, o[0] + +a, b = unpack_fixture("a,b", c) + +def test_function(a, b): + assert a[0] == b +``` + **Parameters** - **argnames**: same as `@pytest.mark.parametrize` `argnames`. @@ -286,10 +178,15 @@ The created fixtures are automatically registered into the callers' module, but ### `fixture_union` ```python -fixture_union(name, fixtures, - scope="function", idstyle='explicit', ids=fixture_alternative_to_str, - unpack_into=None, autouse=False, hook=None, **kwargs) - -> +def fixture_union(name: str, + fixtures: Iterable[Union[str, Callable]], + scope: str = "function", + idstyle: Optional[str] = 'explicit', + ids: Union[Callable, List[str]] = None, + unpack_into: Iterable[str] = None, + autouse: bool = False, + hook: Callable = None, + **kwargs) -> ``` Creates a fixture that will take all values of the provided fixtures in order. That fixture is automatically registered into the callers' module, but you may wish to assign it to a variable for convenience. In that case make sure that you use the same name, e.g. `a = fixture_union('a', ['b', 'c'])` @@ -317,9 +214,14 @@ The style of test ids corresponding to the union alternatives can be changed wit ### `param_fixtures` ```python -param_fixtures(argnames, argvalues, - autouse=False, ids=None, scope="function", hook=None, **kwargs) - -> Tuple[] +def param_fixtures(argnames: str, + argvalues: Iterable[Any], + autouse: bool = False, + ids: Union[Callable, List[str]] = None, + scope: str = "function", + hook: Callable = None, + debug: bool = False, + **kwargs) -> Tuple[] ``` Creates one or several "parameters" fixtures - depending on the number or coma-separated names in `argnames`. The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. `p, q = param_fixtures('p,q', [(0, 1), (2, 3)])`. @@ -327,6 +229,21 @@ Creates one or several "parameters" fixtures - depending on the number or coma-s Note that the `(argnames, argvalues, ids)` signature is similar to `@pytest.mark.parametrize` for consistency, see [pytest doc on parametrize](https://docs.pytest.org/en/latest/reference.html?highlight=pytest.param#pytest-mark-parametrize). +```python +import pytest +from pytest_cases import param_fixtures, param_fixture + +# create a 2-tuple parameter fixture +arg1, arg2 = param_fixtures("arg1, arg2", [(1, 2), (3, 4)]) + +@pytest.fixture +def fixture_uses_param2(arg2): + ... + +def test_uses_param2(arg1, arg2, fixture_uses_param2): + ... +``` + **Parameters:** - `argnames`: same as `@pytest.mark.parametrize` `argnames`. @@ -348,22 +265,66 @@ param_fixture(argname, argvalues, Identical to `param_fixtures` but for a single parameter name, so that you can assign its output to a single variable. -### `@parametrize_plus` +### `@parametrize` ```python -parametrize_plus(argnames, argvalues, - indirect=False, ids=None, idstyle='explicit', - scope=None, hook=None, debug=False, **kwargs) +@parametrize_plus(argnames: str=None, + argvalues: Iterable[Any]=None, + indirect: bool = False, + ids: Union[Callable, List[str]] = None, + idstyle: str = 'explicit', + idgen: Union[str, Callable] = _IDGEN, + scope: str = None, + hook: Callable = None, + debug: bool = False, + **args) ``` -Equivalent to `@pytest.mark.parametrize` but also supports new possibilities in argvalues: +Equivalent to `@pytest.mark.parametrize` but also supports - - one can include references to fixtures with `fixture_ref()` where can be the fixture name or fixture function. When such a fixture reference is detected in the argvalues, a new function-scope "union" fixture will be created with a unique name, and the test function will be wrapped so as to be injected with the correct parameters from this fixture. Special test ids will be created to illustrate the switching between the various normal parameters and fixtures. You can see debug print messages about all fixtures created using `debug=True` +**New alternate style for argnames/argvalues**. One can also use `**args` to pass additional `{argnames: argvalues}` in the same parametrization call. This can be handy in combination with `idgen` to master the whole id template associated with several parameters. Note that you can pass coma-separated argnames too, by de-referencing a dict: e.g. `**{'a,b': [(0, True), (1, False)], 'c': [-1, 2]}`. - - one can include lazy argvalues with `lazy_value(, [id=..., marks=...])`. A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. +**New alternate style for ids**. One can use `idgen` instead of `ids`. `idgen` can be a callable receiving all parameters at once (`**args`) and returning an id ; or it can be a string template using the new-style string formatting where the argnames can be used as variables (e.g. `idgen=lambda **args: "a={a}".format(**args)` or `idgen="my_id where a={a}"`). The special `idgen=AUTO` symbol can be used to generate a default string template equivalent to `lambda **args: "-".join("%s=%s" % (n, v) for n, v in args.items())`. This is enabled by default if you use the alternate style for argnames/argvalues (e.g. if `len(args) > 0`). + +**New possibilities in argvalues**: + + - one can include *references to fixtures* with [`fixture_ref()`](#fixture_ref) where can be the fixture name or fixture function. When such a fixture reference is detected in the argvalues, a new function-scope "union" fixture will be created with a unique name, and the test function will be wrapped so as to be injected with the correct parameters from this fixture. Special test ids will be created to illustrate the switching between the various normal parameters and fixtures. You can see debug print messages about all fixtures created using `debug=True` + + - one can include lazy argvalues with [`lazy_value(, [id=..., marks=...])`](#lazy_value). A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. Both `fixture_ref` and `lazy_value` can be used to represent a single argvalue, or a whole tuple of argvalues when there are several argnames. Several of them can be used in a tuple. Finally, `pytest.param` is supported even when there are `fixture_ref` and `lazy_value`. Here as for all functions above, an optional `hook` can be passed, to apply on each fixture function that is created during this call. The hook function will be called everytime a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use `saved_fixture` from `pytest-harvest` as a hook in order to save all such created fixtures in the fixture store. + +### `lazy_value` + +```python +def lazy_value(valuegetter: Callable[[], Any], + id: str = None, + marks: Union[Any, Sequence[Any]] = () + ) -> LazyValue +``` + +A reference to a value getter (an argvalue-providing callable), to be used in [`@parametrize`](#parametrize). + +A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. + +Note that a `lazy_value` can be included in a `pytest.param` without problem. In that case the id defined by `pytest.param` will take precedence over the one defined in `lazy_value` if any. The marks, however, will all be kept wherever they are defined. + +**Parameters** + + - `valuegetter`: a callable without mandatory arguments + - `id`: an optional id. Otherwise `valuegetter.__name__` will be used by default + - `marks`: optional marks. `valuegetter` marks will also be preserved. + + +### `fixture_ref` + +```python +def fixture_ref(fixture: Union[str, Fixture] + ) +``` + +A reference to a fixture to be used with [`@parametrize`](#parametrize). Create it with `fixture_ref()` where can be the fixture name or actual fixture function. diff --git a/docs/changelog.md b/docs/changelog.md index 5fbcb5e7..579b4077 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -1,5 +1,67 @@ # Changelog +### 2.0.0 - Less boilerplate & full `pytest` alignment ! + +**Case functions** + + - New `@case(id=None, tags=(), marks=())` decorator to replace `@case_name`, `@case_tags` and `@test_target` (all deprecated): a single simple way to customize all aspects of a case function. Also, `target` disappears from the picture as it was just a tag like others - this could be misleading. + + - `@cases_generator` is now deprecated and replaced with `@parametrize` : now, cases can be parametrized exactly the same way than tests. This includes the ability to put marks on the whole or on some specific parameter values. `@parametrize_plus` has been renamed `@parametrize` for readability. It was also improved in order to support the alternate parametrization mode that was previously offered by `@cases_generator`. That way, users will be able to choose the style of their choice. Fixes [#57](https://github.com/smarie/python-pytest-cases/issues/57) and [#106](https://github.com/smarie/python-pytest-cases/issues/106). + + - Since `@cases_generat`Marks can now + + - Now case functions can require fixtures ! In that case they will be transformed into fixtures and injected as `fixture_ref` in the parametrization. Fixes [#56](https://github.com/smarie/python-pytest-cases/issues/56). + + +**Test functions** + +New `@parametrize_with_cases(argnames, cases, ...)` decorator to replace `@cases_data` (deprecated): + + - Aligned with `pytest` behaviour: + + - now `argnames` can contain several names, and the cases are unpacked automatically. **Less boilerplate code**: no need to perform a `case.get()` in the test anymore ! + + @parametrize_with_cases("a,b") + def test_foo(a, b): + # use a and b directly ! + ... + + + - cases are unpacked at test *setup* time, so *the clock does not run while the case is created* - in case you use `pytest-harvest` to collect the timings. + + - `@parametrize_with_cases` can be used on test functions *as well as fixture functions* (it was already the case in v1) + + + - **A single `cases` argument** to indicate the cases, wherever they come from: + + - Default (`cases=AUTO`) *automatically looks for cases in the associated case module* named `test_xxx_cases.py`. Users can easily switch to alternate pattern `cases_xxx.py` with `cases=AUTO2`. Fixes [#91](https://github.com/smarie/python-pytest-cases/issues/91). + + - *Cases can sit inside a class*, which makes it much more convenient to organize when they sit in the same file than the tests. Fixes [#93](https://github.com/smarie/python-pytest-cases/issues/93). + + - an explicit sequence can be provided, *it can mix all kind of sources*: functions, classes, modules, and *module names as strings* (even relative ones!). + + @parametrize_with_cases("a", cases=(CasesClass, '.my_extra_cases')) + def test_foo(a): + ... + +**Misc / pytest goodies** + + - `parametrize_plus` now raises an explicit error message when the user makes a mistake with the argnames. Fixes [#105](https://github.com/smarie/python-pytest-cases/issues/105). + + - `parametrize_plus` now provides an alternate way to pass argnames, argvalues and ids. Fixes [#106](https://github.com/smarie/python-pytest-cases/issues/106). + + - New aliases for readability: `fixture` for `fixture_plus`, and`parametrize` for `parametrize_plus` (both aliases will coexist with the old names). Fixes [#107](https://github.com/smarie/python-pytest-cases/issues/107). + + - New pytest goodie `assert_exception` that can be used as a context manager. Fixes [#104](https://github.com/smarie/python-pytest-cases/issues/104). + + - More readable error messages when `lazy_value` does not return the same number of argvalues than expected by the `parametrize`. + + - Any error message associated to a `lazy_value` function call is not caught and hidden anymore but is emitted to the user. + + - Fixed issue with `lazy_value` when a single mark is passed in the constructor. + + - `lazy_value` used as a tuple for several arguments now have a correct id generated even in old pytest version 2. + ### 1.17.0 - `lazy_value` improvements + annoying warnings suppression - `lazy_value` are now resolved at pytest `setup` stage, not pytest `call` stage. This is important for execution time recorded in the reports (see also `pytest-harvest` plugin). Fixes [#102](https://github.com/smarie/python-pytest-cases/issues/102) diff --git a/docs/imgs/1_files_overview.png b/docs/imgs/1_files_overview.png new file mode 100644 index 00000000..6dcf1f45 Binary files /dev/null and b/docs/imgs/1_files_overview.png differ diff --git a/docs/imgs/2_class_overview.png b/docs/imgs/2_class_overview.png new file mode 100644 index 00000000..dbf22aae Binary files /dev/null and b/docs/imgs/2_class_overview.png differ diff --git a/docs/imgs/source.pptx b/docs/imgs/source.pptx new file mode 100644 index 00000000..8532f283 Binary files /dev/null and b/docs/imgs/source.pptx differ diff --git a/docs/index.md b/docs/index.md index a675025c..55195e27 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,23 +6,17 @@ [![Documentation](https://img.shields.io/badge/doc-latest-blue.svg)](https://smarie.github.io/python-pytest-cases/) [![PyPI](https://img.shields.io/pypi/v/pytest-cases.svg)](https://pypi.python.org/pypi/pytest-cases/) [![Downloads](https://pepy.tech/badge/pytest-cases)](https://pepy.tech/project/pytest-cases) [![Downloads per week](https://pepy.tech/badge/pytest-cases/week)](https://pepy.tech/project/pytest-cases) [![GitHub stars](https://img.shields.io/github/stars/smarie/python-pytest-cases.svg)](https://github.com/smarie/python-pytest-cases/stargazers) -!!! success "You can now use `pytest.param` in the argvalues provided to `fixture_union`, `param_fixture[s]` and `parametrize_plus`, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test)" +!!! success "Brand new v2, [check the changes](./changelog.md#200---less-boilerplate-and-full-pytest-alignment-unlocking-advanced-features) !" -!!! success "New `lazy_value` feature for parametrize, [check it out](#parametrize_plus) !" - -!!! warning "Test execution order" - Installing pytest-cases now has effects on the order of `pytest` tests execution, even if you do not use its features. One positive side effect is that it fixed [pytest#5054](https://github.com/pytest-dev/pytest/issues/5054). But if you see less desirable ordering please [report it](https://github.com/smarie/python-pytest-cases/issues). +!!! warning "Installing pytest-cases has effects on the order of `pytest` tests execution. Details [here](#installing)" Did you ever think that most of your test functions were actually *the same test code*, but with *different data inputs* and expected results/exceptions ? -`pytest-cases` leverages `pytest` and its great `@pytest.mark.parametrize` decorator, so that you can **separate your test cases from your test functions**. For example with `pytest-cases` you can now write your tests with the following pattern: - - * on one hand, the usual `test_xxxx.py` file containing your test functions - * on the other hand, a new `test_xxxx_cases.py` containing your cases functions + - `pytest-cases` leverages `pytest` and its great `@pytest.mark.parametrize` decorator, so that you can **separate your test cases from your test functions**. -In addition, `pytest-cases` improves `pytest`'s fixture mechanism to support "fixture unions". This is a **major change** in the internal `pytest` engine, unlocking many possibilities such as using fixture references as parameter values in a test function. See [below](#fixture_union). + - In addition, `pytest-cases` improves `pytest`'s fixture mechanism to support "fixture unions". This is a **major change** in the internal `pytest` engine, unlocking many possibilities such as using fixture references as parameter values in a test function. See [here](pytest_goodies.md#fixture_union). -`pytest-cases` is fully compliant with [pytest-steps](https://smarie.github.io/python-pytest-steps/) so you can create test suites with several steps and send each case on the full suite. See [usage page for details](./usage/advanced/#test-suites-several-steps-on-each-case). +`pytest-cases` is fully compliant with [pytest-harvest](https://smarie.github.io/python-pytest-harvest/) and [pytest-steps](https://smarie.github.io/python-pytest-steps/) so you can create test suites with several steps, send each case on the full suite, and monitor the execution times and created artifacts. See also [this example](https://smarie.github.io/pytest-patterns/) (a bit old, needs to be refreshed) showing how to combine the various plugins to create data science benchmarks. ## Installing @@ -30,409 +24,434 @@ In addition, `pytest-cases` improves `pytest`'s fixture mechanism to support "fi > pip install pytest_cases ``` -## Usage - 'Data' cases - -### a- Some code to test +Note: Installing pytest-cases has effects on the order of `pytest` tests execution, even if you do not use its features. One positive side effect is that it fixed [pytest#5054](https://github.com/pytest-dev/pytest/issues/5054). But if you see less desirable ordering please [report it](https://github.com/smarie/python-pytest-cases/issues). -Let's consider the following `foo` function under test: +## Why `pytest-cases` ? + +**`pytest` philosophy** + +Let's consider the following `foo` function under test, located in `example.py`: ```python def foo(a, b): return a + 1, b + 1 ``` -### b- Case functions - -First we create a `test_foo_cases.py` file. This file will contain *test cases generator* functions, that we will call **case functions** for brevity: +If we were using plain `pytest` to test it with various inputs, we would create a `test_foo.py` file and use `@pytest.mark.parametrize`: ```python -def case_two_positive_ints(): - """ Inputs are two positive integers """ - return dict(a=1, b=2) +import pytest +from example import foo -def case_two_negative_ints(): - """ Inputs are two negative integers """ - return dict(a=-1, b=-2) +@pytest.mark.parametrize("a,b", [(1, 2), (-1, -2)]) +def test_foo(a, b): + # check that foo runs correctly and that the result is a tuple. + assert isinstance(foo(a, b), tuple) ``` -In these functions, you will typically either parse some test data files, or generate some simulated test data and expected results. +This is the fastest and most compact thing to do when you have a few number of test cases, that do not require code to generate each test case. -Case functions **do not have any particular requirement**, apart from their names starting with `case_`. They can return anything that is considered useful to run the associated test. +**`pytest` current limitations** +Now imagine that instead of `(1, 2)` and `(-1, -2)` **each** of our test cases -!!! note "Support for pytest marks" - Pytest marks such as `@pytest.mark.skip` can be used on case functions, the corresponding case will be handled according to the expected behaviour (failed if `@pytest.mark.fail`, skipped under condition if `@pytest.mark.skipif`, etc.) + - requires **a few lines of code** to be generated. For example artificial data creation using `numpy` and/or `pandas`: - -### c- Test functions +```python +import numpy as np +import pandas as pd + +# case 1: non-sorted uniformly sampled timeseries with 2 holes +case1 = pd.DataFrame({"datetime": pd.date_range(start='20/1/1', periods=20, + freq='-1d', tz='UTC'), + "data1": np.arange(0, 20), + "data2": np.arange(1, 21), + "data3": np.arange(1, 21)}) +case1.drop([3, 12], inplace=True) +``` + + - requires **documentation** to explain the other developers the intent of that precise test case + + - requires **external resources** (data files on the filesystem, databases...), with a variable number of cases depending on what is available on the resource - but of course not all the cases would come from the same resource, that would be too easy :). -Then, as usual we write our `pytest` functions starting with `test_`, in a `test_foo.py` file: + - requires **a readable `id`**, such as `'uniformly_sampled_nonsorted_with_holes'` for the above example. Of course we *could* use [`pytest.param`](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test) or [`ids=`](https://docs.pytest.org/en/stable/example/parametrize.html#different-options-for-test-ids) but that is "a pain to maintain" according to `pytest` doc (I agree!). Such a design does not feel right as the id is detached from the case. +With standard `pytest` there is no particular pattern to simplify your life here. Investigating a little bit, people usually end up trying to mix parameters and fixtures and asking this kind of question: [so1](https://stackoverflow.com/questions/50231627/python-pytest-unpack-fixture), [so2](https://stackoverflow.com/questions/50482416/use-pytest-lazy-fixture-list-values-as-parameters-in-another-fixture). But by design it is not possible to solve this problem using fixtures, because `pytest` [does not handle "unions" of fixtures](pytest_goodies.md#fixture_union). + +So all in all, the final answer is "you have to do this yourself", and have `pytest` use your handcrafted list of parameters as the list of argvalues in `@pytest.mark.parametrize`. Typically we would end up creating a `get_all_foo_test_cases` function, independently from `pytest`: + ```python -from pytest_cases import cases_data -from example import foo +@pytest.mark.parametrize("a,b", get_all_foo_test_cases()) +def test_foo(a, b): + ... +``` -# import the module containing the test cases -import test_foo_cases +There is also an example in `pytest` doc [with a `metafunc` hook](https://docs.pytest.org/en/stable/example/parametrize.html#a-quick-port-of-testscenarios). -@cases_data(module=test_foo_cases) -def test_foo(case_data): - """ Example unit test that is automatically parametrized with @cases_data """ +The issue with such workarounds is that you can do *anything*. And *anything* is a bit too much: this does not provide any convention / "good practice" on how to organize test cases, which is an open door to developing ad-hoc unreadable or unmaintainable solutions. - # 1- Grab the test case data - inputs = case_data.get() +`pytest_cases` was created to provide an answer to this precise situation. It proposes a simple framework to separate test cases from test functions. The test cases are typically located in a separate "companion" file: - # 2- Use it - foo(**inputs) -``` + - `test_foo.py` is your usual test file containing the test **functions** (named `test_`), + - `test_foo_cases.py` contains the test **cases**, that are also functions (named `case_` or even `_` if you prefer). Note: an alternate file naming style `cases_foo.py` is also available if you prefer it. -*Note: as explained [here](https://smarie.github.io/python-pytest-cases/usage/basics/#cases-in-the-same-file-than-tests), cases can also be located inside the test file.* +![files_overview](./imgs/1_files_overview.png) -As you can see above there are three things that are needed to parametrize a test function with associated case functions: +Test cases can also be provided explicitly, for example in a class container: - * decorate your test function with `@cases_data`, indicating which module contains the cases functions - * add an input argument to your test function, named `case_data` with optional type hint `CaseData` - * use that input argument at the beginning of the test function, to retrieve the test data: `inputs = case_data.get()` +![class_overview](./imgs/2_class_overview.png) +And many more as we'll see [below](#a-cases-collection). -Once you have done these three steps, executing `pytest` will run your test function **once for every case function**: +## Basic usage -```bash ->>> pytest -============================= test session starts ============================= -(...) -/tests/test_foo.py::test_foo[case_two_positive_ints] PASSED [ 50%] -/tests/test_foo.py::test_foo[case_two_negative_ints] PASSED [ 100%] +### a- Case functions -========================== 2 passed in 0.24 seconds ========================== +Let's create a `test_foo_cases.py` file. This file will contain *test cases generator functions*, that we will call **case functions** for brevity. In these functions, you will typically either parse some test data files, generate some simulated test data, expected results, etc. + +```python +def case_two_positive_ints(): + """ Inputs are two positive integers """ + return 1, 2 + +def case_two_negative_ints(): + """ Inputs are two negative integers """ + return -1, -2 ``` -### d- Case fixtures +Case functions **do not have any particular requirement**, apart from the default name convention `case_` - but even that can be customized: **you can use distinct prefixes** to denote distinct kind of parameters, such as `data_`, `user_`, `model_`... -You might be concerned that case data is gathered or created *during* test execution. +Case functions can return anything that is considered useful to run the associated test. We will see [below](#b-case-functions) that you can use all classic pytest mechanism on case functions (id customization, skip/fail marks, parametrization, fixtures injection). -Indeed creating or collecting case data is not part of the test *per se*. Besides, if you benchmark your tests durations (for example with [pytest-harvest](https://smarie.github.io/python-pytest-harvest/)), you may want the test duration to be computed without acccounting for the data retrieval time - especially if you decide to add some caching mechanism as explained [here](https://smarie.github.io/python-pytest-cases/usage/advanced/#caching). +### b- Test functions -It might therefore be more interesting for you to parametrize **case fixtures** instead of parametrizing your test function. Thanks to our new [`@fixture_plus`](#pytest_fixture_plus) decorator, this works exactly the same way than for test functions: +As usual we write our `pytest` test functions starting with `test_`, in a `test_foo.py` file. The only difference is that we now decorate it with `@parametrize_with_cases` instead of `@pytest.mark.parametrize` as we were doing [previously](#why-pytest-cases): ```python -from pytest_cases import fixture_plus, cases_data from example import foo +from pytest_cases import parametrize_with_cases + +@parametrize_with_cases("a,b") +def test_foo(a, b): + # check that foo runs correctly and that the result is a tuple. + assert isinstance(foo(a, b), tuple) +``` + +As simple as that ! The syntax is basically the same than in [`pytest.mark.parametrize`](https://docs.pytest.org/en/stable/example/parametrize.html). -# import the module containing the test cases -import test_foo_cases +Executing `pytest` will now run our test function **once for every case function**: -@fixture_plus -@cases_data(module=test_foo_cases) -def inputs(case_data): - """ Example fixture that is automatically parametrized with @cases_data """ - # retrieve case data - return case_data.get() +```bash +>>> pytest -s -v +============================= test session starts ============================= +(...) +/tests/test_foo.py::test_foo[two_positive_ints] PASSED [ 50%] +/tests/test_foo.py::test_foo[two_negative_ints] PASSED [ 100%] -def test_foo(inputs): - # Use case data - foo(**inputs) +========================== 2 passed in 0.24 seconds ========================== ``` -In the above example, the `test_foo` test does not spend time collecting or generating data. When it is executed, it receives the required data directly as `inputs`. The test case creation instead happens when each `inputs` fixture instance is created by `pytest` - this is done in a separate pytest phase (named "setup"), and therefore is not counted in the test duration. +## Tools for daily use -Note: you can still use `request` in your fixture's signature if you wish to. +### a- Cases collection -## Usage - 'True' test cases +#### Alternate source(s) -#### a- Case functions update +It is not mandatory that case functions should be in a different file than the test functions: both can be in the same file. For this you can use `cases='.'` or `cases=THIS_MODULE` to refer to the module in which the test function is located: -In the above example the cases were only containing inputs for the function to test. In real-world applications we often need more: we need both inputs **and an expected outcome**. +```python +from pytest_cases import parametrize_with_cases -For this, `pytest_cases` proposes to adopt a convention where the case functions returns a tuple of inputs/outputs/errors. A handy `CaseData` PEP484 type hint can be used to denote that. But of course this is only a proposal, which is not mandatory as we saw above. +def case_one_positive_int(): + return 1 -!!! note "A case function can return **anything**" - Even if in most examples in this documentation we chose to return a tuple (inputs/outputs/errors) (type hint `CaseData`), you can decide to return anything: a single variable, a dictionary, a tuple of a different length, etc. Whatever you return will be available through `case_data.get()`. +def case_one_negative_int(): + return -1 -Here is how we can rewrite our case functions with an expected outcome: +@parametrize_with_cases("i", cases='.') +def test_with_this_module(i): + assert i == int(i) +``` -```python -def case_two_positive_ints() -> CaseData: - """ Inputs are two positive integers """ +However **WARNING**: only the case functions defined BEFORE the test function in the module file will be taken into account! - ins = dict(a=1, b=2) - outs = 2, 3 +`@parametrize_with_cases(cases=...)` also accepts explicit list of case functions, classes containing case functions, and modules. See [API Reference](./api_reference.md#parametrize_with_cases) for details. A typical way to organize cases is to use classes for example: - return ins, outs, None +```python +from pytest_cases import parametrize_with_cases -def case_two_negative_ints() -> CaseData: - """ Inputs are two negative integers """ +class Foo: + def case_a_positive_int(self): + return 1 - ins = dict(a=-1, b=-2) - outs = 0, -1 + def case_another_positive_int(self): + return 2 - return ins, outs, None +@parametrize_with_cases("a", cases=Foo) +def test_foo(a): + assert a > 0 ``` -We propose that the "expected error" (`None` above) may contain exception type, exception instances, or callables. If you follow this convention, you will be able to write your test more easily with the provided utility function `unfold_expected_err`. See [here for details](https://smarie.github.io/python-pytest-cases/usage/basics/#handling-exceptions). +Note that as for `pytest`, `self` is recreated for every test and therefore should not be used to store any useful information. -### b- Test body update +#### Alternate prefix -With our new case functions, a case will be made of three items. So `case_data.get()` will return a tuple. Here is how we can update our test function body to retrieve it correctly, and check that the outcome is as expected: +`case_` might not be your preferred prefix, especially if you wish to store in the same module or class various **kind** of case data. `@parametrize_with_cases` offers a `prefix=...` argument to select an alternate prefix for your case functions. That way, you can store **in the same module or class** case functions as diverse as datasets (e.g. `data_`), user descriptions (e.g. `user_`), algorithms or machine learning models (e.g. `model_` or `algo_`), etc. ```python -@cases_data(module=test_foo_cases) -def test_foo(case_data: CaseDataGetter): - """ Example unit test that is automatically parametrized with @cases_data """ - - # 1- Grab the test case data: now a tuple ! - i, expected_o, expected_e = case_data.get() - - # 2- Use it: we can now do some asserts ! - if expected_e is None: - # **** Nominal test **** - outs = foo(**i) - assert outs == expected_o - else: - # **** Error tests: see page to fill this **** - pass -``` +from pytest_cases import parametrize_with_cases, parametrize -See [Usage](./usage) for complete examples with custom case names, case generators, exceptions handling, and more. +def data_a(): + return 'a' +@parametrize("hello", [True, False]) +def data_b(hello): + return "hello" if hello else "world" -## `pytest` Goodies +def case_c(): + return dict(name="hi i'm not used") -### `--with-reorder` +def user_bob(): + return "bob" -`pytest` postprocesses the order of the collected items in order to optimize setup/teardown of session, module and class fixtures. This optimization algorithm happens at the `pytest_collection_modifyitems` stage, and is still under improvement, as can be seen in [pytest#3551](https://github.com/pytest-dev/pytest/pull/3551), [pytest#3393](https://github.com/pytest-dev/pytest/issues/3393), [#2846](https://github.com/pytest-dev/pytest/issues/2846)... +@parametrize_with_cases("data", cases='.', prefix="data_") +@parametrize_with_cases("user", cases='.', prefix="user_") +def test_with_data(data, user): + assert data in ('a', "hello", "world") + assert user == 'bob' +``` -Besides other plugins such as [pytest-reorder](https://github.com/not-raspberry/pytest_reorder) can modify the order as well. +yields +``` +test_doc_filters_n_tags.py::test_with_data[bob-a] PASSED [ 33%] +test_doc_filters_n_tags.py::test_with_data[bob-b-True] PASSED [ 66%] +test_doc_filters_n_tags.py::test_with_data[bob-b-False] PASSED [ 100%] +``` -This new commandline is a goodie to change the reordering: +#### Filters and tags - * `--with-reorder normal` is the default behaviour: it lets pytest and all the plugins execute their reordering in each of their `pytest_collection_modifyitems` hooks, and simply does not interact - - * `--with-reorder skip` allows you to restore the original order that was active before `pytest_collection_modifyitems` was initially called, thus not taking into account any reordering done by pytest or by any of its plugins. - +The easiest way to select only a subset of case functions in a module or a class, is to specify a custom `prefix` instead of the default one (`'case_'`), as shown [above](#alternate-prefix). -### `@fixture_plus` +However sometimes more advanced filtering is required. In that case, you can also rely on three additional mechanisms provided in `@parametrize_with_cases`: -`@fixture_plus` is similar to `pytest.fixture` but without its `param` and `ids` arguments. Instead, it is able to pick the parametrization from `@pytest.mark.parametrize` marks applied on fixtures. This makes it very intuitive for users to parametrize both their tests and fixtures. As a bonus, its `name` argument works even in old versions of pytest (which is not the case for `fixture`). + - the `glob` argument can contain a glob-like pattern for case ids. This can become handy to separate for example good or bad cases, the latter returning an expected error type and/or message for use with `pytest.raises` or with our alternative [`assert_exception`](pytest_goodies.md#assert_exception). + +```python +from math import sqrt +import pytest +from pytest_cases import parametrize_with_cases -Finally it now supports unpacking, see [unpacking feature](#unpack_fixture-unpack_into). -!!! note "`@fixture_plus` deprecation if/when `@pytest.fixture` supports `@pytest.mark.parametrize`" - The ability for pytest fixtures to support the `@pytest.mark.parametrize` annotation is a feature that clearly belongs to `pytest` scope, and has been [requested already](https://github.com/pytest-dev/pytest/issues/3960). It is therefore expected that `@fixture_plus` will be deprecated in favor of `@pytest_fixture` if/when the `pytest` team decides to add the proposed feature. As always, deprecation will happen slowly across versions (at least two minor, or one major version update) so as for users to have the time to update their code bases. +def case_int_success(): + return 1 -### `unpack_fixture` / `unpack_into` +def case_negative_int_failure(): + # note that we decide to return the expected type of failure to check it + return -1, ValueError, "math domain error" -In some cases fixtures return a tuple or a list of items. It is not easy to refer to a single of these items in a test or another fixture. With `unpack_fixture` you can easily do it: -```python -import pytest -from pytest_cases import unpack_fixture, fixture_plus +@parametrize_with_cases("data", cases='.', glob="*success") +def test_good_datasets(data): + assert sqrt(data) > 0 -@fixture_plus -@pytest.mark.parametrize("o", ['hello', 'world']) -def c(o): - return o, o[0] +@parametrize_with_cases("data, err_type, err_msg", cases='.', glob="*failure") +def test_bad_datasets(data, err_type, err_msg): + with pytest.raises(err_type, match=err_msg): + sqrt(data) +``` -a, b = unpack_fixture("a,b", c) -def test_function(a, b): - assert a[0] == b -``` + - the `has_tag` argument allows you to filter cases based on tags set on case functions using the `@case` decorator. See API reference of [`@case`](./api_reference.md#case) and [`@parametrize_with_cases`](./api_reference.md#parametrize_with_cases). -Note that you can also use the `unpack_into=` argument of `@fixture_plus` to do the same thing: ```python -import pytest -from pytest_cases import fixture_plus +from pytest_cases import parametrize_with_cases, case -@fixture_plus(unpack_into="a,b") -@pytest.mark.parametrize("o", ['hello', 'world']) -def c(o): - return o, o[0] +class FooCases: + def case_two_positive_ints(self): + return 1, 2 + + @case(tags='foo') + def case_one_positive_int(self): + return 1 -def test_function(a, b): - assert a[0] == b +@parametrize_with_cases("a", cases=FooCases, has_tag='foo') +def test_foo(a): + assert a > 0 ``` -And it is also available in `fixture_union`: + - Finally if none of the above matches your expectations, you can provide a callable to `filter`. This callable will receive each collected case function and should return `True` (or a truth-value convertible object) in case of success. Note that your function can leverage the `_pytestcase` attribute available on the case function to read the tags, marks and id found on it. ```python -import pytest -from pytest_cases import fixture_plus, fixture_union +@parametrize_with_cases("data", cases='.', + filter=lambda cf: "success" in cf._pytestcase.id) +def test_good_datasets2(data): + assert sqrt(data) > 0 +``` + -@fixture_plus -@pytest.mark.parametrize("o", ['hello', 'world']) -def c(o): - return o, o[0] +### b- Case functions -@fixture_plus -@pytest.mark.parametrize("o", ['yeepee', 'yay']) -def d(o): - return o, o[0] +#### Custom case name -fixture_union("c_or_d", [c, d], unpack_into="a, b") +The id used by `pytest` for a given case is automatically taken from the case function name by removing the `case_` (or other custom) prefix. It can instead be customized explicitly by decorating your case function with the `@case(id=)` decorator. See [API reference](./api_reference.md#case). -def test_function(a, b): - assert a[0] == b -``` +```python +from pytest_cases import case -### `param_fixture[s]` +@case(id="2 positive integers") +def case_two_positive_ints(): + return 1, 2 +``` -If you wish to share some parameters across several fixtures and tests, it might be convenient to have a fixture representing this parameter. This is relatively easy for single parameters, but a bit harder for parameter tuples. +#### Pytest marks (`skip`, `xfail`...) -The two utilities functions `param_fixture` (for a single parameter name) and `param_fixtures` (for a tuple of parameter names) handle the difficulty for you: +pytest marks such as `@pytest.mark.skipif` can be applied on case functions the same way [as with test functions](https://docs.pytest.org/en/stable/skipping.html). ```python +import sys import pytest -from pytest_cases import param_fixtures, param_fixture -# create a single parameter fixture -my_parameter = param_fixture("my_parameter", [1, 2, 3, 4]) +@pytest.mark.skipif(sys.version_info < (3, 0), reason="Not useful on python 2") +def case_two_positive_ints(): + return 1, 2 +``` -@pytest.fixture -def fixture_uses_param(my_parameter): - ... +#### Case generators -def test_uses_param(my_parameter, fixture_uses_param): - ... +In many real-world usage we want to generate one test case *per* ``. The most intuitive way would be to use a `for` loop to create the case functions, and to use the `@case` decorator to set their names ; however this would not be very readable. -# ----- -# create a 2-tuple parameter fixture -arg1, arg2 = param_fixtures("arg1, arg2", [(1, 2), (3, 4)]) +Instead, case functions can be parametrized the same way [as with test functions](https://docs.pytest.org/en/stable/parametrize.html): simply add the parameter names as arguments in their signature and decorate with `@pytest.mark.parametrize`. Even better, you can use the enhanced [`@parametrize`](./api_reference.md#parametrize) from `pytest-cases` so as to benefit from its additional usability features (see [API reference](./api_reference.md#parametrize)): -@pytest.fixture -def fixture_uses_param2(arg2): - ... +```python +from pytest_cases import parametrize, parametrize_with_cases -def test_uses_param2(arg1, arg2, fixture_uses_param2): - ... +class CasesFoo: + def case_hello(self): + return "hello world" + + @parametrize(who=('you', 'there')) + def case_simple_generator(self, who): + return "hello %s" % who + + +@parametrize_with_cases("msg", cases=CasesFoo) +def test_foo(msg): + assert isinstance(msg, str) and msg.startswith("hello") ``` -You can mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). +Yields -### `fixture_union` +``` +test_generators.py::test_foo[hello] PASSED [ 33%] +test_generators.py::test_foo[simple_generator-who=you] PASSED [ 66%] +test_generators.py::test_foo[simple_generator-who=there] PASSED [100%] +``` -As of `pytest` 5, it is not possible to create a "union" fixture, i.e. a parametrized fixture that would first take all the possible values of fixture A, then all possible values of fixture B, etc. Indeed all fixture dependencies (a.k.a. "closure") of each test node are grouped together, and if they have parameters a big "cross-product" of the parameters is done by `pytest`. +#### Cases requiring fixtures -The topic has been largely discussed in [pytest-dev#349](https://github.com/pytest-dev/pytest/issues/349) and a [request for proposal](https://docs.pytest.org/en/latest/proposals/parametrize_with_fixtures.html) has been finally made. +Cases can use fixtures the same way as [test functions do](https://docs.pytest.org/en/stable/fixture.html#fixtures-as-function-arguments): simply add the fixture names as arguments in their signature and make sure the fixture exists either in the same module, or in a [`conftest.py`](https://docs.pytest.org/en/stable/fixture.html?highlight=conftest.py#conftest-py-sharing-fixture-functions) file in one of the parent packages. See [`pytest` documentation on sharing fixtures](https://docs.pytest.org/en/stable/fixture.html?highlight=conftest.py#conftest-py-sharing-fixture-functions). -`fixture_union` is an implementation of this proposal. It is also used by `parametrize_plus` to support `fixture_ref` in parameter values, see [below](#parametrize_plus). ```python -from pytest_cases import fixture_plus, fixture_union +from pytest_cases import parametrize_with_cases, fixture, parametrize -@fixture_plus -def first(): - return 'hello' +@fixture(scope='session') +def db(): + return {0: 'louise', 1: 'bob'} -@fixture_plus(params=['a', 'b']) -def second(request): - return request.param +def user_bob(db): + return db[1] -# c will first take all the values of 'first', then all of 'second' -c = fixture_union('c', [first, second]) +@parametrize(id=range(2)) +def user_from_db(db, id): + return db[id] -def test_basic_union(c): - print(c) +@parametrize_with_cases("a", cases='.', prefix='user_') +def test_users(a, db, request): + print("this is test %r" % request.node.nodeid) + assert a in db.values() ``` yields ``` -<...>::test_basic_union[c_is_first] hello PASSED -<...>::test_basic_union[c_is_second-a] a PASSED -<...>::test_basic_union[c_is_second-b] b PASSED +test_fixtures.py::test_users[a_is_bob] +test_fixtures.py::test_users[a_is_from_db-id=0] +test_fixtures.py::test_users[a_is_from_db-id=1] ``` -As you can see the ids of union fixtures are slightly different from standard ids, so that you can easily understand what is going on. You can change this feature with `ìdstyle`, see [API documentation](./api_reference.md#fixture_union) for details. +## Advanced topics -You can mark any of the alternatives with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). +### a- Test fixtures -Fixture unions also support unpacking with the `unpack_into` argument, see [unpacking feature](#unpack_fixture-unpack_into). +In some scenarii you might wish to parametrize a fixture with the cases, rather than the test function. For example -Fixture unions are a **major change** in the internal pytest engine, as fixture closures (the ordered set of all fixtures required by a test node to run - directly or indirectly) now become trees where branches correspond to alternative paths taken in the "unions", and leafs are the alternative fixture closures. This feature has been tested in very complex cases (several union fixtures, fixtures that are not selected by a given union but that is requested by the test function, etc.). But if you find some strange behaviour don't hesitate to report it in the [issues](https://github.com/smarie/python-pytest-cases/issues) page ! + - to inject the same test cases in several test functions **without duplicating** the `@parametrize_with_cases` decorator on each of them, + + - to generate the test cases **once** for the whole session, using a `scope='session'` fixture or [another scope](https://docs.pytest.org/en/stable/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session), + + - to modify the test cases, log some message, or perform some other action **before injecting them** into the test functions, and/or **after executing** the test function (thanks to [yield fixtures](https://docs.pytest.org/en/stable/fixture.html#fixture-finalization-executing-teardown-code)) + + - ... -**IMPORTANT** if you do not use `@fixture_plus` but only `@pytest.fixture`, then you will see that your fixtures are called even when they are not used, with a parameter `NOT_USED`. This symbol is automatically ignored if you use `@fixture_plus`, otherwise you have to handle it. Alternatively you can use `@ignore_unused` on your fixture function. +For this, simply use `@fixture` from `pytest_cases` instead of `@pytest.fixture` to define your fixture. That allows your fixtures to be easily parametrized with `@parametrize_with_cases`, `@parametrize`, and even `@pytest.mark.parametrize`. -!!! note "fixture unions vs. cases" - If you're familiar with `pytest-cases` already, you might note that `@cases_data` is not so different than a fixture union: we do a union of all case functions. If one day union fixtures are directly supported by `pytest`, we will probably refactor this lib to align all the concepts. +```python +from pytest_cases import fixture, parametrize_with_cases -### `@parametrize_plus` +@fixture +@parametrize_with_cases("a,b") +def c(a, b): + return a + b -`@parametrize_plus` is a replacement for `@pytest.mark.parametrize` that allows you to include references to fixtures and to value-generating functions in the parameter values. +def test_foo(c): + assert isinstance(c, int) +``` - - Simply use `fixture_ref()` in the parameter values, where `` can be the fixture name or fixture function. - - if you do not wish to create a fixture, you can also use `lazy_value()` - - Note that when parametrizing several argnames, both `fixture_ref` and `lazy_value` can be used *as* the tuple, or *in* the tuple. Several `fixture_ref` and/or `lazy_value` can be used in the same tuple, too. +### b- Caching cases -For example, with a single argument: +After starting to reuse cases in several test functions, you might end-up thinking *"why do I have to spend the data parsing/generation time several times ? It is the same case."*. There are several ways to solve this issue: + - the easiest way is to **use fixtures with a broad scope**, as explained [above](#a-test-fixtures). However in some parametrization scenarii, `pytest` does not guarantee that the fixture will be setup only once for the whole session, even if it is a session-scoped fixture. Also the cases will be parsed everytime you run pytest, which might be cumbersome + ```python -import pytest -from pytest_cases import parametrize_plus, fixture_plus, fixture_ref, lazy_value - -@pytest.fixture -def world_str(): - return 'world' - -def whatfun(): - return 'what' - -@fixture_plus -@parametrize_plus('who', [fixture_ref(world_str), - 'you']) -def greetings(who): - return 'hello ' + who - -@parametrize_plus('main_msg', ['nothing', - fixture_ref(world_str), - lazy_value(whatfun), - fixture_ref(greetings)]) -@pytest.mark.parametrize('ending', ['?', '!']) -def test_prints(main_msg, ending): - print(main_msg + ending) -``` +from pytest_cases import parametrize, parametrize_with_cases, fixture -yields the following -```bash -> pytest -s -v -collected 10 items -test_prints[main_msg_is_nothing-?] PASSED [ 10%]nothing? -test_prints[main_msg_is_nothing-!] PASSED [ 20%]nothing! -test_prints[main_msg_is_world_str-?] PASSED [ 30%]world? -test_prints[main_msg_is_world_str-!] PASSED [ 40%]world! -test_prints[main_msg_is_whatfun-?] PASSED [ 50%]what? -test_prints[main_msg_is_whatfun-!] PASSED [ 60%]what! -test_prints[main_msg_is_greetings-who_is_world_str-?] PASSED [ 70%]hello world? -test_prints[main_msg_is_greetings-who_is_world_str-!] PASSED [ 80%]hello world! -test_prints[main_msg_is_greetings-who_is_you-?] PASSED [ 90%]hello you? -test_prints[main_msg_is_greetings-who_is_you-!] PASSED [100%]hello you! -``` +@parametrize(a=range(2)) +def case_dummy(a): + # this is read only once per a, while there are 4 test runs + return a -You can also mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). +@fixture(scope='session') +@parametrize_with_cases("a", cases='.') +def cached_a(a): + return a -As you can see in the example above, the default ids are a bit more explicit than usual when you use at least one `fixture_ref`. This is because the parameters need to be replaced with a fixture union that will "switch" between alternative groups of parameters, and the appropriate fixtures referenced. As opposed to `fixture_union`, the style of these ids is not configurable for now, but feel free to propose alternatives in the [issues page](https://github.com/smarie/python-pytest-cases/issues). Note that this does not happen if you only use `lazy_value`s, as they do not require to create a fixture union behind the scenes. -Another consequence of using `fixture_ref` is that the priority order of the parameters, relative to other standard `pytest.mark.parametrize` parameters that you would place on the same function, will get impacted. You may solve this by replacing your other `@pytest.mark.parametrize` calls with `param_fixture`s so that all the parameters are fixtures (see [above](#param_fixtures).) +@parametrize(d=range(2)) +def test_caching(cached_a, d): + assert d < 2 + assert 0 <= cached_a <= 1 +``` -### passing a `hook` + - an alternative is to use `functools.lru_cache` to explicitly set a memory cache on a case function. For simple cases you could simply decorate your case function with `@lru_cache(maxsize=1)` since simple case functions do not have arguments. However for case generators this is a bit more tricky to size the cache - the easiest thing is probably to let it to its default size of 128 with the no-argument version `@lru_cache`, or to remove the max limit and let it auto-grow, with `@lru_cache(max_size=None)`. See [`lru_cache` documentation for details](https://docs.python.org/3/library/functools.html#functools.lru_cache). Note that an older version of `pytest-cases` was offering some facilities to set the cache size, this has been removed from the library in version `2.0.0` as it seemed to provide little added value. + + - finally, you might wish to persist some cases on disk in order for example to avoid downloading them again from their original source, and/or to avoid costly processing on every pytest session. For this, the perfect match for you is to use [`joblib`'s excellent `Memory` cache](https://joblib.readthedocs.io/en/latest/memory.html). -As per version `1.14`, all the above functions now support passing a `hook` argument. This argument should be a callable. It will be called everytime a fixture is about to be created by `pytest_cases` on your behalf. The fixture function is passed as the argument of the hook, and the hook should return it as the result. -You can use this fixture to better understand which fixtures are created behind the scenes, and also to decorate the fixture functions before they are created. For example you can use `hook=saved_fixture` (from [`pytest-harvest`](https://smarie.github.io/python-pytest-harvest/)) in order to save the created fixtures in the fixture store. ## Main features / benefits * **Separation of concerns**: test code on one hand, test cases data on the other hand. This is particularly relevant for data science projects where a lot of test datasets are used on the same block of test code. - * **Everything in the test or in the fixture**, not outside. A side-effect of `@pytest.mark.parametrize` is that users tend to create or parse their datasets outside of the test function. `pytest_cases` suggests a model where the potentially time and memory consuming step of case data generation/retrieval is performed *inside* the test node or the required fixture, thus keeping every test case run more independent. It is also easy to put debug breakpoints on specific test cases. + * **Everything in the test case or in the fixture**, not outside. A side-effect of `@pytest.mark.parametrize` is that users tend to create or parse their datasets outside of the test function. `pytest_cases` suggests a model where the potentially time and memory consuming step of case data generation/retrieval is performed *inside* the test node or the required fixture, thus keeping every test case run more independent. It is also easy to put debug breakpoints on specific test cases. - * **Easier iterable-based test case generation**. If you wish to generate several test cases using the same function, `@cases_generator` makes it very intuitive to do so. See [here](./usage#case-generators) for details. + * **User experience fully aligned with pytest**. Cases collection and filtering, cases parametrization, cases output unpacking as test arguments, cases using fixtures... all of this will look very familiar to `pytest` users. - * **User-friendly features**: easily customize your test cases with friendly names, reuse the same cases for different test functions by tagging/filtering, and more... See [Usage](./usage) for details. ## See Also @@ -440,6 +459,7 @@ You can use this fixture to better understand which fixtures are created behind - [pytest documentation on fixtures](https://docs.pytest.org/en/latest/fixture.html#fixture-parametrize) - [pytest-steps](https://smarie.github.io/python-pytest-steps/) - [pytest-harvest](https://smarie.github.io/python-pytest-harvest/) + - [pytest-patterns](https://smarie.github.io/pytest-patterns/) for examples showing how to combine the various plugins to create data science benchmarks. ### Others diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index f6367914..5f8cbb5d 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -5,11 +5,7 @@ docs_dir: . site_dir: ../site nav: - Home: index.md - - Usage details: - - Overview: usage.md - - Basics: usage/basics.md - - Intermediate: usage/intermediate.md - - Advanced: usage/advanced.md + - pytest goodies: pytest_goodies.md - API reference: api_reference.md - Changelog: changelog.md theme: material # readthedocs mkdocs diff --git a/docs/pytest_goodies.md b/docs/pytest_goodies.md new file mode 100644 index 00000000..8dad04b8 --- /dev/null +++ b/docs/pytest_goodies.md @@ -0,0 +1,232 @@ +# `pytest` Goodies + +Many `pytest` features were missing to make `pytest_cases` work with such a "no-boilerplate" experience. Many of these can be of interest to the general `pytest` audience, so they are exposed in the public API. + + +## `@fixture` + +`@fixture` is similar to `pytest.fixture` but without its `param` and `ids` arguments. Instead, it is able to pick the parametrization from `@pytest.mark.parametrize` marks applied on fixtures. This makes it very intuitive for users to parametrize both their tests and fixtures. As a bonus, its `name` argument works even in old versions of pytest (which is not the case for `fixture`). + +Finally it now supports unpacking, see [unpacking feature](#unpack_fixture-unpack_into). + +!!! note "`@fixture` deprecation if/when `@pytest.fixture` supports `@pytest.mark.parametrize`" + The ability for pytest fixtures to support the `@pytest.mark.parametrize` annotation is a feature that clearly belongs to `pytest` scope, and has been [requested already](https://github.com/pytest-dev/pytest/issues/3960). It is therefore expected that `@fixture` will be deprecated in favor of `@pytest_fixture` if/when the `pytest` team decides to add the proposed feature. As always, deprecation will happen slowly across versions (at least two minor, or one major version update) so as for users to have the time to update their code bases. + +## `unpack_fixture` / `unpack_into` + +In some cases fixtures return a tuple or a list of items. It is not easy to refer to a single of these items in a test or another fixture. With `unpack_fixture` you can easily do it: + +```python +import pytest +from pytest_cases import unpack_fixture, fixture + +@fixture +@pytest.mark.parametrize("o", ['hello', 'world']) +def c(o): + return o, o[0] + +a, b = unpack_fixture("a,b", c) + +def test_function(a, b): + assert a[0] == b +``` + +Note that you can also use the `unpack_into=` argument of `@fixture` to do the same thing: + +```python +import pytest +from pytest_cases import fixture + +@fixture(unpack_into="a,b") +@pytest.mark.parametrize("o", ['hello', 'world']) +def c(o): + return o, o[0] + +def test_function(a, b): + assert a[0] == b +``` + +And it is also available in `fixture_union`: + +```python +import pytest +from pytest_cases import fixture, fixture_union + +@fixture +@pytest.mark.parametrize("o", ['hello', 'world']) +def c(o): + return o, o[0] + +@fixture +@pytest.mark.parametrize("o", ['yeepee', 'yay']) +def d(o): + return o, o[0] + +fixture_union("c_or_d", [c, d], unpack_into="a, b") + +def test_function(a, b): + assert a[0] == b +``` + +## `param_fixture[s]` + +If you wish to share some parameters across several fixtures and tests, it might be convenient to have a fixture representing this parameter. This is relatively easy for single parameters, but a bit harder for parameter tuples. + +The two utilities functions `param_fixture` (for a single parameter name) and `param_fixtures` (for a tuple of parameter names) handle the difficulty for you: + +```python +import pytest +from pytest_cases import param_fixtures, param_fixture + +# create a single parameter fixture +my_parameter = param_fixture("my_parameter", [1, 2, 3, 4]) + +@pytest.fixture +def fixture_uses_param(my_parameter): + ... + +def test_uses_param(my_parameter, fixture_uses_param): + ... + +# ----- +# create a 2-tuple parameter fixture +arg1, arg2 = param_fixtures("arg1, arg2", [(1, 2), (3, 4)]) + +@pytest.fixture +def fixture_uses_param2(arg2): + ... + +def test_uses_param2(arg1, arg2, fixture_uses_param2): + ... +``` + +You can mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). + +## `fixture_union` + +As of `pytest` 5, it is not possible to create a "union" fixture, i.e. a parametrized fixture that would first take all the possible values of fixture A, then all possible values of fixture B, etc. Indeed all fixture dependencies (a.k.a. "closure") of each test node are grouped together, and if they have parameters a big "cross-product" of the parameters is done by `pytest`. + +The topic has been largely discussed in [pytest-dev#349](https://github.com/pytest-dev/pytest/issues/349) and a [request for proposal](https://docs.pytest.org/en/latest/proposals/parametrize_with_fixtures.html) has been finally made. + +`fixture_union` is an implementation of this proposal. It is also used by `parametrize` to support `fixture_ref` in parameter values, see [below](#parametrize). + +```python +from pytest_cases import fixture, fixture_union + +@fixture +def first(): + return 'hello' + +@fixture(params=['a', 'b']) +def second(request): + return request.param + +# c will first take all the values of 'first', then all of 'second' +c = fixture_union('c', [first, second]) + +def test_basic_union(c): + print(c) +``` + +yields + +``` +<...>::test_basic_union[c_is_first] hello PASSED +<...>::test_basic_union[c_is_second-a] a PASSED +<...>::test_basic_union[c_is_second-b] b PASSED +``` + +As you can see the ids of union fixtures are slightly different from standard ids, so that you can easily understand what is going on. You can change this feature with `ìdstyle`, see [API documentation](./api_reference.md#fixture_union) for details. + +You can mark any of the alternatives with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). + +Fixture unions also support unpacking with the `unpack_into` argument, see [unpacking feature](#unpack_fixture-unpack_into). + +Fixture unions are a **major change** in the internal pytest engine, as fixture closures (the ordered set of all fixtures required by a test node to run - directly or indirectly) now become trees where branches correspond to alternative paths taken in the "unions", and leafs are the alternative fixture closures. This feature has been tested in very complex cases (several union fixtures, fixtures that are not selected by a given union but that is requested by the test function, etc.). But if you find some strange behaviour don't hesitate to report it in the [issues](https://github.com/smarie/python-pytest-cases/issues) page ! + +**IMPORTANT** if you do not use `@fixture` but only `@pytest.fixture`, then you will see that your fixtures are called even when they are not used, with a parameter `NOT_USED`. This symbol is automatically ignored if you use `@fixture`, otherwise you have to handle it. Alternatively you can use `@ignore_unused` on your fixture function. + +!!! note "fixture unions vs. cases" + If you're familiar with `pytest-cases` already, you might note that `@cases_data` is not so different than a fixture union: we do a union of all case functions. If one day union fixtures are directly supported by `pytest`, we will probably refactor this lib to align all the concepts. + + +## `@parametrize` + +`@parametrize` is a replacement for `@pytest.mark.parametrize` with many additional features to make the most of parametrization. See [API reference](./api_reference.md#parametrize) for details about all the new features. In particular it allows you to include references to fixtures and to value-generating functions in the parameter values. + + - Simply use `fixture_ref()` in the parameter values, where `` can be the fixture name or fixture function. + - if you do not wish to create a fixture, you can also use `lazy_value()` + - Note that when parametrizing several argnames, both `fixture_ref` and `lazy_value` can be used *as* the tuple, or *in* the tuple. Several `fixture_ref` and/or `lazy_value` can be used in the same tuple, too. + +For example, with a single argument: + +```python +import pytest +from pytest_cases import parametrize, fixture, fixture_ref, lazy_value + +@pytest.fixture +def world_str(): + return 'world' + +def whatfun(): + return 'what' + +@fixture +@parametrize('who', [fixture_ref(world_str), + 'you']) +def greetings(who): + return 'hello ' + who + +@parametrize('main_msg', ['nothing', + fixture_ref(world_str), + lazy_value(whatfun), + fixture_ref(greetings)]) +@pytest.mark.parametrize('ending', ['?', '!']) +def test_prints(main_msg, ending): + print(main_msg + ending) +``` + +yields the following + +```bash +> pytest -s -v +collected 10 items +test_prints[main_msg_is_nothing-?] PASSED [ 10%]nothing? +test_prints[main_msg_is_nothing-!] PASSED [ 20%]nothing! +test_prints[main_msg_is_world_str-?] PASSED [ 30%]world? +test_prints[main_msg_is_world_str-!] PASSED [ 40%]world! +test_prints[main_msg_is_whatfun-?] PASSED [ 50%]what? +test_prints[main_msg_is_whatfun-!] PASSED [ 60%]what! +test_prints[main_msg_is_greetings-who_is_world_str-?] PASSED [ 70%]hello world? +test_prints[main_msg_is_greetings-who_is_world_str-!] PASSED [ 80%]hello world! +test_prints[main_msg_is_greetings-who_is_you-?] PASSED [ 90%]hello you? +test_prints[main_msg_is_greetings-who_is_you-!] PASSED [100%]hello you! +``` + +You can also mark any of the argvalues with `pytest.mark` to pass a custom id or a custom "skip" or "fail" mark, just as you do in `pytest`. See [pytest documentation](https://docs.pytest.org/en/stable/example/parametrize.html#set-marks-or-test-id-for-individual-parametrized-test). + +As you can see in the example above, the default ids are a bit more explicit than usual when you use at least one `fixture_ref`. This is because the parameters need to be replaced with a fixture union that will "switch" between alternative groups of parameters, and the appropriate fixtures referenced. As opposed to `fixture_union`, the style of these ids is not configurable for now, but feel free to propose alternatives in the [issues page](https://github.com/smarie/python-pytest-cases/issues). Note that this does not happen if you only use `lazy_value`s, as they do not require to create a fixture union behind the scenes. + +Another consequence of using `fixture_ref` is that the priority order of the parameters, relative to other standard `pytest.mark.parametrize` parameters that you would place on the same function, will get impacted. You may solve this by replacing your other `@pytest.mark.parametrize` calls with `param_fixture`s so that all the parameters are fixtures (see [above](#param_fixtures).) + +## passing a `hook` + +As per version `1.14`, all the above functions now support passing a `hook` argument. This argument should be a callable. It will be called everytime a fixture is about to be created by `pytest_cases` on your behalf. The fixture function is passed as the argument of the hook, and the hook should return it as the result. + +You can use this fixture to better understand which fixtures are created behind the scenes, and also to decorate the fixture functions before they are created. For example you can use `hook=saved_fixture` (from [`pytest-harvest`](https://smarie.github.io/python-pytest-harvest/)) in order to save the created fixtures in the fixture store. + +## `assert_exception` + +`assert_exception` context manager is an alternative to `pytest.raises` to check exceptions in your tests. You can either check type, instance equality, repr string pattern, or use custom validation functions. See [API reference](./api_reference.md). + +## `--with-reorder` + +`pytest` postprocesses the order of the collected items in order to optimize setup/teardown of session, module and class fixtures. This optimization algorithm happens at the `pytest_collection_modifyitems` stage, and is still under improvement, as can be seen in [pytest#3551](https://github.com/pytest-dev/pytest/pull/3551), [pytest#3393](https://github.com/pytest-dev/pytest/issues/3393), [#2846](https://github.com/pytest-dev/pytest/issues/2846)... + +Besides other plugins such as [pytest-reorder](https://github.com/not-raspberry/pytest_reorder) can modify the order as well. + +This new commandline is a goodie to change the reordering: + + * `--with-reorder normal` is the default behaviour: it lets pytest and all the plugins execute their reordering in each of their `pytest_collection_modifyitems` hooks, and simply does not interact + + * `--with-reorder skip` allows you to restore the original order that was active before `pytest_collection_modifyitems` was initially called, thus not taking into account any reordering done by pytest or by any of its plugins. diff --git a/docs/usage.md b/docs/usage.md deleted file mode 100644 index dc5dc93f..00000000 --- a/docs/usage.md +++ /dev/null @@ -1,10 +0,0 @@ -# Usage - -You have seen in the [main page](./index) a small example to understand the concepts. `pytest_cases` provides a few additional goodies to go further. - - - [Basic](usage/basics.md): the "must read" to get started - - [Intermediate](usage/intermediate.md) - - [Advanced](usage/advanced.md) - - -You can also wish to look at the [API reference](./api_reference.md) in addition. diff --git a/docs/usage/advanced.md b/docs/usage/advanced.md deleted file mode 100644 index 2fc75c61..00000000 --- a/docs/usage/advanced.md +++ /dev/null @@ -1,414 +0,0 @@ -# Advanced usage - -## Case arguments - -Case functions can have arguments. This makes it very easy to combine test cases with more elaborate pytest concepts (fixtures, parameters): - -```python -import pytest -from pytest_cases import CaseData, cases_data, CaseDataGetter, THIS_MODULE - -def case_simple(version: str) -> CaseData: - print("using version " + version) - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None - -def case_simple2(version: str) -> CaseData: - print("using version " + version) - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None - -# the order of the loops will be [for version] > [for case] -@cases_data(module=THIS_MODULE) -@pytest.mark.parametrize("version", ["1.0.0", "2.0.0"]) -def test_with_parameters(case_data: CaseDataGetter, version): - # 1- Grab the test case data with the parameter - i, expected_o, expected_e = case_data.get(version) - - # 2- Use it as usual... - # ... -``` - -This also works with case generators: simply add the argument(s) to the function signature, *without* declaring them in the `@cases_generator` decorator. - -```python -@cases_generator("gen case i={i}, j={j}", i=range(2), j=range(2)) -def case_gen(version: str, i: int, j: int) -> CaseData: - print("using version " + version) - ins = dict(a=i, b=j) - outs = i+1, j+1 - return ins, outs, None -``` - - -## Reusing cases in several Tests - -You might wish to use the same test cases in several test functions. This works out of the box: simply refer to the same test case module in the `@case_data` decorator of several test functions, and you're set! - -```python -import pytest -from pytest_cases import cases_data, CaseDataGetter - -# import the module containing the test cases -import test_foo_cases - - -@cases_data(module=test_foo_cases) -def test_1(case_data: CaseDataGetter): - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - # ... - -@cases_data(module=test_foo_cases) -def test_2(case_data: CaseDataGetter): - """ Another test that uses exactly the same test case data than test_1 """ - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - # ... -``` - -### With variants - -If the tests use the same shared cases but with small differences, you may wish to add [arguments](#case_arguments) to your case functions. - -### Caching - -After starting to reuse cases in several test functions, you might end-up thinking *"why do I have to spend the data parsing/generation time several times ? It is the same case."*. You can solve this issue by using a cache. - -For simple cases you can simply decorate your case function with `@lru_cache(maxsize=1)` since simple case functions do not have arguments: - -```python -from functools import lru_cache - -@lru_cache(maxsize=1) -def case_a(): - # ... (as usual) -``` - -For case generators you **can** also use `@lru_cache(maxsize=x)`, but you will have to set the max size according to the number of generated cases (or `None` to allow auto-grow). This can be automated: simply use the `lru_cache=True` parameter and `pytest-cases` will do it for you: - -```python -from pytest_cases import CaseData, cases_data, CaseDataGetter, THIS_MODULE, \ - cases_generator - -# ----------------------CASES-------------------------- -# case generator with caching enabled -@cases_generator("case {i}", i=range(3), lru_cache=True) -def case_gen(i) -> CaseData: - print("generating case " + str(i)) - ins = i - outs, err = None, None - return ins, outs, err - -# ----------------------TESTS-------------------------- -@cases_data(module=THIS_MODULE) -def test_a(case_data: CaseDataGetter): - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - ... - - -@cases_data(module=THIS_MODULE) -def test_b(case_data: CaseDataGetter): - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - ... -``` - -yields: - -```bash -============================= test session starts ============================= -... -collecting ... collected 6 items -test_memoize_generators.py::test_a[case 0] PASSED [ 16%]generating case 0 -0 -test_memoize_generators.py::test_a[case 1] PASSED [ 33%]generating case 1 -1 -test_memoize_generators.py::test_a[case 2] PASSED [ 50%]generating case 2 -2 -test_memoize_generators.py::test_b[case 0] PASSED [ 66%]0 -test_memoize_generators.py::test_b[case 1] PASSED [ 83%]1 -test_memoize_generators.py::test_b[case 2] PASSED [100%]2 -========================== 6 passed in 0.16 seconds =========================== -``` - -You can see that the second time each case is needed, the cached value is used instead of executing the case generation function again. - -See [doc on lru_cache](https://docs.python.org/3/library/functools.html#functools.lru_cache) for implementation details. - -**WARNING** if you use [case arguments](#case_arguments), do not forget to take the additional parameter values into account to estimate the total cache size. Note that the `lru_cache=` option of `@cases_generator` is not intelligent enough to handle additional arguments: do not use it, and instead manually apply the `@lru_cache` decorator. - - -## Incremental tests with [pytest-steps](https://smarie.github.io/python-pytest-steps/) - -Sometimes you wish to execute a series of test steps on the same dataset, and then to move to another one. There are many ways to do this with `pytest` but some of them are not easy to blend with the notion of 'cases' in an intuitive manner. `pytest-cases` is compliant with [pytest-steps](https://smarie.github.io/python-pytest-steps/): you can easily create incremental tests and throw your cases on them. - -This tutorial assumes that you are already familiar with [pytest-steps](https://smarie.github.io/python-pytest-steps/). - - -### 1- If steps can run with the same data - -If all of the test steps require the same data to execute, it is straightforward, both in parametrizer mode (shown below) or in the new pytest steps generator mode (not shown): - - -```python -from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, CaseData -from pytest_steps import test_steps - -# -------- test cases -def case_simple() -> CaseData: - ins = dict(a=1, b=2) - return ins, None, None - -def case_simple2() -> CaseData: - ins = dict(a=-1, b=2) - return ins, None, None - -# ------- test steps -def step_a(steps_data, ins, expected_o, expected_e): - """ Step a of the test """ - # Use the three items as usual - ... - # You can also store intermediate results in steps_data - -def step_b(steps_data, ins, expected_o, expected_e): - """ Step b of the test """ - # Use the three items as usual - ... - # You can also retrieve intermediate results from steps_data - -# ------- test suite -@test_steps(step_a, step_b) -@cases_data(module=THIS_MODULE) -def test_suite(test_step, case_data: CaseDataGetter, steps_data): - # Get the data for this step - ins, expected_o, expected_e = case_data.get() - - # Execute the step - test_step(steps_data, ins, expected_o, expected_e) -``` - -This yields: - -```bash -============================= test session starts ============================= -... -test_p.py::test_suite[case_simple-step_a] PASSED [ 25%]{'a': 1, 'b': 2} -test_p.py::test_suite[case_simple-step_b] PASSED [ 50%]{'a': 1, 'b': 2} -test_p.py::test_suite[case_simple2-step_a] PASSED [ 75%]{'a': -1, 'b': 2} -test_p.py::test_suite[case_simple2-step_b] PASSED [100%]{'a': -1, 'b': 2} -========================== 4 passed in 0.13 seconds =========================== -``` - -You see that for each case data, all steps are executed in order. If you use an IDE it will appear in this intuitive order too: - -```bash -case_simple - - step_a - - step_b -case_simple2 - - step_a - - step_b -``` - -If for some reason you wish to invert the order (executing all cases on step a then all cases on step b etc.) simply invert the order of decorators and it will work (`pytest` is great!). This is not recommended though, as it is a lot less intuitive. - -Of course you might want to [enable caching](#caching) so that the cases will be read only once, and not once for each test step. - -### 2- If steps require different data (A: dicts) - -In real-world usage, each step will probably have different expected output or errors for the same case - except if the steps are very similar. The steps may even need slightly different input, for example the same dataset but in two different formats. - -This is actually quite straightforward: simply adapt your custom case data format definition! Since `pytest-cases` does not impose **any** format for your case functions outputs, you can decide that your case functions return lists, dictionaries, etc. - -For example you can choose the format proposed by the `MultipleStepsCaseData` type hint, where each item in the returned inputs/outputs/errors tuple can either be a single element, or a dictionary of name -> element. This allows your case functions to return alternate contents depending on the test step being executed. - -The example below shows a test suite where the inputs of the steps are the same, but the outputs and expected errors are different. Note that once again the example relies on the legacy "parametrizer" mode of pytest-steps, but it would be similar with the new "generator" mode. - -```python -from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, \ - MultipleStepsCaseData -from pytest_steps import test_steps - -# -------- test cases -def case_simple() -> MultipleStepsCaseData: - # common input - ins = dict(a=1, b=2) - - # one expected output for each step - outs_for_a = 2, 3 - outs_for_b = 5, 4 - outs = dict(step_check_a=outs_for_a, step_check_b=outs_for_b) - - return ins, outs, None - -def case_simple2() -> MultipleStepsCaseData: - # common input - ins = dict(a=-1, b=2) - - # one expected output for each step - outs_for_a = 2, 3 - outs_for_b = 5, 4 - outs = dict(step_check_a=outs_for_a, step_check_b=outs_for_b) - - return ins, outs, None - -# ------- test steps -def step_check_a(steps_data, ins, expected_o, expected_e): - """ Step a of the test """ - # Use the three items as usual - ... - # You can also store intermediate results in steps_data - -def step_check_b(steps_data, ins, expected_o, expected_e): - """ Step b of the test """ - # Use the three items as usual - ... - # You can also retrieve intermediate results from steps_data - -# ------- test suite -@test_steps(step_check_a, step_check_b) -@cases_data(module=THIS_MODULE) -def test_suite(test_step, case_data: CaseDataGetter, steps_data): - # Get the case data for all steps (sad...) - ins, expected_o, expected_e = case_data.get() - - # Filter it, based on the step name - key = test_step.__name__ - expected_o = None if expected_o is None else expected_o[key] - expected_e = None if expected_e is None else expected_e[key] - - # Execute the step - test_step(steps_data, ins, expected_o, expected_e) -``` - -There are two main differences with the previous example: - - - in the `case_simple` and `case_simple2` case functions, we choose to provide the expected output and expected error as **dictionaries indexed by the test step name** when they are non-`None`. We also choose that the input is the same for all steps, but we could have done otherwise - using a dictionary with step name keys as well. - - in the final `test_suite` we use the test step name dictionary key to get the case data contents **for a given step**. - -Once again you might want to [enable caching](#caching) in order for the cases to be read only once, and not once for each test step. - -### 3- If steps require different data (B: arguments, recommended) - -The above example might seem a bit disappointing as it breaks the philosophy of doing each data access **only when it is needed**. Indeed everytime a single step is run it actually gets the data for all steps and then has to do some filtering. - -The style suggested below, making use of [case arguments](./advanced#case_arguments), is probably better: - -```python -from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, CaseData -from pytest_steps import test_steps - -# -------- test cases -def case_simple(step_name: str) -> CaseData: - # reuse the same input whatever the step - ins = dict(a=1, b=2) - - # adapt the expected output to the current step - if step_name == 'step_check_a': - outs = 2, 3 - elif step_name == 'step_check_b': - outs = 5, 4 - - return ins, outs, None - -def case_simple2(step_name: str) -> CaseData: - # reuse the same input whatever the step - ins = dict(a=-1, b=2) - - # adapt the expected output to the current step - if step_name == 'step_check_a': - outs = 0, 3 - elif step_name == 'step_check_b': - outs = 1, 4 - - return ins, outs, None - -# ------- test steps -def step_check_a(steps_data, ins, expected_o, expected_e): - """ Step a of the test """ - # Use the three items as usual - ... - # You can also store intermediate results in steps_data - -def step_check_b(steps_data, ins, expected_o, expected_e): - """ Step b of the test """ - # Use the three items as usual - ... - # You can also retrieve intermediate results from steps_data - -# ------- test suite -@test_steps(step_check_a, step_check_b) -@cases_data(module=THIS_MODULE) -def test_suite(test_step, case_data: CaseDataGetter, steps_data): - - # Get the data for this particular case - ins, expected_o, expected_e = case_data.get(test_step.__name__) - - # Execute the step - test_step(steps_data, ins, expected_o, expected_e) -``` - -Notice that now **the test step name is a parameter of the case function**. So for each step, only the data relevant to this step is retrieved. - -Once again you might want to enable caching in order for the cases to be read only once, and not once for each test step. However since the case functions now have arguments, you should **not** use `@lru_cache()` directly on the case function but you should put it in a separate subfunction: - -```python -from pytest_cases import CaseData -from functools import lru_cache - -@lru_cache() -def input_for_case_simple(): - return dict(a=1, b=2) - -def case_simple(step_name: str) -> CaseData: - # reuse the same input whatever the step - ins = input_for_case_simple() - - # adapt the expected output to the current step - if step_name == 'step_check_a': - outs = 2, 3 - elif step_name == 'step_check_b': - outs = 5, 4 - - return ins, outs, None -``` - -That way, `input_for_case_simple` will be cached across the steps. - -See [caching](#caching) for details. - - -## Advanced Pytest: Manual parametrization - -The `@cases_data` decorator is just syntactic sugar for the following two-steps process, that you may wish to rely on for advanced pytest usages: - -```python -import pytest -from pytest_cases import get_all_cases, get_pytest_parametrize_args - -# import the module containing the test cases -import test_foo_cases - -# manually list the available cases -cases = get_all_cases(module=test_foo_cases) - -# transform into required arguments for pytest (applying the pytest marks if needed) -marked_cases, cases_ids = get_pytest_parametrize_args(cases) - -# parametrize the test function manually -@pytest.mark.parametrize('case_data', marked_cases, ids=cases_ids) -def test_with_cases_decorated(case_data): - ... -``` diff --git a/docs/usage/basics.md b/docs/usage/basics.md deleted file mode 100644 index e49d8da8..00000000 --- a/docs/usage/basics.md +++ /dev/null @@ -1,201 +0,0 @@ -# Usage basics - -This page assumes that you have read the [initial example](../#usage). - -## Customizing case names - -By default the name of the case function is used for the generated test case name. To override this, you can use the `@case_name` decorator: - -```python -from pytest_cases import CaseData, case_name - -@case_name("Simplest") -def case_simple_named() -> CaseData: - """ The simplest case but with a custom name using @case_name annotation """ - - ins = dict(a=1, b=2) - outs = 2, 3 - - return ins, outs, None -``` - -## Pytest marks - -Pytest marks such as `@pytest.mark.skipif` can be applied to case functions, the corresponding case will behave as expected. - -## Case generators - -A case function generator is a function that will generate several test cases, once for every combination of its declared input parameters. - - - The function should have at least one parameter - - It should be decorated using `@cases_generator`, passing as keyword arguments one iterable for each parameter - - Since all generated cases need to have a unique name, you should also provide a name template, following the [new string formatting](https://docs.python.org/3.7/library/stdtypes.html#str.format) syntax, and referencing all parameters as keyword arguments: - -```python -from pytest_cases import cases_generator, CaseData - -@cases_generator("case i={i}, j={j}", i=range(2), j=range(3)) -def case_simple_generator(i, j) -> CaseData: - ins = dict(a=i, b=j) - outs = i+1, j+1 - return ins, outs, None -``` - -The above case generator will generate one test case for every combination of argument values, so 6 test cases: - -```bash ->>> pytest -============================= test session starts ============================= -(...) -/tests/test_foo.py::test_foo[case i=0, j=0] PASSED [ 17%] -/tests/test_foo.py::test_foo[case i=0, j=1] PASSED [ 33%] -/tests/test_foo.py::test_foo[case i=0, j=2] PASSED [ 50%] -/tests/test_foo.py::test_foo[case i=1, j=0] PASSED [ 67%] -/tests/test_foo.py::test_foo[case i=1, j=1] PASSED [ 83%] -/tests/test_foo.py::test_foo[case i=1, j=2] PASSED [ 100%] - -========================== 9 passed in 0.84 seconds ========================== -``` - -## Handling Exceptions - -Let's consider the following `foo` function under test, that may raise an exception: - -```python -from math import isfinite - -class InfiniteInput(Exception): - def __init__(self, name): - super(InfiniteInput, self).__init__(name) - - def __eq__(self, other): - return self.args == other.args - -def foo(a, b): - """ - An example function to be tested - - :param a: - :param b: - :return: - """ - if not isfinite(b): - raise InfiniteInput('b') - return a + 1, b + 1 -``` - -`pytest_cases` proposes three ways to perform exception checking: - - - either you provide an expected exception **type**, - - or an expected exception **instance**, - - or an exception validation **callable**. - -The example below illustrates the three ways: - -```python -from math import inf -from example import InfiniteInput -from pytest_cases import CaseData - - -def case_simple_error_type() -> CaseData: - """ An error case with an exception type """ - - ins = dict(a=1, b="a_string_not_an_int") - err = TypeError - - return ins, None, err - - -def case_simple_error_instance() -> CaseData: - """ An error case with an exception instance """ - - ins = dict(a=1, b=inf) - err = InfiniteInput('b') - - return ins, None, err - - -def case_simple_error_callable() -> CaseData: - """ An error case with an exception validation callable """ - - ins = dict(a=1, b=inf) - def is_good_error(e): - return type(e) is InfiniteInput and e.args == ('b',) - - return ins, None, is_good_error -``` - -In order to perform the associated assertions in your test functions, you can simply use the `unfold_expected_err` utility function bundled in `pytest_cases`: - -```python -import pytest -from pytest_cases import cases_data, CaseDataGetter, unfold_expected_err -from example import foo - -# import the module containing the test cases -import test_foo_cases - - -@cases_data(module=test_foo_cases) -def test_with_cases_decorated(case_data: CaseDataGetter): - """ Example unit test that is automatically parametrized with @cases_data """ - - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - if expected_e is None: - # **** Nominal test **** - outs = foo(**i) - assert outs == expected_o - - else: - # **** Error test **** - # First see what we need to assert - err_type, err_inst, err_checker = unfold_expected_err(expected_e) - - # Run with exception capture and type check - with pytest.raises(err_type) as err_info: - foo(**i) - - # Optional - Additional Exception instance check - if err_inst is not None: - assert err_info.value == err_inst - - # Optional - Additional exception instance check - if err_checker is not None: - err_checker(err_info.value) -``` - -## Cases in the same file than Tests - -It is not mandatory that case functions should be in a different file than the test functions: both can be in the same file. For this you can use the `THIS_MODULE` constant to refer to the module in which the test function is located: - -```python -from pytest_cases import CaseData, cases_data, THIS_MODULE, CaseDataGetter - -def case_simple() -> CaseData: - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None - -def case_simple2() -> CaseData: - ins = dict(a=-1, b=2) - outs = 0, 3 - return ins, outs, None - -@cases_data(module=THIS_MODULE) -def test_with_cases_decorated(case_data: CaseDataGetter): - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it - # ... -``` - -However **WARNING**: only the case functions located BEFORE the test function in the module file will be taken into account! - -## To go further - -Are you at ease with the above concepts ? It's time to move to the [intermediate](./intermediate.md) section! diff --git a/docs/usage/intermediate.md b/docs/usage/intermediate.md deleted file mode 100644 index 42082f77..00000000 --- a/docs/usage/intermediate.md +++ /dev/null @@ -1,200 +0,0 @@ -# Intermediate Usage - -You might feel a bit stuck when using only the basics. In particular you might feel frustrated by having to put all cases for the same function in the same dedicated file. - -In this section we see that this is not mandatory at all: `pytest-cases` offers flexibility mechanisms to organize the files the way you wish. - -## Gathering Cases from different files - -You can gather cases from different files (modules) in the same test function: - -```python -from pytest_cases import CaseDataGetter, cases_data - -# the 2 modules containing cases -from . import shared_cases, shared_cases2 - -@cases_data(module=[shared_cases, shared_cases2]) -def test_bar(case_data: CaseDataGetter): - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it: nominal test only - # ... -``` - -## Storing Cases with different purposes in the same file - -This is the opposite of the above: sometimes it would just be tideous to create a **dedicated** cases file for every single test function, you just want to be able to put all of your cases in the same place, even if they are not used in the same test. - -To do this we need to **associate test functions with test cases** in a more fine-grain way. - -### a- Hardcoded cases list - -On the test functions side, you can precisely select the required cases in `@cases_data` using `cases=` - -```python -from pytest_cases import CaseDataGetter, cases_data - -# the module containing the cases above -from .shared_cases import case1, case2, case3 - -# the 2 functions that we want to test -from mycode import foo, bar - -@cases_data(cases=[case1, case2]) -def test_foo(case_data: CaseDataGetter): - """ This test will only be executed on cases tagged with 'foo'""" - - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it: nominal test only - assert expected_e is None - outs = foo(**i) - assert outs == expected_o - -@cases_data(cases=case3) -def test_bar(case_data: CaseDataGetter): - """ This test will only be executed on cases tagged with 'bar'""" - - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it: nominal test only - assert expected_e is None - outs = bar(**i) - assert outs == expected_o -``` - -In the example above, `test_foo` will be applied on `case1` and `case2`, while `test_bar` will be applied on `case3`. They can live in the same file or not, it does not matter. - - -### b- Simple: declare a test target - -The simplest non-hardcoded approach is to use a common reference so that each test function finds the appropriate cases. The function or class under test (in other words, the "test target") might be a good idea to serve this purpose. - -On the cases side, simply use the `@test_target` decorator: - -```python -from pytest_cases import CaseData, test_target - -# the 2 functions that we want to test -from mycode import foo, bar - -# a case only to be used when function foo is under test -@test_target(foo) -def case_foo_simple() -> CaseData: - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None - -# a case only to be used when function bar is under test -@test_target(bar) -def case_bar_simple() -> CaseData: - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None -``` - -On the test functions side, filter the cases in `@cases_data` using `has_tag=`: - -```python -from pytest_cases import CaseDataGetter, cases_data - -# the module containing the cases above -from . import shared_cases - -# the 2 functions that we want to test -from mycode import foo, bar - -@cases_data(module=shared_cases, has_tag=foo) -def test_foo(case_data: CaseDataGetter): - """ This test will only be executed on cases tagged with 'foo'""" - - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it: nominal test only - assert expected_e is None - outs = foo(**i) - assert outs == expected_o - - -@cases_data(module=shared_cases, has_tag=bar) -def test_bar(case_data: CaseDataGetter): - """ This test will only be executed on cases tagged with 'bar'""" - - # 1- Grab the test case data - i, expected_o, expected_e = case_data.get() - - # 2- Use it: nominal test only - assert expected_e is None - outs = bar(**i) - assert outs == expected_o -``` - -Of course this does not prevent other test functions to use all cases by not using any filter. - -### c- Advanced: Tagging & Filtering - -The above example is just a particular case of **tag** put on a case, and **filter** put on the test function. You can actually put **several** tags on the cases, not only a single one (like `@test_target` does): - -```python -from pytest_cases import CaseData, case_tags - -# a case with two tags -@case_tags(bar, 'fast') -def case_multitag_simple() -> CaseData: - ins = dict(a=1, b=2) - outs = 2, 3 - return ins, outs, None -``` - -Test functions can use two things to perform their selection: - - - the `has_tag` parameter, has seen above - - the `filter` parameter, that should be a callable taking as input a list of tags and returning a boolean. - -If both are provided, a AND will be applied. - -For example: - -```python -from pytest_cases import THIS_MODULE, cases_data, CaseDataGetter - -def has_a_or_b(tags): - return 'a' in tags or 'b' in tags - -@cases_data(module=THIS_MODULE, filter=has_a_or_b) -def test_with_cases_a_or_b(case_data: CaseDataGetter): - # ... -``` - -Or with a lambda function: - -```python -from pytest_cases import THIS_MODULE, cases_data, CaseDataGetter - -@cases_data(module=THIS_MODULE, filter=lambda tags: 'a' in tags or 'b' in tags) -def test_with_cases_a_or_b(case_data: CaseDataGetter): - # ... -``` - -Or with a mini lambda expression: - -```python -from pytest_cases import THIS_MODULE, cases_data, CaseDataGetter - -from mini_lambda import InputVar, _ -tags = InputVar('tags', list) - -@cases_data(module=THIS_MODULE, filter=_(tags.contains('a') | tags.contains('b'))) -def test_with_cases_a_or_b(case_data: CaseDataGetter): - # ... -``` - - -## To go further - -Are you at ease with the above concepts ? It's time to move to the [advanced](./advanced.md) section! diff --git a/pytest_cases/__init__.py b/pytest_cases/__init__.py index c65ffaed..d271a5e5 100644 --- a/pytest_cases/__init__.py +++ b/pytest_cases/__init__.py @@ -1,11 +1,20 @@ +from .common_pytest_lazy_values import lazy_value +from .common_others import unfold_expected_err, assert_exception, AUTO, AUTO2 + from .fixture_core1_unions import fixture_union, NOT_USED, unpack_fixture, ignore_unused from .fixture_core2 import pytest_fixture_plus, fixture_plus, param_fixtures, param_fixture -from .fixture_parametrize_plus import pytest_parametrize_plus, parametrize_plus, fixture_ref, lazy_value +from .fixture_parametrize_plus import pytest_parametrize_plus, parametrize_plus, fixture_ref + +# additional symbols without the 'plus' suffix +parametrize = parametrize_plus +fixture = fixture_plus -from .case_funcs import case_name, test_target, case_tags, cases_generator -from .case_parametrizer import cases_data, CaseDataGetter, unfold_expected_err, get_all_cases, THIS_MODULE, \ - get_pytest_parametrize_args, cases_fixture +from .case_funcs_legacy import case_name, test_target, case_tags, cases_generator +from .case_parametrizer_legacy import cases_data, CaseDataGetter, get_all_cases_legacy, \ + get_pytest_parametrize_args_legacy, cases_fixture +from .case_funcs_new import case, CaseInfo +from .case_parametrizer_new import parametrize_with_cases, THIS_MODULE, get_all_cases, get_parametrize_args try: # -- Distribution mode -- @@ -21,21 +30,37 @@ __all__ = [ '__version__', # the submodules - 'case_funcs', 'case_parametrizer', 'fixture_core1_unions', 'fixture_core2', 'fixture_parametrize_plus', + 'common_pytest_lazy_values', 'common_pytest', 'common_others', 'common_mini_six', + 'case_funcs_legacy', 'case_funcs_new', 'case_parametrizer_legacy', 'case_parametrizer_new', + 'fixture_core1_unions', 'fixture_core2', 'fixture_parametrize_plus', + # all symbols imported above + 'unfold_expected_err', 'assert_exception', + + # --fixture core1 + 'fixture_union', 'NOT_USED', 'unpack_fixture', 'ignore_unused', + # -- fixture core2 + 'pytest_fixture_plus', 'fixture_plus', 'fixture', 'param_fixtures', 'param_fixture', + # -- fixture parametrize plus + 'pytest_parametrize_plus', 'parametrize_plus', 'parametrize', 'fixture_ref', 'lazy_value', + + # V1 - DEPRECATED symbols # --cases_funcs - 'lazy_value', 'case_name', 'test_target', 'case_tags', 'cases_generator', - # --main_fixtures - 'cases_fixture', 'pytest_fixture_plus', 'fixture_plus', 'param_fixtures', 'param_fixture', 'ignore_unused', - 'fixture_union', 'NOT_USED', 'pytest_parametrize_plus', 'parametrize_plus', 'fixture_ref', 'unpack_fixture', # --main params - 'cases_data', 'CaseDataGetter', 'THIS_MODULE', 'unfold_expected_err', 'get_all_cases', - 'get_pytest_parametrize_args', + 'cases_data', 'CaseDataGetter', 'get_all_cases_legacy', + 'get_pytest_parametrize_args_legacy', 'cases_fixture', + + # V2 symbols + 'AUTO', 'AUTO2', + # case functions + 'case', 'CaseInfo', 'get_all_cases', + # test functions + 'parametrize_with_cases', 'THIS_MODULE', 'get_parametrize_args' ] try: # python 3.5+ type hints - from pytest_cases.case_funcs import CaseData, Given, ExpectedNormal, ExpectedError, MultipleStepsCaseData + from pytest_cases.case_funcs_legacy import CaseData, Given, ExpectedNormal, ExpectedError, MultipleStepsCaseData __all__ += ['CaseData', 'Given', 'ExpectedNormal', 'ExpectedError', 'MultipleStepsCaseData'] except ImportError: pass diff --git a/pytest_cases/case_funcs.py b/pytest_cases/case_funcs_legacy.py similarity index 71% rename from pytest_cases/case_funcs.py rename to pytest_cases/case_funcs_legacy.py index 743c338d..1b1e066d 100644 --- a/pytest_cases/case_funcs.py +++ b/pytest_cases/case_funcs_legacy.py @@ -1,52 +1,29 @@ from __future__ import division +from itertools import product +from warnings import warn + from decopatch import DECORATED, function_decorator, with_parenthesis try: # python 3.2+ from functools import lru_cache as lru except ImportError: - from functools32 import lru_cache as lru - -from itertools import product + from functools32 import lru_cache as lru # noqa try: # python 3.5+ - from typing import Callable, Union, Optional, Any, Tuple, Dict, Iterable - - # Type hints that you can use in your functions - Given = Any - """The input(s) for the test. It can be anything""" - - ExpectedNormal = Optional[Any] - """The expected test results in case success is expected, or None if this test should fail""" - - ExpectedError = Optional[Union['Type[Exception]', Exception, Callable[[Exception], Optional[bool]]]] - """The expected error in case failure is expected, or None if the test should succeed. It is proposed that expected - error can be defined as an exception type, an exception instance, or an exception validation function""" - - CaseData = Tuple[Given, ExpectedNormal, ExpectedError] - - MultipleStepsCaseData = Tuple[Union[Given, Dict[Any, Given]], - Union[ExpectedNormal, Dict[Any, ExpectedNormal]], - Union[ExpectedError, Dict[Any, ExpectedError]]] -except: - pass - -try: # python 3.5.4+ - from typing import Type -except: - # on old versions of typing module the above does not work. Since our code below has all Type hints quoted it's ok + from typing import Type, Callable, Union, Optional, Any, Tuple, Dict, Iterable +except ImportError: pass - -_GENERATOR_FIELD = '__cases_generator__' -"""Internal marker used for cases generators""" +from .case_funcs_new import CaseInfo @function_decorator def case_name(name, # type: str - test_func=DECORATED + case_func=DECORATED # noqa ): - """ + """ DEPRECATED - use `@case(id=...)` + Decorator to override the name of a case function. The new name will be used instead of the function name, in test names. @@ -59,17 +36,16 @@ def case_simple(): :param name: the name that will be used in the test case instead of the case function name :return: """ - test_func.__name__ = name - return test_func - - -CASE_TAGS_FIELD = '__case_tags__' + warn("`@case_name` is deprecated. Please use `@case(id=)`.", category=DeprecationWarning, stacklevel=2) + case_func.__name__ = name + return case_func @function_decorator(custom_disambiguator=with_parenthesis) def case_tags(*tags # type: Any ): - """ + """ DEPRECATED - use `@case(tags=...)` + Decorator to tag a case function with a list of tags. These tags can then be used in the `@cases_data` test function decorator to filter cases within the selected module(s). @@ -77,15 +53,11 @@ def case_tags(*tags # type: Any functions...) :return: """ + warn("`@case_tags` is deprecated. Please use `@case(tags=(,...))`.", category=DeprecationWarning, stacklevel=2) + # we have to use "nested" mode for this decorator because in the decorator signature we have a var-positional def _apply(case_func): - existing_tags = getattr(case_func, CASE_TAGS_FIELD, None) - if existing_tags is None: - # there are no tags yet. Use the provided - setattr(case_func, CASE_TAGS_FIELD, list(tags)) - else: - # there are some tags already, let's try to add the new to the existing - setattr(case_func, CASE_TAGS_FIELD, existing_tags + list(tags)) + CaseInfo.get_from(case_func, create=True).add_tags(tags) return case_func return _apply @@ -93,7 +65,8 @@ def _apply(case_func): def test_target(target # type: Any ): - """ + """ DEPRECATED - use `@case(target=...)` + A simple decorator to declare that a case function is associated with a particular target. >>> @test_target(int) @@ -105,17 +78,32 @@ def test_target(target # type: Any :param target: for example a function, a class... or a string representing a function, a class... :return: """ + warn("`@test_target` is deprecated. Please use `@case(target=)`.", category=DeprecationWarning, stacklevel=2) return case_tags(target) test_target.__test__ = False # disable this function in pytest (otherwise name starts with 'test' > it will appear) +_GENERATOR_FIELD = '__cases_generator__' +"""Internal marker used for cases generators""" + + +def is_case_generator(f): + return hasattr(f, _GENERATOR_FIELD) + + +def get_case_generator_details(f): + return getattr(f, _GENERATOR_FIELD, False) + + @function_decorator -def cases_generator(names=None, # type: Union[str, Callable[[Any], str], Iterable[str]] - lru_cache=False, # type: bool, - case_func=DECORATED, - **param_ranges # type: Iterable[Any] +def cases_generator(names=None, # type: Union[str, Callable[[Any], str], Iterable[str]] + target=None, # type: Any + tags=None, # type: Iterable[Any] + lru_cache=False, # type: bool, + case_func=DECORATED, # noqa + **param_ranges # type: Iterable[Any] ): """ Decorator to declare a case function as being a cases generator. `param_ranges` should be a named list of parameter @@ -147,9 +135,35 @@ def cases_generator(names=None, # type: Union[str, Callable[[Any], str], I """ kwarg_values = list(product(*param_ranges.values())) setattr(case_func, _GENERATOR_FIELD, (names, param_ranges.keys(), kwarg_values)) + if lru_cache: nb_cases = len(kwarg_values) # decorate the function with the appropriate lru cache size case_func = lru(maxsize=nb_cases)(case_func) return case_func + + +# Optional type hints that you can use in your case functions +try: + from typing import Any + from .common_pytest import ExpectedError + + Given = Any + """The input(s) for the test. It can be anything""" + + ExpectedNormal = Optional[Any] + """The expected test results in case success is expected, or None if this test should fail""" + + # ------ The two main symbols ---- + + CaseData = Tuple[Given, ExpectedNormal, ExpectedError] + """Represents the output of a case function when you choose to use this in/out/err pattern""" + + MultipleStepsCaseData = Tuple[Union[Given, Dict[Any, Given]], + Union[ExpectedNormal, Dict[Any, ExpectedNormal]], + Union[ExpectedError, Dict[Any, ExpectedError]]] + """Represents the output of a case function when you have a different expected result for each test step""" + +except: # noqa + pass diff --git a/pytest_cases/case_funcs_new.py b/pytest_cases/case_funcs_new.py new file mode 100644 index 00000000..ab7434da --- /dev/null +++ b/pytest_cases/case_funcs_new.py @@ -0,0 +1,220 @@ +from copy import copy + +from decopatch import function_decorator, DECORATED + +# try: # python 3.2+ +# from functools import lru_cache as lru +# except ImportError: +# from functools32 import lru_cache as lru # noqa + +try: # python 3.5+ + from typing import Type, Callable, Union, Optional, Any, Tuple, Dict, Iterable, List, Set +except ImportError: + pass + +from .common_mini_six import string_types +from .common_pytest import safe_isclass + + +CASE_FIELD = '_pytestcase' + + +class CaseInfo(object): + """ + Contains all information available about a case. + It is attached to a case function as an attribute + """ + __slots__ = ('id', 'marks', 'tags') + + def __init__(self, + id=None, # type: str + marks=(), # type: Tuple[MarkDecorator] + tags=() # type: Tuple[Any] + ): + self.id = id + self.marks = marks + self.tags = () + self.add_tags(tags) + + @classmethod + def get_from(cls, case_func, create=False, prefix_for_ids='case_'): + """ + Returns the CaseInfo associated with case_fun ; creates it and attaches it if needed and required. + If not present, a case id is automatically created from the function name based on the collection prefix. + + :param case_func: + :param create: + :param prefix_for_ids: + :return: + """ + case_info = getattr(case_func, CASE_FIELD, None) + + if create: + if case_info is None: + case_info = CaseInfo() + case_info.attach_to(case_func) + + if case_info.id is None: + # default test id from function name + if case_func.__name__.startswith(prefix_for_ids): + case_info.id = case_func.__name__[len(prefix_for_ids):] + else: + case_info.id = case_func.__name__ + + return case_info + + def attach_to(self, + case_func # type: Callable + ): + """attach this case_info to the given case function""" + setattr(case_func, CASE_FIELD, self) + + def add_tags(self, + tags # type: Union[Any, Union[List, Set, Tuple]] + ): + """add the given tag or tags""" + if tags: + if isinstance(tags, string_types) or not isinstance(tags, (set, list, tuple)): + # a single tag, create a tuple around it + tags = (tags,) + + self.tags += tuple(tags) + + def matches_tag_query(self, + has_tag=None, # type: Union[str, Iterable[str]] + ): + """ + Returns True if the case function with this case_info is selected by the query + + :param has_tag: + :return: + """ + if has_tag is None: + return True + + if not isinstance(has_tag, (tuple, list, set)): + has_tag = (has_tag,) + + return all(t in self.tags for t in has_tag) + + @classmethod + def copy_info(cls, from_case_func, to_case_func): + case_info = cls.get_from(from_case_func) + if case_info is not None: + cp = copy(case_info) + cp.attach_to(to_case_func) + + +def matches_tag_query(case_fun, + has_tag=None, # type: Union[str, Iterable[str]] + filter=None, # type: Union[Callable[[Iterable[Any]], bool], Iterable[Callable[[Iterable[Any]], bool]]] # noqa + ): + """ + Returns True if the case function is selected by the query: + + - if `has_tag` contains one or several tags, they should ALL be present in the tags + set on `case_fun` (`case_fun._pytestcase.tags`) + + - if `filter` contains one or several filter callables, they are all called in sequence and the + case_fun is only selected if ALL of them return a True truth value + + :param case_fun: + :param has_tag: + :param filter: + :return: True if the case_fun is selected by the query. + """ + selected = True + + # query on tags + if has_tag is not None: + selected = selected and CaseInfo.get_from(case_fun).matches_tag_query(has_tag) + + # filter function + if filter is not None: + if not isinstance(filter, (tuple, set, list)): + filter = (filter,) + + for _filter in filter: + # break if already unselected + if not selected: + return selected + + # try next filter + try: + res = _filter(case_fun) + # keep this in the try catch in case there is an issue with the truth value of result + selected = selected and res + except: # noqa + # any error leads to a no-match + selected = False + + return selected + + +@function_decorator +def case(id=None, # type: str # noqa + tags=None, # type: Union[Any, Iterable[Any]] + marks=(), # type: Union[MarkDecorator, Iterable[MarkDecorator]] + case_func=DECORATED # noqa + ): + """ + Optional decorator for case functions so as to customize some information. + + ```python + @case(id='hey') + def case_hi(): + return 1 + ``` + + :param id: the custom pytest id that should be used when this case is active. Replaces the deprecated `@case_name` + decorator from v1. If no id is provided, the id is generated from case functions by removing their prefix, + see `@parametrize_with_cases(prefix='case_')`. + :param tags: custom tags to be used for filtering in `@parametrize_with_cases(has_tags)`. Replaces the deprecated + `@case_tags` and `@target` decorators. + :param marks: optional pytest marks to add on the case. Note that decorating the function directly with the mark + also works, and if marks are provided in both places they are merged. + :return: + """ + case_info = CaseInfo(id, marks, tags) + case_info.attach_to(case_func) + return case_func + + +CASE_PREFIX_CLS = 'Case' +"""Prefix used by default to identify case classes""" + +CASE_PREFIX_FUN = 'case_' +"""Prefix used by default to identify case functions within a module""" + + +def is_case_class(cls, case_marker_in_name=CASE_PREFIX_CLS, check_name=True): + """ + Returns True if the given object is a class and, if `check_name=True` (default), if its name contains + `case_marker_in_name`. + + :param cls: the object to check + :param case_marker_in_name: the string that should be present in a class name so that it is selected. Default is + 'Case'. + :param check_name: a boolean (default True) to enforce that the name contains the word `case_marker_in_name`. + If False, all classes will lead to a `True` result whatever their name. + :return: True if this is a case class + """ + return safe_isclass(cls) and (not check_name or case_marker_in_name in cls.__name__) + + +def is_case_function(f, prefix=CASE_PREFIX_FUN, check_prefix=True): + """ + Returns True if the provided object is a function or callable and, if `check_prefix=True` (default), if it starts + with `prefix`. + + :param f: + :param prefix: + :param check_prefix: + :return: + """ + if not callable(f): + return False + elif safe_isclass(f): + return False + else: + return f.__name__.startswith(prefix) if check_prefix else True diff --git a/pytest_cases/case_parametrizer.py b/pytest_cases/case_parametrizer_legacy.py similarity index 61% rename from pytest_cases/case_parametrizer.py rename to pytest_cases/case_parametrizer_legacy.py index fce6b5b9..571c08ca 100644 --- a/pytest_cases/case_parametrizer.py +++ b/pytest_cases/case_parametrizer_legacy.py @@ -4,7 +4,6 @@ import sys from abc import abstractmethod, ABCMeta from decopatch import function_decorator, DECORATED, with_parenthesis -from inspect import getmembers from warnings import warn try: # python 3.3+ @@ -16,7 +15,7 @@ try: # type hints, python 3+ from typing import Type, Callable, Union, Optional, Any, Tuple, List, Dict, Iterable # noqa - from pytest_cases.case_funcs import CaseData, ExpectedError # noqa + from pytest_cases.case_funcs_legacy import CaseData, ExpectedError # noqa from types import ModuleType # noqa # Type hint for the simple functions @@ -24,18 +23,22 @@ # Type hint for generator functions GeneratedCaseFunc = Callable[[Any], CaseData] -except ImportError: +except: # noqa pass from .common_mini_six import with_metaclass, string_types -from .common_pytest import make_marked_parameter_value, get_pytest_marks_on_function +from .common_pytest_marks import get_pytest_marks_on_function, make_marked_parameter_value + +from .case_funcs_legacy import is_case_generator, get_case_generator_details +from .case_funcs_new import matches_tag_query +from .case_parametrizer_new import THIS_MODULE, extract_cases_from_module -from .case_funcs import _GENERATOR_FIELD, CASE_TAGS_FIELD from .fixture_core2 import fixture_plus class CaseDataGetter(with_metaclass(ABCMeta)): - """ + """ DEPRECATED - this was created in v1 only by `@cases_data` (not by v2 `@parametrize_with_cases`) + A proxy for a test case. Instances of this class are created by `@cases_data` or `get_all_cases`. It provides a single method: `get(self, *args, **kwargs) -> CaseData` @@ -84,7 +87,8 @@ def get_for(self, key): class CaseDataFromFunction(CaseDataGetter): - """ + """ DEPRECATED - this was created in v1 only by `@cases_data` (not by v2 `@parametrize_with_cases`) + A CaseDataGetter relying on a function """ @@ -128,13 +132,6 @@ def get(self, *args, **kwargs): return self.f(*args, **kwargs) -CASE_PREFIX = 'case_' -"""Prefix used by default to identify case functions within a module""" - -THIS_MODULE = object() -"""Marker that can be used instead of a module name to indicate that the module is the current one""" - - @function_decorator(custom_disambiguator=with_parenthesis) def cases_data(cases=None, # type: Union[Callable[[Any], Any], Iterable[Callable[[Any], Any]]] module=None, # type: Union[ModuleType, Iterable[ModuleType]] @@ -143,7 +140,8 @@ def cases_data(cases=None, # type: Union[Callable[[Any], A filter=None, # type: Callable[[List[Any]], bool] # noqa test_func=DECORATED, # noqa ): - """ + """ DEPRECATED (V1) - use V2 `@parametrize_with_cases(argnames, ...)` + Decorates a test function so as to automatically parametrize it with all cases listed in module `module`, or with all cases listed explicitly in `cases`. @@ -197,13 +195,16 @@ def test_foo(case_data: CaseData): `module`. It both `has_tag` and `filter` are set, both will be applied in sequence. :return: """ + warn("`@cases_data` is deprecated. Please use `@parametrize_with_cases`.", category=DeprecationWarning, + stacklevel=2) + # equivalent to @mark.parametrize('case_data', cases) where cases is a tuple containing a CaseDataGetter for # First list all cases according to user preferences - _cases = get_all_cases(cases, module, test_func, has_tag, filter) + _cases = get_all_cases_legacy(cases, module, test_func, has_tag, filter) # Then transform into required arguments for pytest (applying the pytest marks if needed) - marked_cases, cases_ids = get_pytest_parametrize_args(_cases) + marked_cases, cases_ids = get_pytest_parametrize_args_legacy(_cases) # Finally create the pytest decorator and apply it parametrizer = pytest.mark.parametrize(case_data_argname, marked_cases, ids=cases_ids) @@ -211,7 +212,7 @@ def test_foo(case_data: CaseData): return parametrizer(test_func) -def get_pytest_parametrize_args(cases): +def get_pytest_parametrize_args_legacy(cases): """ Transforms a list of cases into a tuple containing the arguments to use in `@pytest.mark.parametrize` the tuple is (marked_cases, ids) where @@ -227,18 +228,18 @@ def get_pytest_parametrize_args(cases): case_ids = [str(c) for c in cases] # create the pytest parameter values with the appropriate pytest marks - marked_cases = [c if len(c.get_marks()) == 0 else make_marked_parameter_value(c, marks=c.get_marks()) + marked_cases = [c if len(c.get_marks()) == 0 else make_marked_parameter_value((c,), marks=c.get_marks()) for c in cases] return marked_cases, case_ids -def get_all_cases(cases=None, # type: Union[Callable[[Any], Any], Iterable[Callable[[Any], Any]]] - module=None, # type: Union[ModuleType, Iterable[ModuleType]] - this_module_object=None, # type: Any - has_tag=None, # type: Any - filter=None # type: Callable[[List[Any]], bool] # noqa - ): +def get_all_cases_legacy(cases=None, # type: Union[Callable[[Any], Any], Iterable[Callable[[Any], Any]]] + module=None, # type: Union[ModuleType, Iterable[ModuleType]] + this_module_object=None, # type: Any + has_tag=None, # type: Any + filter=None # type: Callable[[List[Any]], bool] # noqa + ): # type: (...) -> List[CaseDataGetter] """ Lists all desired cases from the user inputs. This function may be convenient for debugging purposes. @@ -255,6 +256,7 @@ def get_all_cases(cases=None, # type: Union[Callable[[Any], Any], `module`. It both `has_tag` and `filter` are set, both will be applied in sequence. :return: """ + _facto = _get_case_getter_s(has_tag=has_tag, filter=filter) _cases = None if module is not None and cases is not None: @@ -264,19 +266,19 @@ def get_all_cases(cases=None, # type: Union[Callable[[Any], Any], # Hardcoded sequence of cases, or single case if callable(cases): # single element - _cases = [case_getter for case_getter in _get_case_getter_s(cases)] + _cases = [case_getter for case_getter in _facto(cases)] elif cases is THIS_MODULE: raise ValueError("`THIS_MODULE` should only be used in the `module` argument, not in the `cases` argument") else: # already a sequence - _cases = [case_getter for c in cases for case_getter in _get_case_getter_s(c)] + _cases = [case_getter for c in cases for case_getter in _facto(c)] else: # Gather all cases from the reference module(s) try: _cases = [] for m in module: # noqa m = sys.modules[this_module_object.__module__] if m is THIS_MODULE else m - _cases += extract_cases_from_module(m, has_tag=has_tag, filter=filter) + _cases += extract_cases_from_module(m, _case_param_factory=_facto) success = True except TypeError: success = False @@ -284,76 +286,11 @@ def get_all_cases(cases=None, # type: Union[Callable[[Any], Any], if not success: # 'module' object is not iterable: a single module was provided m = sys.modules[this_module_object.__module__] if module is THIS_MODULE else module - _cases = extract_cases_from_module(m, has_tag=has_tag, filter=filter) + _cases = extract_cases_from_module(m, _case_param_factory=_facto) return _cases -def _get_code(f): - """ - Returns the source code associated to function f. It is robust to wrappers such as @lru_cache - :param f: - :return: - """ - if hasattr(f, '__wrapped__'): - return _get_code(f.__wrapped__) - elif hasattr(f, '__code__'): - return f.__code__ - else: - raise ValueError("Cannot get code information for function " + str(f)) - - -def extract_cases_from_module(module, # type: ModuleType - has_tag=None, # type: Any - filter=None # type: Callable[[List[Any]], bool] # noqa - ): - # type: (...) -> List[CaseDataGetter] - """ - Internal method used to create a list of `CaseDataGetter` for all cases available from the given module. - See `@cases_data` - - :param module: - :param has_tag: a tag used to filter the cases. Only cases with the given tag will be selected - :param filter: a function taking as an input a list of tags associated with a case, and returning a boolean - indicating if the case should be selected - :return: - """ - if filter is not None and not callable(filter): - raise ValueError("`filter` should be a callable starting in pytest-cases 0.8.0. If you wish to provide a single" - " tag to match, use `has_tag` instead.") - - # First gather all case data providers in the reference module - cases_dct = dict() - for f_name, f in getmembers(module, callable): - # only keep the functions - # - from the module file (not the imported ones), - # - starting with prefix 'case_' - if f_name.startswith(CASE_PREFIX): - code = _get_code(f) - # check if the function is actually defined in this module (not imported from elsewhere) - # Note: we used code.co_filename == module.__file__ in the past - # but on some targets the file changes to a cached one so this does not work reliably, - # see https://github.com/smarie/python-pytest-cases/issues/72 - if f.__module__ == module.__name__: - # - with the optional filter/tag - _tags = getattr(f, CASE_TAGS_FIELD, ()) - - selected = True # by default select the case, then AND the conditions - if has_tag is not None: - selected = selected and (has_tag in _tags) - if filter is not None: - selected = selected and filter(_tags) - - if selected: - # update the dictionary with the case getters - _get_case_getter_s(f, code, cases_dct) - - # convert into a list, taking all cases in order of appearance in the code (sort by source code line number) - cases = [cases_dct[k] for k in sorted(cases_dct.keys())] - - return cases - - class InvalidNamesTemplateException(Exception): """ Raised when a `@cases_generator` is used with an improper name template and formatting fails. @@ -369,123 +306,99 @@ def __str__(self): "%s. Please check the name template." % (self.cases_func.__name__, self.names_template, self.params) -def _get_case_getter_s(f, - f_code=None, - cases_dct=None): - # type: (...) -> Optional[List[CaseDataFromFunction]] - """ - Creates the case function getter or the several cases function getters (in case of a generator) associated with - function f. If cases_dct is provided, they are stored in this dictionary with a key equal to their code line number. - For generated cases, a floating line number is created to preserve order. - - :param f: - :param f_code: should be provided if cases_dct is provided. - :param cases_dct: an optional dictionary where to store the created function wrappers - :return: - """ - - # create a return variable if needed - if cases_dct is None: - cases_list = [] - else: - cases_list = None - - # Handle case generators - gen = getattr(f, _GENERATOR_FIELD, False) - if gen: - already_used_names = [] - - names, param_ids, all_param_values_combinations = gen - nb_cases_generated = len(all_param_values_combinations) - - if names is None: - # default template based on parameter names - names = "%s__%s" % (f.__name__, ', '.join("%s={%s}" % (p_name, p_name) for p_name in param_ids)) - - if isinstance(names, string_types): - # then this is a string formatter creating the names. Create the corresponding callable - _formatter = names - - def names(**params): - try: - return _formatter.format(**params) - except Exception: - raise InvalidNamesTemplateException(f, _formatter, params) - - if not callable(names): - # This is an explicit list - if len(names) != nb_cases_generated: - raise ValueError("An explicit list of names has been provided but it has not the same length (%s) than" - " the number of cases to be generated (%s)" % (len(names), nb_cases_generated)) - - for gen_case_id, case_params_values in enumerate(all_param_values_combinations): - # build the dictionary of parameters for the case functions - gen_case_params_dct = dict(zip(param_ids, case_params_values)) - - # generate the case name by applying the name template - if callable(names): - gen_case_name = names(**gen_case_params_dct) - else: - # an explicit list is provided - gen_case_name = names[gen_case_id] - - if gen_case_name in already_used_names: - raise ValueError("Generated function names for generator case function {} are not " - "unique. Please use all parameter names in the string format variables" - "".format(f.__name__)) - else: - already_used_names.append(gen_case_name) - case_getter = CaseDataFromFunction(f, gen_case_name, gen_case_params_dct) +def _get_case_getter_s(has_tag=None, filter=None): + if filter is not None and not callable(filter): + raise ValueError("`filter` should be a callable starting in pytest-cases 0.8.0. If you wish to provide a single" + " tag to match, use `has_tag` instead.") - # save the result in the list or the dict - if cases_dct is None: - cases_list.append(case_getter) - else: - # with an artificial floating point line number to keep order in dict - gen_line_nb = f_code.co_firstlineno + (gen_case_id / nb_cases_generated) - cases_dct[gen_line_nb] = case_getter - else: - # single case - case_getter = CaseDataFromFunction(f) + def __get_case_getter_s(f, + co_firstlineno=None, + cases_dct=None): + # type: (...) -> Optional[List[CaseDataFromFunction]] + """ + Creates the case function getter or the several cases function getters (in case of a generator) associated with + function f. If cases_dct is provided, they are stored in this dictionary with a key equal to their code line number. + For generated cases, a floating line number is created to preserve order. - # save the result + :param f: + :param co_firstlineno: should be provided if cases_dct is provided. + :param cases_dct: an optional dictionary where to store the created function wrappers + :return: + """ + # create a return variable if needed if cases_dct is None: - cases_list.append(case_getter) + cases_list = [] else: - cases_dct[f_code.co_firstlineno] = case_getter - - if cases_dct is None: - return cases_list - - -def unfold_expected_err(expected_e # type: ExpectedError - ): - # type: (...) -> Tuple[Optional['Type[Exception]'], Optional[Exception], Optional[Callable[[Exception], bool]]] - """ - 'Unfolds' the expected error `expected_e` to return a tuple of - - expected error type - - expected error instance - - error validation callable - - If `expected_e` is an exception type, returns `expected_e, None, None` - If `expected_e` is an exception instance, returns `type(expected_e), expected_e, None` - If `expected_e` is an exception validation function, returns `Exception, None, expected_e` - - :param expected_e: an `ExpectedError`, that is, either an exception type, an exception instance, or an exception - validation function - :return: - """ - if type(expected_e) is type and issubclass(expected_e, Exception): - return expected_e, None, None + cases_list = None + + if matches_tag_query(f, has_tag=has_tag, filter=filter): + # Handle case generators + if is_case_generator(f): + already_used_names = [] + + names, param_ids, all_param_values_combinations = get_case_generator_details(f) + nb_cases_generated = len(all_param_values_combinations) + + if names is None: + # default template based on parameter names + names = "%s__%s" % (f.__name__, ', '.join("%s={%s}" % (p_name, p_name) for p_name in param_ids)) + + if isinstance(names, string_types): + # then this is a string formatter creating the names. Create the corresponding callable + _formatter = names + + def names(**params): + try: + return _formatter.format(**params) + except Exception: + raise InvalidNamesTemplateException(f, _formatter, params) + + if not callable(names): + # This is an explicit list + if len(names) != nb_cases_generated: + raise ValueError("An explicit list of names has been provided but it has not the same length (%s) than" + " the number of cases to be generated (%s)" % (len(names), nb_cases_generated)) + + for gen_case_id, case_params_values in enumerate(all_param_values_combinations): + # build the dictionary of parameters for the case functions + gen_case_params_dct = dict(zip(param_ids, case_params_values)) + + # generate the case name by applying the name template + if callable(names): + gen_case_name = names(**gen_case_params_dct) + else: + # an explicit list is provided + gen_case_name = names[gen_case_id] + + if gen_case_name in already_used_names: + raise ValueError("Generated function names for generator case function {} are not " + "unique. Please use all parameter names in the string format variables" + "".format(f.__name__)) + else: + already_used_names.append(gen_case_name) + case_getter = CaseDataFromFunction(f, gen_case_name, gen_case_params_dct) + + # save the result in the list or the dict + if cases_dct is None: + cases_list.append(case_getter) + else: + # with an artificial floating point line number to keep order in dict + gen_line_nb = co_firstlineno + (gen_case_id / nb_cases_generated) + cases_dct[gen_line_nb] = case_getter + else: + # single case + case_getter = CaseDataFromFunction(f) - elif issubclass(type(expected_e), Exception): - return type(expected_e), expected_e, None + # save the result + if cases_dct is None: + cases_list.append(case_getter) + else: + cases_dct[co_firstlineno] = case_getter - elif callable(expected_e): - return Exception, None, expected_e + if cases_dct is None: + return cases_list - raise ValueError("ExpectedNormal error should either be an exception type, an exception instance, or an exception " - "validation callable") + return __get_case_getter_s @function_decorator @@ -497,8 +410,7 @@ def cases_fixture(cases=None, # type: Union[Callable[[Any] f=DECORATED, # noqa **kwargs ): - """ - DEPRECATED - use double annotation `@fixture_plus` + `@cases_data` instead + """ DEPRECATED - use double annotation `@fixture_plus` + `@parametrize_with_cases` instead ```python @fixture_plus @@ -562,6 +474,9 @@ def foo_fixture(request): `module`. It both `has_tag` and `filter` are set, both will be applied in sequence. :return: """ + warn("`@cases_fixture` is deprecated. Please use `@fixture_plus` with `@parametrize_with_cases(id=)` (V2) " + "or `@cases_data` (V1).", category=DeprecationWarning, stacklevel=2) + # apply @cases_data (that will translate to a @pytest.mark.parametrize) parametrized_f = cases_data(cases=cases, module=module, case_data_argname=case_data_argname, has_tag=has_tag, filter=filter)(f) diff --git a/pytest_cases/case_parametrizer_new.py b/pytest_cases/case_parametrizer_new.py new file mode 100644 index 00000000..6771ccef --- /dev/null +++ b/pytest_cases/case_parametrizer_new.py @@ -0,0 +1,613 @@ +# Use true division operator always even in old python 2.x (used in `_extract_cases_from_module`) +from __future__ import division + +from functools import partial +from importlib import import_module +from inspect import getmembers +import re +from warnings import warn + +from pytest_cases import fixture + +try: + from typing import Union, Callable, Iterable, Any, Type, List, Tuple # noqa +except ImportError: + pass + +from .common_mini_six import string_types +from .common_others import get_code_first_line, AUTO, AUTO2 +from .common_pytest_marks import copy_pytest_marks, make_marked_parameter_value +from .common_pytest_lazy_values import lazy_value +from .common_pytest import safe_isclass, MiniMetafunc + +from .case_funcs_new import matches_tag_query, is_case_function, is_case_class, CaseInfo, CASE_PREFIX_FUN +from .fixture_parametrize_plus import fixture_ref, _parametrize_plus + +THIS_MODULE = object() +"""Singleton that can be used instead of a module name to indicate that the module is the current one""" + +try: + from typing import Literal # noqa + from types import ModuleType # noqa + + ModuleRef = Union[str, ModuleType, Literal[AUTO], Literal[AUTO2], Literal[THIS_MODULE]] # noqa + +except: # noqa + pass + + +def parametrize_with_cases(argnames, # type: str + cases=AUTO, # type: Union[Callable, Type, ModuleRef] + prefix=CASE_PREFIX_FUN, # type: str + glob=None, # type: str + has_tag=None, # type: Any + filter=None, # type: Callable[[Callable], bool] # noqa + **kwargs + ): + # type: (...) -> Callable[[Callable], Callable] + """ + A decorator for test functions or fixtures, to parametrize them based on test cases. It works similarly to + `@pytest.mark.parametrize`: argnames represent a coma-separated string of arguments to inject in the decorated + test function or fixture. The argument values (argvalues in `pytest.mark.parametrize`) are collected from the + various case functions found according to `cases`, and injected as lazy values so that the case functions are called + just before the test or fixture is executed. + + By default (`cases=AUTO`) the list of test cases is automatically drawn from the python module file named + `test__cases.py` where `test_` is the current module name. An alternate naming convention + `cases_.py` can be used by setting `cases=AUTO2`. + + Finally, the `cases` argument also accepts an explicit case function, cases-containing class, module or module name; + or a list of such elements. Note that both absolute and relative module names are suported. + + Note that `@parametrize_with_cases` collection and parameter creation steps are strictly equivalent to + `get_all_cases` + `get_parametrize_args`. This can be handy for debugging purposes. + + ```python + # Collect all cases + cases_funs = get_all_cases(f, cases=cases, prefix=prefix, glob=glob, has_tag=has_tag, filter=filter) + + # Transform the various functions found + argvalues = get_parametrize_args(cases_funs, prefix=prefix) + ``` + + :param argnames: same than in @pytest.mark.parametrize + :param cases: a case function, a class containing cases, a module object or a module name string (relative module + names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module. + `AUTO` (default) means that the module named `test__cases.py` will be loaded, where `test_.py` is + the module file of the decorated function. `AUTO2` allows you to use the alternative naming scheme + `case_.py`. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag` + are selected, including those functions nested in classes following naming pattern `*Case*`. When classes are + explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern. + :param prefix: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to + denote different kind of cases, for example 'data_', 'algo_', 'user_', etc. + :param glob: an optional glob-like pattern for case ids, for example "*_success" or "*_failure". Note that this + is applied on the case id, and therefore if it is customized through `@case(id=...)` it should be taken into + account. + :param has_tag: a single tag or a tuple, set, list of tags that should be matched by the ones set with the `@case` + decorator on the case function(s) to be selected. + :param filter: a callable receiving the case function and returning True or a truth value in case the function + needs to be selected. + :return: + """ + def _apply_parametrization(f): + """ execute parametrization of test function or fixture `f` """ + + # Collect all cases + cases_funs = get_all_cases(f, cases=cases, prefix=prefix, glob=glob, has_tag=has_tag, filter=filter) + + # Transform the various functions found + argvalues = get_parametrize_args(cases_funs) + + # Finally apply parametrization - note that we need to call the private method so that fixture are created in + # the right module (not here) + _parametrize_with_cases = _parametrize_plus(argnames, argvalues, **kwargs) + return _parametrize_with_cases(f) + + return _apply_parametrization + + +def create_glob_name_filter(glob_str # type: str + ): + """ + Creates a glob-like matcher for the name of case functions + + :param case_fun: + :return: + """ + name_matcher = re.compile(glob_str.replace("*", ".*")) + + def _glob_name_filter(case_fun): + case_fun_id = case_fun._pytestcase.id + assert case_fun_id is not None + return name_matcher.match(case_fun_id) + + return _glob_name_filter + + +def get_all_cases(parametrization_target, # type: Callable + cases=None, # type: Union[Callable, Type, ModuleRef] + prefix=CASE_PREFIX_FUN, # type: str + glob=None, # type: str + has_tag=None, # type: Union[str, Iterable[str]] + filter=None # type: Callable[[Callable], bool] # noqa + ): + # type: (...) -> List[Callable] + """ + Lists all desired cases for a given `parametrization_target` (a test function or a fixture). This function may be + convenient for debugging purposes. See `@parametrize_with_cases` for details on the parameters. + + :param parametrization_target: a test function + :param cases: a case function, a class containing cases, a module or a module name string (relative module + names accepted). Or a list of such items. You may use `THIS_MODULE` or `'.'` to include current module. + `AUTO` (default) means that the module named `test__cases.py` will be loaded, where `test_.py` is + the module file of the decorated function. `AUTO2` allows you to use the alternative naming scheme + `case_.py`. When a module is listed, all of its functions matching the `prefix`, `filter` and `has_tag` + are selected, including those functions nested in classes following naming pattern `*Case*`. When classes are + explicitly provided in the list, they can have any name and do not need to follow this `*Case*` pattern. + :param prefix: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to + denote different kind of cases, for example 'data_', 'algo_', 'user_', etc. + :param glob: an optional glob-like pattern for case ids, for example "*_success" or "*_failure". Note that this + is applied on the case id, and therefore if it is customized through `@case(id=...)` it should be taken into + account. + :param has_tag: a single tag or a tuple, set, list of tags that should be matched by the ones set with the `@case` + decorator on the case function(s) to be selected. + :param filter: a callable receiving the case function and returning True or a truth value in case the function + needs to be selected. + """ + # Handle single elements + try: + cases = tuple(cases) + except TypeError: + cases = (cases,) + + # validate prefix + if not isinstance(prefix, str): + raise TypeError("`prefix` should be a string, found: %r" % prefix) + + # validate glob and filter and merge them in a single tuple of callables + filters = () + if glob is not None: + if not isinstance(glob, string_types): + raise TypeError("`glob` should be a string containing a glob-like pattern (not a regex).") + + filters += (create_glob_name_filter(glob),) + if filter is not None: + if not callable(filter): + raise TypeError( + "`filter` should be a callable starting in pytest-cases 0.8.0. If you wish to provide a single" + " tag to match, use `has_tag` instead.") + + filters += (filter,) + + # parent package + caller_module_name = getattr(parametrization_target, '__module__', None) + parent_pkg_name = '.'.join(caller_module_name.split('.')[:-1]) if caller_module_name is not None else None + + # start collecting all cases + cases_funs = [] + for c in cases: + # load case or cases depending on type + if safe_isclass(c): + # class + new_cases = extract_cases_from_class(c, case_fun_prefix=prefix, check_name=False) # do not check name, it was explicitly passed + cases_funs += new_cases + elif callable(c): + # function + if is_case_function(c, check_prefix=False): # do not check prefix, it was explicitly passed + cases_funs.append(c) + else: + raise ValueError("Unsupported case function: %r" % c) + else: + # module + if c is AUTO: + c = import_default_cases_module(parametrization_target) + elif c is AUTO2: + c = import_default_cases_module(parametrization_target, alt_name=True) + elif c is THIS_MODULE or c == '.': + c = caller_module_name + new_cases = extract_cases_from_module(c, package_name=parent_pkg_name, case_fun_prefix=prefix) + cases_funs += new_cases + + # filter last, for easier debugging (collection will be slightly less performant when a large volume of cases exist) + return [c for c in cases_funs + # IMPORTANT: with the trick below we create and attach a case info on each `c` in the same loop + if CaseInfo.get_from(c, create=True, prefix_for_ids=prefix) + # this second member below is the only actual test performing query filtering + and matches_tag_query(c, has_tag=has_tag, filter=filters)] + + +def get_parametrize_args(cases_funs, # type: List[Callable] + ): + # type: (...) -> List[Union[lazy_value, fixture_ref]] + """ + Transforms a list of cases (obtained from `get_all_cases`) into a list of argvalues for `@parametrize`. + Each case function `case_fun` is transformed into one or several `lazy_value`(s) or a `fixture_ref`: + + - If `case_fun` requires at least on fixture, a fixture will be created if not yet present, and a `fixture_ref` + will be returned. + - If `case_fun` is a parametrized case, one `lazy_value` with a partialized version will be created for each + parameter combination. + - Otherwise, `case_fun` represents a single case: in that case a single `lazy_value` is returned. + + :param cases_funs: a list of case functions returned typically by `get_all_cases` + :return: + """ + return [c for _f in cases_funs for c in case_to_argvalues(_f)] + + +def case_to_argvalues(case_fun, # type: Callable + ): + # type: (...) -> Tuple[lazy_value] + """Transform a single case into one or several `lazy_value`(s) or a `fixture_ref` to be used in `@parametrize` + + If `case_fun` requires at least on fixture, a fixture will be created if not yet present, and a `fixture_ref` will + be returned. + + If `case_fun` is a parametrized case, one `lazy_value` with a partialized version will be created for each parameter + combination. + + Otherwise, `case_fun` represents a single case: in that case a single `lazy_value` is returned. + + :param case_fun: + :return: + """ + case_info = CaseInfo.get_from(case_fun) + case_id = case_info.id + case_marks = case_info.marks + + # get the list of all calls that pytest *would* have made for such a (possibly parametrized) function + meta = MiniMetafunc(case_fun) + + if not meta.requires_fixtures: + if not meta.is_parametrized: + # single unparametrized case function + return (lazy_value(case_fun, id=case_id, marks=case_marks),) + else: + # parametrized. create one version of the callable for each parametrized call + return tuple(lazy_value(partial(case_fun, **c.funcargs), id="%s-%s" % (case_id, c.id), marks=c.marks) + for c in meta._calls) + else: + # at least a required fixture: create a fixture + # unwrap any partial that would have been created by us because the fixture was in a class + if isinstance(case_fun, partial): + host_cls = case_fun.host_class + case_fun = case_fun.func + else: + host_cls = None + + host_module = import_module(case_fun.__module__) + + # create a new fixture and place it on the host (note: if already done, no need to recreate it) + existing_fix = getattr(host_cls or host_module, case_id, None) + if existing_fix is None: + # if meta.is_parametrized: + # nothing to do, the parametrization marks are already there + new_fix = fixture(name=case_id)(case_fun) + setattr(host_cls or host_module, case_id, new_fix) + else: + raise NotImplementedError("We should check if this is the same or another and generate a new name in that " + "case") + + # now reference the new or existing fixture + argvalues_tuple = (fixture_ref(case_id),) + return make_marked_parameter_value(argvalues_tuple, marks=case_marks) if case_marks else argvalues_tuple + + +def import_default_cases_module(f, alt_name=False): + """ + Implements the `module=AUTO` behaviour of `@parameterize_cases`: based on the decorated test function `f`, + it finds its containing module name ".py" and then tries to import the python module + "_cases.py". + + Alternately if `alt_name=True` the name pattern to use will be `cases_.py` when the test module is named + `test_.py`. + + :param f: the decorated test function + :param alt_name: a boolean (default False) to use the alternate naming scheme. + :return: + """ + if alt_name: + parts = f.__module__.split('.') + assert parts[-1][0:5] == 'test_' + cases_module_name = "%s.cases_%s" % ('.'.join(parts[:-1]), parts[-1][5:]) + else: + cases_module_name = "%s_cases" % f.__module__ + + try: + cases_module = import_module(cases_module_name) + except ImportError: + raise ValueError("Error importing test cases module to parametrize function %r: unable to import AUTO%s " + "cases module %r. Maybe you wish to import cases from somewhere else ? In that case please " + "specify `cases=...`." + % (f, '2' if alt_name else '', cases_module_name)) + return cases_module + + +def hasinit(obj): + init = getattr(obj, "__init__", None) + if init: + return init != object.__init__ + + +def hasnew(obj): + new = getattr(obj, "__new__", None) + if new: + return new != object.__new__ + + +class CasesCollectionWarning(UserWarning): + """ + Warning emitted when pytest cases is not able to collect a file or symbol in a module. + """ + __module__ = "pytest_cases" + + +def extract_cases_from_class(cls, + check_name=True, + case_fun_prefix=CASE_PREFIX_FUN, + _case_param_factory=None + ): + # type: (...) -> List[Callable] + """ + + :param cls: + :param check_name: + :param case_fun_prefix: + :param _case_param_factory: + :return: + """ + if is_case_class(cls, check_name=check_name): + # see from _pytest.python import pytest_pycollect_makeitem + + if hasinit(cls): + warn( + CasesCollectionWarning( + "cannot collect cases class %r because it has a " + "__init__ constructor" + % (cls.__name__, ) + ) + ) + return [] + elif hasnew(cls): + warn( + CasesCollectionWarning( + "cannot collect test class %r because it has a " + "__new__ constructor" + % (cls.__name__, ) + ) + ) + return [] + + return _extract_cases_from_module_or_class(cls=cls, case_fun_prefix=case_fun_prefix, + _case_param_factory=_case_param_factory) + else: + return [] + + +def extract_cases_from_module(module, # type: ModuleRef + package_name=None, # type: str + case_fun_prefix=CASE_PREFIX_FUN, # type: str + _case_param_factory=None + ): + # type: (...) -> List[Callable] + """ + Internal method used to create a list of `CaseDataGetter` for all cases available from the given module. + See `@cases_data` + + See also `_pytest.python.PyCollector.collect` and `_pytest.python.PyCollector._makeitem` and + `_pytest.python.pytest_pycollect_makeitem`: we could probably do this in a better way in pytest_pycollect_makeitem + + :param module: + :param package_name: + :param _case_param_factory: + :return: + """ + # optionally import module if passed as module name string + if isinstance(module, string_types): + module = import_module(module, package=package_name) + + return _extract_cases_from_module_or_class(module=module, _case_param_factory=_case_param_factory, + case_fun_prefix=case_fun_prefix) + + +def _extract_cases_from_module_or_class(module=None, # type: ModuleRef + cls=None, # type: Type + case_fun_prefix=CASE_PREFIX_FUN, # type: str + _case_param_factory=None + ): + """ + + :param module: + :param _case_param_factory: + :return: + """ + if not ((cls is None) ^ (module is None)): + raise ValueError("Only one of cls or module should be provided") + + container = cls or module + + # We will gather all cases in the reference module and put them in this dict (line no, case) + cases_dct = dict() + + # List members - only keep the functions from the module file (not the imported ones) + if module is not None: + def _of_interest(f): + # check if the function is actually *defined* in this module (not imported from elsewhere) + # Note: we used code.co_filename == module.__file__ in the past + # but on some targets the file changes to a cached one so this does not work reliably, + # see https://github.com/smarie/python-pytest-cases/issues/72 + try: + return f.__module__ == module.__name__ + except: # noqa + return False + else: + def _of_interest(x): # noqa + return True + + for m_name, m in getmembers(container, _of_interest): + if is_case_class(m): + co_firstlineno = get_code_first_line(m) + cls_cases = extract_cases_from_class(m, case_fun_prefix=case_fun_prefix, _case_param_factory=_case_param_factory) + for _i, _m_item in enumerate(cls_cases): + gen_line_nb = co_firstlineno + (_i / len(cls_cases)) + cases_dct[gen_line_nb] = _m_item + + elif is_case_function(m, prefix=case_fun_prefix): + co_firstlineno = get_code_first_line(m) + if cls is not None: + if isinstance(cls.__dict__[m_name], (staticmethod, classmethod)): + # skip it + continue + # partialize the function to get one without the 'self' argument + new_m = partial(m, cls()) + # remember the class + new_m.host_class = cls + # we have to recopy all metadata concerning the case function + new_m.__name__ = m.__name__ + CaseInfo.copy_info(m, new_m) + copy_pytest_marks(m, new_m, override=True) + m = new_m + del new_m + + if _case_param_factory is None: + # Nominal usage: put the case in the dictionary + cases_dct[co_firstlineno] = m + else: + # Legacy usage where the cases generators were expanded here and inserted with a virtual line no + _case_param_factory(m, co_firstlineno, cases_dct) + + # convert into a list, taking all cases in order of appearance in the code (sort by source code line number) + cases = [cases_dct[k] for k in sorted(cases_dct.keys())] + + return cases + + +# Below is the beginning of a switch from our code scanning tool above to the same one than pytest. +# from .common_pytest import is_fixture, safe_isclass, compat_get_real_func, compat_getfslineno +# +# +# class PytestCasesWarning(UserWarning): +# """ +# Bases: :class:`UserWarning`. +# +# Base class for all warnings emitted by pytest cases. +# """ +# +# __module__ = "pytest_cases" +# +# +# class PytestCasesCollectionWarning(PytestCasesWarning): +# """ +# Bases: :class:`PytestCasesWarning`. +# +# Warning emitted when pytest cases is not able to collect a file or symbol in a module. +# """ +# +# __module__ = "pytest_cases" +# +# +# class CasesModule(object): +# """ +# A collector for test cases +# This is a very lightweight version of `_pytest.python.Module`,the pytest collector for test functions and classes. +# +# See also pytest_collect_file and pytest_pycollect_makemodule hooks +# """ +# __slots__ = 'obj' +# +# def __init__(self, module): +# self.obj = module +# +# def collect(self): +# """ +# A copy of pytest Module.collect (PyCollector.collect actually) +# :return: +# """ +# if not getattr(self.obj, "__test__", True): +# return [] +# +# # NB. we avoid random getattrs and peek in the __dict__ instead +# # (XXX originally introduced from a PyPy need, still true?) +# dicts = [getattr(self.obj, "__dict__", {})] +# for basecls in getmro(self.obj.__class__): +# dicts.append(basecls.__dict__) +# seen = {} +# values = [] +# for dic in dicts: +# for name, obj in list(dic.items()): +# if name in seen: +# continue +# seen[name] = True +# res = self._makeitem(name, obj) +# if res is None: +# continue +# if not isinstance(res, list): +# res = [res] +# values.extend(res) +# +# def sort_key(item): +# fspath, lineno, _ = item.reportinfo() +# return (str(fspath), lineno) +# +# values.sort(key=sort_key) +# return values +# +# def _makeitem(self, name, obj): +# """ An adapted copy of _pytest.python.pytest_pycollect_makeitem """ +# if safe_isclass(obj): +# if self.iscaseclass(obj, name): +# raise ValueError("Case classes are not yet supported: %r" % obj) +# elif self.iscasefunction(obj, name): +# # mock seems to store unbound methods (issue473), normalize it +# obj = getattr(obj, "__func__", obj) +# # We need to try and unwrap the function if it's a functools.partial +# # or a functools.wrapped. +# # We mustn't if it's been wrapped with mock.patch (python 2 only) +# if not (isfunction(obj) or isfunction(compat_get_real_func(obj))): +# filename, lineno = compat_getfslineno(obj) +# warn_explicit( +# message=PytestCasesCollectionWarning( +# "cannot collect %r because it is not a function." % name +# ), +# category=None, +# filename=str(filename), +# lineno=lineno + 1, +# ) +# elif getattr(obj, "__test__", True): +# if isgeneratorfunction(obj): +# filename, lineno = compat_getfslineno(obj) +# warn_explicit( +# message=PytestCasesCollectionWarning( +# "cannot collect %r because it is a generator function." % name +# ), +# category=None, +# filename=str(filename), +# lineno=lineno + 1, +# ) +# else: +# res = list(self._gencases(name, obj)) +# outcome.force_result(res) +# +# def iscasefunction(self, obj, name): +# """Similar to PyCollector.istestfunction""" +# if name.startswith("case_"): +# if isinstance(obj, staticmethod): +# # static methods need to be unwrapped +# obj = getattr(obj, "__func__", False) +# return ( +# getattr(obj, "__call__", False) +# and not is_fixture(obj) is None +# ) +# else: +# return False +# +# def iscaseclass(self, obj, name): +# """Similar to PyCollector.istestclass""" +# return name.startswith("Case") +# +# def _gencases(self, name, funcobj): +# # generate the case associated with a case function object. +# # note: the original PyCollector._genfunctions has a "metafunc" mechanism here, we do not need it. +# return [] +# +# diff --git a/pytest_cases/common_others.py b/pytest_cases/common_others.py new file mode 100644 index 00000000..0be4f51c --- /dev/null +++ b/pytest_cases/common_others.py @@ -0,0 +1,206 @@ +from inspect import findsource +import re + +try: + from typing import Union, Callable, Any, Optional, Tuple, Type # noqa +except ImportError: + pass + +from .common_mini_six import string_types + + +def get_code_first_line(f): + """ + Returns the source code associated to function or class f. It is robust to wrappers such as @lru_cache + :param f: + :return: + """ + # todo maybe use inspect.unwrap instead? + if hasattr(f, '__wrapped__'): + return get_code_first_line(f.__wrapped__) + elif hasattr(f, '__code__'): + # a function + return f.__code__.co_firstlineno + else: + # a class ? + try: + _, lineno = findsource(f) + return lineno + except: # noqa + raise ValueError("Cannot get code information for function or class %r" % f) + + +# Below is the beginning of a switch from our code scanning tool above to the same one than pytest. See `case_parametrizer_new` +# from _pytest.compat import get_real_func as compat_get_real_func +# +# try: +# from _pytest._code.source import getfslineno as compat_getfslineno +# except ImportError: +# from _pytest.compat import getfslineno as compat_getfslineno + +try: + ExpectedError = Optional[Union[Type[Exception], str, Exception, Callable[[Exception], Optional[bool]]]] + """The expected error in case failure is expected. An exception type, instance, or a validation function""" +except: # noqa + pass + + +def unfold_expected_err(expected_e # type: ExpectedError + ): + # type: (...) -> Tuple[Optional[Type[BaseException]], Optional[re.Pattern], Optional[BaseException], Optional[Callable[[BaseException], Optional[bool]]]] + """ + 'Unfolds' the expected error `expected_e` to return a tuple of + - expected error type + - expected error representation pattern (a regex Pattern) + - expected error instance + - error validation callable + + If `expected_e` is an exception type, returns `expected_e, None, None, None` + + If `expected_e` is a string, returns `BaseException, re.compile(expected_e), None, None` + + If `expected_e` is an exception instance, returns `type(expected_e), None, expected_e, None` + + If `expected_e` is an exception validation function, returns `BaseException, None, None, expected_e` + + :param expected_e: an `ExpectedError`, that is, either an exception type, a regex string, an exception + instance, or an exception validation function + :return: + """ + if type(expected_e) is type and issubclass(expected_e, BaseException): + return expected_e, None, None, None + + elif isinstance(expected_e, string_types): + return BaseException, re.compile(expected_e), None, None # noqa + + elif issubclass(type(expected_e), Exception): + return type(expected_e), None, expected_e, None + + elif callable(expected_e): + return BaseException, None, None, expected_e + + raise ValueError("ExpectedNormal error should either be an exception type, an exception instance, or an exception " + "validation callable") + + +def assert_exception(expected # type: ExpectedError + ): + """ + A context manager to check that some bit of code raises an exception. Sometimes it might be more + handy than `with pytest.raises():`. + + `expected` can be: + + - an expected error type, in which case `isinstance(caught, expected)` will be used for validity checking + + - an expected error representation pattern (a regex pattern string), in which case + `expected.match(repr(caught))` will be used for validity checking + + - an expected error instance, in which case BOTH `isinstance(caught, type(expected))` AND + `caught == expected` will be used for validity checking + + - an error validation callable, in which case `expected(caught) is not False` will be used for validity + checking + + Upon failure, this raises an `ExceptionCheckingError` (a subclass of `AssertionError`) + + ```python + # good type - ok + with assert_exception(ValueError): + raise ValueError() + + # good type - inherited - ok + class MyErr(ValueError): + pass + with assert_exception(ValueError): + raise MyErr() + + # no exception - raises ExceptionCheckingError + with assert_exception(ValueError): + pass + + # wrong type - raises ExceptionCheckingError + with assert_exception(ValueError): + raise TypeError() + + # good repr pattern - ok + with assert_exception(r"ValueError\('hello'[,]+\)"): + raise ValueError("hello") + + # good instance equality check - ok + class MyExc(Exception): + def __eq__(self, other): + return vars(self) == vars(other) + with assert_exception(MyExc('hello')): + raise MyExc("hello") + + # good equality but wrong type - raises ExceptionCheckingError + with assert_exception(MyExc('hello')): + raise Exception("hello") + ``` + + :param expected: an exception type, instance, repr string pattern, or a callable + """ + return AssertException(expected) + + +class ExceptionCheckingError(AssertionError): + pass + + +class AssertException(object): + """ An implementation of the `assert_exception` context manager""" + + __slots__ = ('expected_exception', 'err_type', 'err_ptrn', 'err_inst', 'err_checker') + + def __init__(self, expected_exception): + # First see what we need to assert + err_type, err_ptrn, err_inst, err_checker = unfold_expected_err(expected_exception) + self.expected_exception = expected_exception + self.err_type = err_type + self.err_ptrn = err_ptrn + self.err_inst = err_inst + self.err_checker = err_checker + + def __enter__(self): + pass + + def __exit__(self, exc_type, exc_val, exc_tb): + if exc_type is None: + # bad: no exception caught + raise AssertionError("DID NOT RAISE any BaseException") + + # Type check + if not isinstance(exc_val, self.err_type): + raise ExceptionCheckingError("Caught exception %r is not an instance of expected type %r" + % (exc_val, self.err_type)) + + # Optional - pattern matching + if self.err_ptrn is not None: + if not self.err_ptrn.match(repr(exc_val)): + raise ExceptionCheckingError("Caught exception %r does not match expected pattern %r" + % (exc_val, self.err_ptrn)) + + # Optional - Additional Exception instance check with equality + if self.err_inst is not None: + # note: do not use != because in python 2 that is not equivalent + if not (exc_val == self.err_inst): + raise ExceptionCheckingError("Caught exception %r does not equal expected instance %r" + % (exc_val, self.err_inst)) + + # Optional - Additional Exception instance check with custom checker + if self.err_checker is not None: + if self.err_checker(exc_val) is False: + raise ExceptionCheckingError("Caught exception %r is not valid according to %r" + % (exc_val, self.err_checker)) + + # Suppress the exception since it is valid. + # See https://docs.python.org/2/reference/datamodel.html#object.__exit__ + return True + + +AUTO = object() +"""Marker for automatic defaults""" + +AUTO2 = object() +"""Marker that alternate automatic defaults""" diff --git a/pytest_cases/common_pytest.py b/pytest_cases/common_pytest.py index a5164f4f..c6fcef42 100644 --- a/pytest_cases/common_pytest.py +++ b/pytest_cases/common_pytest.py @@ -1,23 +1,25 @@ from __future__ import division -import warnings - try: # python 3.3+ - from inspect import signature + from inspect import signature, Parameter except ImportError: - from funcsigs import signature # noqa + from funcsigs import signature, Parameter # noqa from distutils.version import LooseVersion -from inspect import isgeneratorfunction -from warnings import warn +from inspect import isgeneratorfunction, isclass try: - from typing import Union, Callable, Any, Optional # noqa + from typing import Union, Callable, Any, Optional, Tuple, Type # noqa except ImportError: pass import pytest +from _pytest.python import Metafunc + from .common_mini_six import string_types +from .common_pytest_marks import make_marked_parameter_value, get_param_argnames_as_list, has_pytest_param, \ + get_pytest_parametrize_marks +from .common_pytest_lazy_values import is_lazy_value # A decorator that will work to create a fixture containing 'yield', whatever the pytest version, and supports hooks @@ -74,6 +76,16 @@ def is_fixture(fixture_fun # type: Any return False +def safe_isclass(obj # type: object + ): + # type: (...) -> bool + """Ignore any exception via isinstance on Python 3.""" + try: + return isclass(obj) + except Exception: # noqa + return False + + def assert_is_fixture(fixture_fun # type: Any ): """ @@ -137,54 +149,6 @@ def get_fixture_scope(fixture_fun): # return fixture_fun.func_scope -def get_param_argnames_as_list(argnames): - """ - pytest parametrize accepts both coma-separated names and list/tuples. - This function makes sure that we always return a list - :param argnames: - :return: - """ - if isinstance(argnames, string_types): - argnames = argnames.replace(' ', '').split(',') - return list(argnames) - - -# ------------ container for the mark information that we grab from the fixtures (`@fixture_plus`) -class _ParametrizationMark: - """ - Represents the information required by `@fixture_plus` to work. - """ - __slots__ = "param_names", "param_values", "param_ids" - - def __init__(self, mark): - bound = get_parametrize_signature().bind(*mark.args, **mark.kwargs) - try: - remaining_kwargs = bound.arguments['kwargs'] - except KeyError: - pass - else: - if len(remaining_kwargs) > 0: - warn("parametrize kwargs not taken into account: %s. Please report it at" - " https://github.com/smarie/python-pytest-cases/issues" % remaining_kwargs) - self.param_names = get_param_argnames_as_list(bound.arguments['argnames']) - self.param_values = bound.arguments['argvalues'] - try: - bound.apply_defaults() - self.param_ids = bound.arguments['ids'] - except AttributeError: - # can happen if signature is from funcsigs so we have to apply ourselves - self.param_ids = bound.arguments.get('ids', None) - - -# -------- tools to get the parametrization mark whatever the pytest version -class _LegacyMark: - __slots__ = "args", "kwargs" - - def __init__(self, *args, **kwargs): - self.args = args - self.kwargs = kwargs - - # ---------------- working on pytest nodes (e.g. Function) def is_function_node(node): @@ -222,79 +186,6 @@ def get_param_names(fnode): return param_names -# ---------------- working on functions -def get_pytest_marks_on_function(f, as_decorators=False): - """ - Utility to return *ALL* pytest marks (not only parametrization) applied on a function - - :param f: - :param as_decorators: transforms the marks into decorators before returning them - :return: - """ - try: - mks = f.pytestmark - except AttributeError: - try: - # old pytest < 3: marks are set as fields on the function object - # but they do not have a particulat type, their type is 'instance'... - mks = [v for v in vars(f).values() if str(v).startswith(" 3.2.0 - marks = getattr(f, 'pytestmark', None) - if marks is not None: - return tuple(_ParametrizationMark(m) for m in marks if m.name == 'parametrize') - else: - # older versions - mark_info = getattr(f, 'parametrize', None) - if mark_info is not None: - # mark_info.args contains a list of (name, values) - if len(mark_info.args) % 2 != 0: - raise ValueError("internal pytest compatibility error - please report") - nb_parametrize_decorations = len(mark_info.args) // 2 - if nb_parametrize_decorations > 1 and len(mark_info.kwargs) > 0: - raise ValueError("Unfortunately with this old pytest version it is not possible to have several " - "parametrization decorators while specifying **kwargs, as all **kwargs are " - "merged, leading to inconsistent results. Either upgrade pytest, remove the **kwargs," - "or merge all the @parametrize decorators into a single one. **kwargs: %s" - % mark_info.kwargs) - res = [] - for i in range(nb_parametrize_decorations): - param_name, param_values = mark_info.args[2*i:2*(i+1)] - res.append(_ParametrizationMark(_LegacyMark(param_name, param_values, **mark_info.kwargs))) - return tuple(res) - else: - return () - - -# noinspection PyUnusedLocal -def _pytest_mark_parametrize(argnames, argvalues, ids=None, indirect=False, scope=None, **kwargs): - """ Fake method to have a reference signature of pytest.mark.parametrize""" - pass - - -def get_parametrize_signature(): - """ - - :return: a reference signature representing - """ - return signature(_pytest_mark_parametrize) - - # ---------- test ids utils --------- def combine_ids(paramid_tuples): """ @@ -430,33 +321,34 @@ def extract_parameterset_info(argnames, argvalues, check_nb=True): raise TypeError("argnames must be an iterable. Found %r" % argnames) nbnames = len(argnames) for v in argvalues: - # is this a pytest.param() ? - if is_marked_parameter_value(v): - # --id - _id = get_marked_parameter_id(v) - pids.append(_id) - # --marks - marks = get_marked_parameter_marks(v) - pmarks.append(marks) # note: there might be several - # --value(a tuple if this is a tuple parameter) - v = get_marked_parameter_values(v) - if nbnames == 1: - pvalues.append(v[0]) - else: - pvalues.append(v) - else: - # normal argvalue - pids.append(None) - pmarks.append(None) - pvalues.append(v) + _pid, _pmark, _pvalue = extract_pset_info_single(nbnames, v) - if check_nb and nbnames > 1 and (len(v) != nbnames): + pids.append(_pid) + pmarks.append(_pmark) + pvalues.append(_pvalue) + + if check_nb and nbnames > 1 and (len(_pvalue) != nbnames): raise ValueError("Inconsistent number of values in pytest parametrize: %s items found while the " - "number of parameters is %s: %s." % (len(v), nbnames, v)) + "number of parameters is %s: %s." % (len(_pvalue), nbnames, _pvalue)) return pids, pmarks, pvalues +def extract_pset_info_single(nbnames, argvalue): + """Return id, marks, value""" + if is_marked_parameter_value(argvalue): + # --id + _id = get_marked_parameter_id(argvalue) + # --marks + marks = get_marked_parameter_marks(argvalue) + # --value(a tuple if this is a tuple parameter) + argvalue = get_marked_parameter_values(argvalue) + return _id, marks, argvalue[0] if nbnames == 1 else argvalue + else: + # normal argvalue + return None, None, argvalue + + try: # pytest 3.x+ from _pytest.mark import ParameterSet # noqa @@ -475,7 +367,10 @@ def get_marked_parameter_id(v): except ImportError: # pytest 2.x from _pytest.mark import MarkDecorator - def ParameterSet(values, id, marks): + # noinspection PyPep8Naming + def ParameterSet(values, + id, # noqa + marks): """ Dummy function (not a class) used only by parametrize_plus """ if id is not None: raise ValueError("This should not happen as `pytest.param` does not exist in pytest 2") @@ -499,83 +394,15 @@ def get_marked_parameter_marks(v): return [v] def get_marked_parameter_values(v): - return v.args[1:] + if v.name in ('skip', 'skipif'): + return v.args[-1] # see MetaFunc.parametrize in pytest 2 to be convinced :) + else: + raise ValueError("Unsupported mark") def get_marked_parameter_id(v): return v.kwargs.get('id', None) -# ---- tools to reapply marks on test parameter values, whatever the pytest version ---- - -# Compatibility for the way we put marks on single parameters in the list passed to @pytest.mark.parametrize -# see https://docs.pytest.org/en/3.3.0/skipping.html?highlight=mark%20parametrize#skip-xfail-with-parametrize - -# check if pytest.param exists -has_pytest_param = hasattr(pytest, 'param') - - -if not has_pytest_param: - # if not this is how it was done - # see e.g. https://docs.pytest.org/en/2.9.2/skipping.html?highlight=mark%20parameter#skip-xfail-with-parametrize - def make_marked_parameter_value(c, marks): - if len(marks) > 1: - raise ValueError("Multiple marks on parameters not supported for old versions of pytest") - else: - # get a decorator for each of the markinfo - marks_mod = transform_marks_into_decorators(marks) - - # decorate - return marks_mod[0](c) -else: - # Otherwise pytest.param exists, it is easier - def make_marked_parameter_value(c, marks): - # get a decorator for each of the markinfo - marks_mod = transform_marks_into_decorators(marks) - - # decorate - return pytest.param(c, marks=marks_mod) - - -def transform_marks_into_decorators(marks): - """ - Transforms the provided marks (MarkInfo) obtained from marked cases, into MarkDecorator so that they can - be re-applied to generated pytest parameters in the global @pytest.mark.parametrize. - - :param marks: - :return: - """ - marks_mod = [] - try: - # suppress the warning message that pytest generates when calling pytest.mark.MarkDecorator() directly - with warnings.catch_warnings(): - warnings.simplefilter("ignore") - for m in marks: - md = pytest.mark.MarkDecorator() - - if LooseVersion(pytest.__version__) >= LooseVersion('3.0.0'): - if isinstance(m, type(md)): - # already a decorator, we can use it - marks_mod.append(m) - else: - md.mark = m - marks_mod.append(md) - else: - # always recreate one, type comparison does not work (all generic stuff) - md.name = m.name - # md.markname = m.name - md.args = m.args - md.kwargs = m.kwargs - - # markinfodecorator = getattr(pytest.mark, markinfo.name) - # markinfodecorator(*markinfo.args) - - marks_mod.append(md) - - except Exception as e: - warn("Caught exception while trying to mark case: [%s] %s" % (type(e), e)) - return marks_mod - - def get_pytest_nodeid(metafunc): try: return metafunc.definition.nodeid @@ -587,7 +414,7 @@ def get_pytest_nodeid(metafunc): from _pytest.fixtures import scopes as pt_scopes except ImportError: # pytest 2 - from _pytest.python import scopes as pt_scopes # noqa + from _pytest.python import scopes as pt_scopes, Metafunc # noqa def get_pytest_scopenum(scope_str): @@ -636,3 +463,179 @@ def mini_idvalset(argnames, argvalues, idx): for val, argname in zip(argvalues, argnames) ] return "-".join(this_id) + + +try: + from _pytest.compat import getfuncargnames # noqa +except ImportError: + import sys + + def num_mock_patch_args(function): + """ return number of arguments used up by mock arguments (if any) """ + patchings = getattr(function, "patchings", None) + if not patchings: + return 0 + + mock_sentinel = getattr(sys.modules.get("mock"), "DEFAULT", object()) + ut_mock_sentinel = getattr(sys.modules.get("unittest.mock"), "DEFAULT", object()) + + return len( + [p for p in patchings if not p.attribute_name and (p.new is mock_sentinel or p.new is ut_mock_sentinel)] + ) + + # noinspection SpellCheckingInspection + def getfuncargnames(function, cls=None): + """Returns the names of a function's mandatory arguments.""" + parameters = signature(function).parameters + + arg_names = tuple( + p.name + for p in parameters.values() + if ( + p.kind is Parameter.POSITIONAL_OR_KEYWORD + or p.kind is Parameter.KEYWORD_ONLY + ) + and p.default is Parameter.empty + ) + + # If this function should be treated as a bound method even though + # it's passed as an unbound method or function, remove the first + # parameter name. + if cls and not isinstance(cls.__dict__.get(function.__name__, None), staticmethod): + arg_names = arg_names[1:] + # Remove any names that will be replaced with mocks. + if hasattr(function, "__wrapped__"): + arg_names = arg_names[num_mock_patch_args(function):] + return arg_names + + +class MiniFuncDef(object): + __slots__ = ('nodeid',) + + def __init__(self, nodeid): + self.nodeid = nodeid + + +class MiniMetafunc(Metafunc): + # noinspection PyMissingConstructor + def __init__(self, func): + self.config = None + self.function = func + self.definition = MiniFuncDef(func.__name__) + self._calls = [] + # non-default parameters + self.fixturenames = getfuncargnames(func) + # get parametrization marks + self.pmarks = get_pytest_parametrize_marks(self.function) + if self.is_parametrized: + self.update_callspecs() + self.required_fixtures = set(self.fixturenames) - set(self._calls[0].funcargs) + else: + self.required_fixtures = self.fixturenames + + @property + def is_parametrized(self): + return len(self.pmarks) > 0 + + @property + def requires_fixtures(self): + return len(self.required_fixtures) > 0 + + def update_callspecs(self): + """ + + :return: + """ + for pmark in self.pmarks: + if len(pmark.param_names) == 1: + argvals = tuple(v if is_marked_parameter_value(v) else (v,) for v in pmark.param_values) + else: + argvals = pmark.param_values + self.parametrize(argnames=pmark.param_names, argvalues=argvals, ids=pmark.param_ids, + # use indirect = False and scope = 'function' to avoid having to implement complex patches + indirect=False, scope='function') + + if not has_pytest_param: + # fix the CallSpec2 instances so that the marks appear + # noinspection PyProtectedMember + for c in self._calls: + c.marks = list(c.keywords.values()) + + +def get_callspecs(func): + """ + Returns a list of pytest CallSpec objects corresponding to calls that should be made for this parametrized function. + This mini-helper assumes no complex things (scope='function', indirect=False, no fixtures, no custom configuration) + + :param func: + :return: + """ + meta = MiniMetafunc(func) + # meta.update_callspecs() + # noinspection PyProtectedMember + return meta._calls + + +def cart_product_pytest(argnames, argvalues): + """ + - do NOT use `itertools.product` as it fails to handle MarkDecorators + - we also unpack tuples associated with several argnames ("a,b") if needed + - we also propagate marks + + :param argnames: + :param argvalues: + :return: + """ + # transform argnames into a list of lists + argnames_lists = [get_param_argnames_as_list(_argnames) if len(_argnames) > 0 else [] for _argnames in argnames] + + # make the cartesian product per se + argvalues_prod = _cart_product_pytest(argnames_lists, argvalues) + + # flatten the list of argnames + argnames_list = [n for nlist in argnames_lists for n in nlist] + + # apply all marks to the arvalues + argvalues_prod = [make_marked_parameter_value(tuple(argvalues), marks=marks) if len(marks) > 0 else tuple(argvalues) + for marks, argvalues in argvalues_prod] + + return argnames_list, argvalues_prod + + +def _cart_product_pytest(argnames_lists, argvalues): + result = [] + + # first perform the sub cartesian product with entries [1:] + sub_product = _cart_product_pytest(argnames_lists[1:], argvalues[1:]) if len(argvalues) > 1 else None + + # then do the final product with entry [0] + for x in argvalues[0]: + # handle x + nb_names = len(argnames_lists[0]) + + # (1) extract meta-info + x_id, x_marks, x_value = extract_pset_info_single(nb_names, x) + x_marks_lst = list(x_marks) if x_marks is not None else [] + if x_id is not None: + raise ValueError("It is not possible to specify a sub-param id when using the new parametrization style. " + "Either use the traditional style or customize all ids at once in `idgen`") + + # (2) possibly unpack + if nb_names > 1: + # if lazy value, we have to do something + if is_lazy_value(x_value): + x_value_lst = x_value.as_lazy_items_list(nb_names) + else: + x_value_lst = list(x_value) + else: + x_value_lst = [x_value] + + # product + if len(argvalues) > 1: + for m, p in sub_product: + # combine marks and values + result.append((x_marks_lst + m, x_value_lst + p)) + else: + result.append((x_marks_lst, x_value_lst)) + + return result diff --git a/pytest_cases/common_pytest_lazy_values.py b/pytest_cases/common_pytest_lazy_values.py new file mode 100644 index 00000000..7bcc16f5 --- /dev/null +++ b/pytest_cases/common_pytest_lazy_values.py @@ -0,0 +1,320 @@ +from distutils.version import LooseVersion +from functools import partial + +try: # python 3.3+ + from inspect import signature +except ImportError: + from funcsigs import signature # noqa + +try: + from typing import Union, Callable, List, Any, Sequence, Optional # noqa +except ImportError: + pass + +import pytest + +from .common_pytest_marks import get_pytest_marks_on_function, transform_marks_into_decorators + + +pytest53 = LooseVersion(pytest.__version__) >= LooseVersion("5.3.0") +if pytest53: + # in the latest versions of pytest, the default _idmaker returns the value of __name__ if it is available, + # even if an object is not a class nor a function. So we do not need to use any special trick. + _LazyValueBase = object +else: + fake_base = int + + class _LazyValueBase(int, object): + """ + in this older version of pytest, the default _idmaker does *not* return the value of __name__ for + objects that are not functions not classes. However it *does* return str(obj) for objects that are + instances of bool, int or float. So that's why lazy_value inherits from int. + """ + __slots__ = () + + def __new__(cls, *args, **kwargs): + """ Inheriting from int is a bit hard in python: we have to override __new__ """ + obj = fake_base.__new__(cls, 111111) # noqa + cls.__init__(obj, *args, **kwargs) # noqa + return obj + + def __getattribute__(self, item): + """Map all default attribute and method access to the ones in object, not in int""" + return object.__getattribute__(self, item) + + def __repr__(self): + """Magic methods are not intercepted by __getattribute__ and need to be overridden manually. + We do not need all of them by at least override this one for easier debugging""" + return object.__repr__(self) + + +class Lazy(object): + """ + All lazy items should inherit from this for good pytest compliance (ids, marks, etc.) + """ + # @abstractmethod + def get_id(self): + """Return the id to use by pytest""" + raise NotImplementedError() + + # @abstractmethod + def get(self): + """Return the value to use by pytest""" + raise NotImplementedError() + + if not pytest53: + def __str__(self): + """in pytest<5.3 we inherit from int so that str(v) is called by pytest _idmaker to get the id""" + return self.get_id() + + @property + def __name__(self): + """for pytest >= 5.3 we override this so that pytest uses it for id""" + return self.get_id() + + +def _unwrap(obj): + """A light copy of _pytest.compat.get_real_func. In our case + we do not wish to unwrap the partial nor handle pytest fixture + Note: maybe from inspect import unwrap could do the same? + """ + start_obj = obj + for i in range(100): + # __pytest_wrapped__ is set by @pytest.fixture when wrapping the fixture function + # to trigger a warning if it gets called directly instead of by pytest: we don't + # want to unwrap further than this otherwise we lose useful wrappings like @mock.patch (#3774) + # new_obj = getattr(obj, "__pytest_wrapped__", None) + # if isinstance(new_obj, _PytestWrapper): + # obj = new_obj.obj + # break + new_obj = getattr(obj, "__wrapped__", None) + if new_obj is None: + break + obj = new_obj + else: + raise ValueError("could not find real function of {start}\nstopped at {current}".format( + start=repr(start_obj), current=repr(obj) + ) + ) + return obj + + +def partial_to_str(partialfun): + """Return a string representation of a partial function, to use in lazy_value ids""" + strwds = ", ".join("%s=%s" % (k, v) for k, v in partialfun.keywords.items()) + if len(partialfun.args) > 0: + strargs = ', '.join(str(i) for i in partialfun.args) + if len(partialfun.keywords) > 0: + strargs = "%s, %s" % (strargs, strwds) + else: + strargs = strwds + return "%s(%s)" % (partialfun.func.__name__, strargs) + + +# noinspection PyPep8Naming +class LazyValue(Lazy, _LazyValueBase): + """ + A reference to a value getter, to be used in `parametrize_plus`. + + A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a + fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. + """ + if pytest53: + __slots__ = 'valuegetter', '_id', '_marks' + else: + # we can not define __slots__ since we extend int, + # see https://docs.python.org/3/reference/datamodel.html?highlight=__slots__#notes-on-using-slots + pass + + # noinspection PyMissingConstructor + def __init__(self, + valuegetter, # type: Callable[[], Any] + id=None, # type: str # noqa + marks=() # type: Union[Any, Sequence[Any]] + ): + self.valuegetter = valuegetter + self._id = id + if isinstance(marks, (tuple, list, set)): + self._marks = marks + else: + self._marks = (marks, ) + + def get_marks(self, as_decorators=False): + """ + Overrides default implementation to return the marks that are on the case function + + :param as_decorators: when True, the marks will be transformed into MarkDecorators before being + returned + :return: + """ + valuegetter_marks = get_pytest_marks_on_function(self.valuegetter, as_decorators=as_decorators) + + if self._marks: + return transform_marks_into_decorators(self._marks) + valuegetter_marks + else: + return valuegetter_marks + + def get_id(self): + """The id to use in pytest""" + if self._id is not None: + return self._id + else: + # default is the __name__ of the value getter + _id = getattr(self.valuegetter, '__name__', None) + if _id is not None: + return _id + + # unwrap and handle partial functions + vg = _unwrap(self.valuegetter) + + if isinstance(vg, partial): + return partial_to_str(vg) + else: + return vg.__name__ + + def get(self): + return self.valuegetter() + + def as_lazy_tuple(self, nb_params): + return LazyTuple(self, nb_params) + + def as_lazy_items_list(self, nb_params): + return [v for v in LazyTuple(self, nb_params)] + + +class LazyTupleItem(Lazy, _LazyValueBase): + """ + An item in a Lazy Tuple + """ + if pytest53: + __slots__ = 'host', 'item' + else: + # we can not define __slots__ since we extend int, + # see https://docs.python.org/3/reference/datamodel.html?highlight=__slots__#notes-on-using-slots + pass + + # noinspection PyMissingConstructor + def __init__(self, + host, # type: LazyTuple + item # type: int + ): + self.host = host + self.item = item + + def get_id(self): + return "%s[%s]" % (self.host.get_id(), self.item) + + def get(self): + return self.host.force_getitem(self.item) + + +class LazyTuple(Lazy): + """ + A wrapper representing a lazy_value used as a tuple = for several argvalues at once. + + - + while not calling the lazy value + - + """ + __slots__ = ('value', 'theoretical_size', 'retrieved') + + # noinspection PyMissingConstructor + def __init__(self, + valueref, # type: Union[LazyValue, Sequence] + theoretical_size # type: int + ): + self.value = valueref + self.theoretical_size = theoretical_size + self.retrieved = False + + def __len__(self): + return self.theoretical_size + + def get_id(self): + """return the id to use by pytest""" + return self.value.get_id() + + def get(self): + """ Call the underlying value getter, then return the tuple (not self) """ + if not self.retrieved: + # retrieve + self.value = self.value.get() + self.retrieved = True + return self.value + + def __getitem__(self, item): + """ + Getting an item in the tuple with self[i] does *not* retrieve the value automatically, but returns + a facade (a LazyTupleItem), so that pytest can store this item independently wherever needed, without + yet calling the value getter. + """ + if self.retrieved: + # this is never called by pytest, but keep it for debugging + return self.value[item] + elif item >= self.theoretical_size: + raise IndexError(item) + else: + # do not retrieve yet: return a facade + return LazyTupleItem(self, item) + + def force_getitem(self, item): + """ Call the underlying value getter, then return self[i]. """ + getter = self.value + argvalue = self.get() + try: + return argvalue[item] + except TypeError as e: + raise ValueError("(lazy_value) The parameter value returned by `%r` is not compliant with the number" + " of argnames in parametrization (%s). A %s-tuple-like was expected. " + "Returned lazy argvalue is %r and argvalue[%s] raised %s: %s" + % (getter.valuegetter, self.theoretical_size, self.theoretical_size, + argvalue, item, e.__class__, e)) + + +def lazy_value(valuegetter, # type: Callable[[], Any] + id=None, # type: str # noqa + marks=() # type: Union[Any, Sequence[Any]] + ): + """ + Creates a reference to a value getter, to be used in `parametrize_plus`. + + A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a + fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. + + Note that a `lazy_value` can be included in a `pytest.param` without problem. In that case the id defined by + `pytest.param` will take precedence over the one defined in `lazy_value` if any. The marks, however, + will all be kept wherever they are defined. + + :param valuegetter: a callable without mandatory arguments + :param id: an optional id. Otherwise `valuegetter.__name__` will be used by default + :param marks: optional marks. `valuegetter` marks will also be preserved. + """ + return LazyValue(valuegetter, id=id, marks=marks) + + +def is_lazy_value(argval): + try: + return isinstance(argval, LazyValue) + except: + return False + + +def is_lazy(argval): + try: + return isinstance(argval, (LazyValue, LazyTuple, LazyTupleItem)) + except: + return False + + +def get_lazy_args(argval): + """ Possibly calls the lazy values contained in argval if needed, before returning it""" + + try: + _is_lazy = is_lazy(argval) + except: # noqa + return argval + else: + if _is_lazy: + return argval.get() + else: + return argval diff --git a/pytest_cases/common_pytest_marks.py b/pytest_cases/common_pytest_marks.py new file mode 100644 index 00000000..4f814f5e --- /dev/null +++ b/pytest_cases/common_pytest_marks.py @@ -0,0 +1,222 @@ +import warnings +from distutils.version import LooseVersion + +try: # python 3.3+ + from inspect import signature +except ImportError: + from funcsigs import signature # noqa + + +import pytest + +from .common_mini_six import string_types + + +def get_param_argnames_as_list(argnames): + """ + pytest parametrize accepts both coma-separated names and list/tuples. + This function makes sure that we always return a list + :param argnames: + :return: + """ + if isinstance(argnames, string_types): + argnames = argnames.replace(' ', '').split(',') + return list(argnames) + + +# noinspection PyUnusedLocal +def _pytest_mark_parametrize(argnames, argvalues, ids=None, indirect=False, scope=None, **kwargs): + """ Fake method to have a reference signature of pytest.mark.parametrize""" + pass + + +def get_parametrize_signature(): + """ + + :return: a reference signature representing + """ + return signature(_pytest_mark_parametrize) + + +class _ParametrizationMark: + """ + Container for the mark information that we grab from the fixtures (`@fixture_plus`) + + Represents the information required by `@fixture_plus` to work. + """ + __slots__ = "param_names", "param_values", "param_ids" + + def __init__(self, mark): + bound = get_parametrize_signature().bind(*mark.args, **mark.kwargs) + try: + remaining_kwargs = bound.arguments['kwargs'] + except KeyError: + pass + else: + if len(remaining_kwargs) > 0: + warnings.warn("parametrize kwargs not taken into account: %s. Please report it at" + " https://github.com/smarie/python-pytest-cases/issues" % remaining_kwargs) + self.param_names = get_param_argnames_as_list(bound.arguments['argnames']) + self.param_values = bound.arguments['argvalues'] + try: + bound.apply_defaults() + self.param_ids = bound.arguments['ids'] + except AttributeError: + # can happen if signature is from funcsigs so we have to apply ourselves + self.param_ids = bound.arguments.get('ids', None) + + +# -------- tools to get the parametrization mark whatever the pytest version +class _LegacyMark: + __slots__ = "args", "kwargs" + + def __init__(self, *args, **kwargs): + self.args = args + self.kwargs = kwargs + + +# ---------------- working on functions +def copy_pytest_marks(from_f, to_f, override=False): + """Copy all pytest marks from a function to another""" + from_marks = get_pytest_marks_on_function(from_f) + to_marks = [] if override else get_pytest_marks_on_function(to_f) + to_f.pytestmark = to_marks + from_marks + + +def get_pytest_marks_on_function(f, as_decorators=False): + """ + Utility to return *ALL* pytest marks (not only parametrization) applied on a function + + :param f: + :param as_decorators: transforms the marks into decorators before returning them + :return: + """ + try: + mks = f.pytestmark + except AttributeError: + try: + # old pytest < 3: marks are set as fields on the function object + # but they do not have a particulat type, their type is 'instance'... + mks = [v for v in vars(f).values() if str(v).startswith(" 3.2.0 + marks = getattr(f, 'pytestmark', None) + if marks is not None: + return tuple(_ParametrizationMark(m) for m in marks if m.name == 'parametrize') + else: + # older versions + mark_info = getattr(f, 'parametrize', None) + if mark_info is not None: + # mark_info.args contains a list of (name, values) + if len(mark_info.args) % 2 != 0: + raise ValueError("internal pytest compatibility error - please report") + nb_parametrize_decorations = len(mark_info.args) // 2 + if nb_parametrize_decorations > 1 and len(mark_info.kwargs) > 0: + raise ValueError("Unfortunately with this old pytest version it is not possible to have several " + "parametrization decorators while specifying **kwargs, as all **kwargs are " + "merged, leading to inconsistent results. Either upgrade pytest, remove the **kwargs," + "or merge all the @parametrize decorators into a single one. **kwargs: %s" + % mark_info.kwargs) + res = [] + for i in range(nb_parametrize_decorations): + param_name, param_values = mark_info.args[2*i:2*(i+1)] + res.append(_ParametrizationMark(_LegacyMark(param_name, param_values, **mark_info.kwargs))) + return tuple(res) + else: + return () + + +# ---- tools to reapply marks on test parameter values, whatever the pytest version ---- + +# Compatibility for the way we put marks on single parameters in the list passed to @pytest.mark.parametrize +# see https://docs.pytest.org/en/3.3.0/skipping.html?highlight=mark%20parametrize#skip-xfail-with-parametrize + +# check if pytest.param exists +has_pytest_param = hasattr(pytest, 'param') + + +if not has_pytest_param: + # if not this is how it was done + # see e.g. https://docs.pytest.org/en/2.9.2/skipping.html?highlight=mark%20parameter#skip-xfail-with-parametrize + def make_marked_parameter_value(argvalues_tuple, marks): + if len(marks) > 1: + raise ValueError("Multiple marks on parameters not supported for old versions of pytest") + else: + if not isinstance(argvalues_tuple, tuple): + raise TypeError("argvalues must be a tuple !") + + # get a decorator for each of the markinfo + marks_mod = transform_marks_into_decorators(marks, function_marks=False) + + # decorate. Warning: the argvalue MUST be in a tuple + return marks_mod[0](argvalues_tuple) +else: + # Otherwise pytest.param exists, it is easier + def make_marked_parameter_value(argvalues_tuple, marks): + if not isinstance(argvalues_tuple, tuple): + raise TypeError("argvalues must be a tuple !") + + # get a decorator for each of the markinfo + marks_mod = transform_marks_into_decorators(marks, function_marks=False) + + # decorate + return pytest.param(*argvalues_tuple, marks=marks_mod) + + +def transform_marks_into_decorators(marks, function_marks=False): + """ + Transforms the provided marks (MarkInfo) obtained from marked cases, into MarkDecorator so that they can + be re-applied to generated pytest parameters in the global @pytest.mark.parametrize. + + :param marks: + :return: + """ + marks_mod = [] + try: + # suppress the warning message that pytest generates when calling pytest.mark.MarkDecorator() directly + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + for m in marks: + md = pytest.mark.MarkDecorator() + + if LooseVersion(pytest.__version__) >= LooseVersion('3.0.0'): + if isinstance(m, type(md)): + # already a decorator, we can use it + marks_mod.append(m) + else: + md.mark = m + marks_mod.append(md) + else: + # always recreate one, type comparison does not work (all generic stuff) + md.name = m.name + # md.markname = m.name + if function_marks: + md.args = m.args # a mark on a function does not include the function in the args + else: + md.args = m.args[:-1] # not a function: the value is in the args, remove it + md.kwargs = m.kwargs + + # markinfodecorator = getattr(pytest.mark, markinfo.name) + # markinfodecorator(*markinfo.args) + + marks_mod.append(md) + + except Exception as e: + warnings.warn("Caught exception while trying to mark case: [%s] %s" % (type(e), e)) + return marks_mod diff --git a/pytest_cases/fixture_core2.py b/pytest_cases/fixture_core2.py index ace6fc37..a9f30b9c 100644 --- a/pytest_cases/fixture_core2.py +++ b/pytest_cases/fixture_core2.py @@ -21,6 +21,7 @@ except ImportError: pass +from .common_pytest_lazy_values import get_lazy_args from .common_pytest import get_pytest_parametrize_marks, make_marked_parameter_value, get_param_argnames_as_list, \ analyze_parameter_set, combine_ids, is_marked_parameter_value, get_marked_parameter_values, pytest_fixture from .fixture__creation import get_caller_module, check_name_available, WARN, CHANGE @@ -28,7 +29,7 @@ def param_fixture(argname, # type: str - argvalues, # type: Sequence[Any] + argvalues, # type: Iterable[Any] autouse=False, # type: bool ids=None, # type: Union[Callable, List[str]] scope="function", # type: str @@ -125,8 +126,8 @@ def __param_fixture(request): return fix -def param_fixtures(argnames, - argvalues, +def param_fixtures(argnames, # type: str + argvalues, # type: Iterable[Any] autouse=False, # type: bool ids=None, # type: Union[Callable, List[str]] scope="function", # type: str @@ -254,7 +255,7 @@ def _param_fixture(**_kwargs): def pytest_fixture_plus(*args, **kwargs): warn("`pytest_fixture_plus` is deprecated. Please use the new alias `fixture_plus`. " - "See https://github.com/pytest-dev/pytest/issues/6475") + "See https://github.com/pytest-dev/pytest/issues/6475", category=DeprecationWarning, stacklevel=2) if len(args) == 1: if callable(args[0]): return _decorate_fixture_plus(args[0], _caller_module_offset_when_unpack=2, **kwargs) @@ -283,6 +284,8 @@ def fixture_plus(scope="function", # type: str - it supports a new argument `unpack_into` where you can provide names for fixtures where to unpack this fixture into. + As a consequence it does not support the `params` and `ids` arguments anymore. + :param scope: the scope for which this fixture is shared, one of "function" (default), "class", "module" or "session". :param autouse: if True, the fixture func is activated for all tests that can see it. If False (the default) then @@ -408,7 +411,7 @@ def _decorate_fixture_plus(fixture_func, # reapply the marks for i, marks in enumerate(final_marks): if marks is not None: - final_values[i] = make_marked_parameter_value(final_values[i], marks=marks) + final_values[i] = make_marked_parameter_value((final_values[i],), marks=marks) else: final_values = list(product(*params_values)) final_ids = combine_ids(product(*params_ids)) @@ -418,7 +421,7 @@ def _decorate_fixture_plus(fixture_func, for i, marks in enumerate(final_marks): ms = [m for mm in marks if mm is not None for m in mm] if len(ms) > 0: - final_values[i] = make_marked_parameter_value(final_values[i], marks=ms) + final_values[i] = make_marked_parameter_value((final_values[i],), marks=ms) if len(final_values) != len(final_ids): raise ValueError("Internal error related to fixture parametrization- please report") @@ -440,8 +443,6 @@ def _decorate_fixture_plus(fixture_func, # --common routine used below. Fills kwargs with the appropriate names and values from fixture_params def _map_arguments(*_args, **_kwargs): - # todo better... - from .fixture_parametrize_plus import handle_lazy_args request = _kwargs['request'] if func_needs_request else _kwargs.pop('request') # populate the parameters @@ -452,12 +453,12 @@ def _map_arguments(*_args, **_kwargs): for p_names, fixture_param_value in zip(params_names_or_name_combinations, _params): if len(p_names) == 1: # a single parameter for that generated fixture (@pytest.mark.parametrize with a single name) - _kwargs[p_names[0]] = handle_lazy_args(fixture_param_value) + _kwargs[p_names[0]] = get_lazy_args(fixture_param_value) else: # several parameters for that generated fixture (@pytest.mark.parametrize with several names) # unpack all of them and inject them in the kwargs for old_p_name, old_p_value in zip(p_names, fixture_param_value): - _kwargs[old_p_name] = handle_lazy_args(old_p_value) + _kwargs[old_p_name] = get_lazy_args(old_p_value) return _args, _kwargs diff --git a/pytest_cases/fixture_parametrize_plus.py b/pytest_cases/fixture_parametrize_plus.py index b5508241..062f3bf2 100644 --- a/pytest_cases/fixture_parametrize_plus.py +++ b/pytest_cases/fixture_parametrize_plus.py @@ -1,9 +1,8 @@ -from collections import Iterable, namedtuple -from distutils.version import LooseVersion -from functools import partial +from collections import Iterable from inspect import isgeneratorfunction from warnings import warn + try: # python 3.3+ from inspect import signature, Parameter except ImportError: @@ -11,15 +10,19 @@ try: from typing import Union, Callable, List, Any, Sequence, Optional # noqa + except ImportError: pass import pytest from makefun import with_signature, remove_signature_parameters, add_signature_parameters, wraps -from .common_pytest import get_fixture_name, remove_duplicates, is_marked_parameter_value, mini_idvalset, \ - get_param_argnames_as_list, extract_parameterset_info, ParameterSet, has_pytest_param, get_pytest_marks_on_function, \ - transform_marks_into_decorators +from .common_mini_six import string_types +from .common_others import AUTO +from .common_pytest_marks import has_pytest_param, get_param_argnames_as_list +from .common_pytest_lazy_values import is_lazy_value, is_lazy, get_lazy_args +from .common_pytest import get_fixture_name, remove_duplicates, mini_idvalset, is_marked_parameter_value, \ + extract_parameterset_info, ParameterSet, cart_product_pytest, mini_idval from .fixture__creation import check_name_available, CHANGE, WARN, get_caller_module from .fixture_core1_unions import InvalidParamsList, NOT_USED, UnionFixtureAlternative, _make_fixture_union, \ @@ -57,7 +60,7 @@ def _fixture_product(caller_module, if not isinstance(fixtures_or_values, (tuple, set, list)): raise TypeError("fixture_product: the `fixtures_or_values` argument should be a tuple, set or list") else: - has_lazy_vals = any(isinstance(v, lazy_value) for v in fixtures_or_values) + has_lazy_vals = any(is_lazy_value(v) for v in fixtures_or_values) _tuple_size = len(fixtures_or_values) @@ -82,7 +85,7 @@ def _tuple_generator(all_fixtures): if fix_at_pos_i is None: # fixed value # note: wouldnt it be almost as efficient but more readable to *always* call handle_lazy_args? - yield handle_lazy_args(fixtures_or_values[i]) if has_lazy_vals else fixtures_or_values[i] + yield get_lazy_args(fixtures_or_values[i]) if has_lazy_vals else fixtures_or_values[i] else: # fixture value yield all_fixtures[fix_at_pos_i] @@ -116,7 +119,7 @@ def _new_fixture(**all_fixtures): class fixture_ref(object): # noqa """ - A reference to a fixture, to be used in `parametrize_plus`. + A reference to a fixture, to be used in `@parametrize_plus`. You can create it from a fixture name or a fixture object (function). """ __slots__ = 'fixture', @@ -125,161 +128,6 @@ def __init__(self, fixture): self.fixture = fixture -pytest53 = LooseVersion(pytest.__version__) >= LooseVersion("5.3.0") -if pytest53: - # in the latest versions of pytest, the default _idmaker returns the value of __name__ if it is available, - # even if an object is not a class nor a function. So we do not need to use any special trick. - _LazyValueBase = object -else: - fake_base = int - - class _LazyValueBase(int, object): - """ - in this older version of pytest, the default _idmaker does *not* return the value of __name__ for - objects that are not functions not classes. However it *does* return str(obj) for objects that are - instances of bool, int or float. So that's why lazy_value inherits from int. - """ - __slots__ = () - - def __new__(cls, - valuegetter, # type: Callable[[], Any] - id=None, # type: str - marks=() # type: Sequence - ): - """ Inheriting from int is a bit hard in python: we have to override __new__ """ - obj = fake_base.__new__(cls, 111111) - cls.__init__(obj, valuegetter=valuegetter, id=id, marks=marks) - return obj - - def __getattribute__(self, item): - """Map all default attribute and method access to the ones in object, not in int""" - return object.__getattribute__(self, item) - - def __repr__(self): - """Magic methods are not intercepted by __getattribute__ and need to be overridden manually. - We do not need all of them by at least override this one for easier debugging""" - return object.__repr__(self) - - -def _unwrap(obj): - """A light copy of _pytest.compat.get_real_func. In our case - we do not wish to unwrap the partial nor handle pytest fixture """ - start_obj = obj - for i in range(100): - # __pytest_wrapped__ is set by @pytest.fixture when wrapping the fixture function - # to trigger a warning if it gets called directly instead of by pytest: we don't - # want to unwrap further than this otherwise we lose useful wrappings like @mock.patch (#3774) - # new_obj = getattr(obj, "__pytest_wrapped__", None) - # if isinstance(new_obj, _PytestWrapper): - # obj = new_obj.obj - # break - new_obj = getattr(obj, "__wrapped__", None) - if new_obj is None: - break - obj = new_obj - else: - raise ValueError("could not find real function of {start}\nstopped at {current}".format( - start=repr(start_obj), current=repr(obj) - ) - ) - return obj - - -def partial_to_str(partialfun): - """Return a string representation of a partial function, to use in lazy_value ids""" - strwds = ", ".join("%s=%s" % (k, v) for k, v in partialfun.keywords.items()) - if len(partialfun.args) > 0: - strargs = ', '.join(str(i) for i in partialfun.args) - if len(partialfun.keywords) > 0: - strargs = "%s, %s" % (strargs, strwds) - else: - strargs = strwds - return "%s(%s)" % (partialfun.func.__name__, strargs) - - -# noinspection PyPep8Naming -class lazy_value(_LazyValueBase): - """ - A reference to a value getter, to be used in `parametrize_plus`. - - A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a - fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. - """ - if pytest53: - __slots__ = 'valuegetter', '_id', '_marks' - else: - # we can not define __slots__ since we extend int, - # see https://docs.python.org/3/reference/datamodel.html?highlight=__slots__#notes-on-using-slots - pass - - # noinspection PyMissingConstructor - def __init__(self, - valuegetter, # type: Callable[[], Any] - id=None, # type: str - marks=() # type: Sequence - ): - """ - Creates a reference to a value getter, to be used in `parametrize_plus`. - - A `lazy_value` is the same thing than a function-scoped fixture, except that the value getter function is not a - fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. - - Note that a `lazy_value` can be included in a `pytest.param` without problem. In that case the id defined by - `pytest.param` will take precedence over the one defined in `lazy_value` if any. The marks, however, - will all be kept wherever they are defined. - - :param valuegetter: a callable without mandatory arguments - :param id: an optional id. Otherwise `valuegetter.__name__` will be used by default - :param marks: optional marks. `valuegetter` marks will also be preserved. - """ - self.valuegetter = valuegetter - self._id = id - self._marks = marks - - def get_marks(self, as_decorators=False): - """ - Overrides default implementation to return the marks that are on the case function - - :param as_decorators: when True, the marks will be transformed into MarkDecorators before being - returned - :return: - """ - valuegetter_marks = get_pytest_marks_on_function(self.valuegetter, as_decorators=as_decorators) - - if self._marks: - return transform_marks_into_decorators(self._marks) + valuegetter_marks - else: - return valuegetter_marks - - def get_id(self): - """The id to use in pytest""" - if self._id is not None: - return self._id - else: - # default is the __name__ of the value getter - _id = getattr(self.valuegetter, '__name__', None) - if _id is not None: - return _id - - # unwrap and handle partial functions - vg = _unwrap(self.valuegetter) - - if isinstance(vg, partial): - return partial_to_str(vg) - else: - return vg.__name__ - - if not pytest53: - def __str__(self): - """in pytest<5.3 we inherit from int so that str(v) is called by pytest _idmaker to get the id""" - return self.get_id() - - @property - def __name__(self): - """for pytest >= 5.3 we override this so that pytest uses it for id""" - return self.get_id() - - # Fix for https://github.com/smarie/python-pytest-cases/issues/71 # In order for pytest to allow users to import this symbol in conftest.py # they should be declared as optional plugin hooks. @@ -289,7 +137,7 @@ def __name__(self): def pytest_parametrize_plus(*args, **kwargs): warn("`pytest_parametrize_plus` is deprecated. Please use the new alias `parametrize_plus`. " - "See https://github.com/pytest-dev/pytest/issues/6475") + "See https://github.com/pytest-dev/pytest/issues/6475", category=DeprecationWarning, stacklevel=2) return _parametrize_plus(*args, **kwargs) @@ -422,17 +270,35 @@ def get(cls, style # type: str raise ValueError("Unknown style: %r" % style) -def parametrize_plus(argnames, - argvalues, +_IDGEN = object() + + +def parametrize_plus(argnames=None, # type: str + argvalues=None, # type: Iterable[Any] indirect=False, # type: bool ids=None, # type: Union[Callable, List[str]] idstyle='explicit', # type: str + idgen=_IDGEN, # type: Union[str, Callable] scope=None, # type: str hook=None, # type: Callable[[Callable], Callable] debug=False, # type: bool - **kwargs): + **args): """ - Equivalent to `@pytest.mark.parametrize` but also supports new possibilities in argvalues: + Equivalent to `@pytest.mark.parametrize` but also supports + + (1) new alternate style for argnames/argvalues. One can also use `**args` to pass additional `{argnames: argvalues}` + in the same parametrization call. This can be handy in combination with `idgen` to master the whole id template + associated with several parameters. Note that you can pass coma-separated argnames too, by de-referencing a dict: + e.g. `**{'a,b': [(0, True), (1, False)], 'c': [-1, 2]}`. + + (2) new alternate style for ids. One can use `idgen` instead of `ids`. `idgen` can be a callable receiving all + parameters at once (`**args`) and returning an id ; or it can be a string template using the new-style string + formatting where the argnames can be used as variables (e.g. `idgen=lambda **args: "a={a}".format(**args)` or + `idgen="my_id where a={a}"`). The special `idgen=AUTO` symbol can be used to generate a default string template + equivalent to `lambda **args: "-".join("%s=%s" % (n, v) for n, v in args.items())`. This is enabled by default + if you use the alternate style for argnames/argvalues (e.g. if `len(args) > 0`). + + (3) new possibilities in argvalues: - one can include references to fixtures with `fixture_ref()` where can be the fixture name or fixture function. When such a fixture reference is detected in the argvalues, a new function-scope "union" fixture @@ -457,7 +323,11 @@ def parametrize_plus(argnames, :param argnames: same as in pytest.mark.parametrize :param argvalues: same as in pytest.mark.parametrize except that `fixture_ref` and `lazy_value` are supported :param indirect: same as in pytest.mark.parametrize - :param ids: same as in pytest.mark.parametrize + :param ids: same as in pytest.mark.parametrize. Note that an alternative way to create ids exists with `idgen`. Only + one non-None `ids` or `idgen should be provided. + :param idgen: an id formatter. Either a string representing a template, or a callable receiving all argvalues + at once (as opposed to the behaviour in pytest ids). This alternative way to generate ids can only be used when + `ids` is not provided (None). :param idstyle: style of ids to be used in generated "union" fixtures. See `fixture_union` for details. :param scope: same as in pytest.mark.parametrize :param hook: an optional hook to apply to each fixture function that is created during this call. The hook function @@ -465,223 +335,108 @@ def parametrize_plus(argnames, implementing the fixture) and should return the function to use. For example you can use `saved_fixture` from `pytest-harvest` as a hook in order to save all such created fixtures in the fixture store. :param debug: print debug messages on stdout to analyze fixture creation (use pytest -s to see them) - :param kwargs: additional arguments for pytest.mark.parametrize + :param args: additional {argnames: argvalues} definition :return: """ - return _parametrize_plus(argnames, argvalues, indirect=indirect, ids=ids, idstyle=idstyle, scope=scope, hook=hook, - debug=debug, **kwargs) - - -def handle_lazy_args(argval): - """ Possibly calls the lazy values contained in argval if needed, before returning it""" - - # First handle the general case - try: - if not isinstance(argval, (lazy_value, LazyTuple, LazyTuple.LazyItem)): - return argval - except: # noqa - return argval - - # Now the lazy ones - if isinstance(argval, lazy_value): - return argval.valuegetter() - elif isinstance(argval, LazyTuple): - return argval.get() - elif isinstance(argval, LazyTuple.LazyItem): - return argval.get() - else: - return argval + return _parametrize_plus(argnames, argvalues, indirect=indirect, ids=ids, idgen=idgen, idstyle=idstyle, scope=scope, + hook=hook, debug=debug, **args) -class LazyTuple(object): +class InvalidIdTemplateException(Exception): """ - A wrapper representing a lazy_value used as a tuple = for several argvalues at once. - - - - while not calling the lazy value - - + Raised when a string template provided in an `idgen` raises an error """ - __slots__ = ('value', 'theoretical_size', 'retrieved') - - def __init__(self, - valueref, # type: Union[lazy_value, Sequence] - theoretical_size # type: int - ): - self.value = valueref - self.theoretical_size = theoretical_size - self.retrieved = False - - def __len__(self): - return self.theoretical_size - - def get_id(self): - """return the id to use by pytest""" - return self.value.get_id() - - class LazyItem(namedtuple('LazyItem', ('host', 'item'))): - def get(self): - return self.host.force_getitem(self.item) - - def __getitem__(self, item): - """ - Getting an item in the tuple with self[i] does *not* retrieve the value automatically, but returns - a facade (a LazyItem), so that pytest can store this item independently wherever needed, without - yet calling the value getter. - """ - if self.retrieved: - # this is never called by pytest, but keep it for debugging - return self.value[item] - else: - # do not retrieve yet: return a facade - return LazyTuple.LazyItem(self, item) + def __init__(self, idgen, params, caught): + self.idgen = idgen + self.params = params + self.caught = caught + super(InvalidIdTemplateException, self).__init__() - def force_getitem(self, item): - """ Call the underlying value getter, then return self[i]. """ - return self.get()[item] + def __str__(self): + return repr(self) - def get(self): - """ Call the underlying value getter, then return the tuple (not self) """ - if not self.retrieved: - # retrieve - self.value = self.value.valuegetter() - self.retrieved = True - return self.value + def __repr__(self): + return "Error generating test id using name template '%s' with parameter values " \ + "%r. Please check the name template. Caught: %s - %s" \ + % (self.idgen, self.params, self.caught.__class__, self.caught) -def _parametrize_plus(argnames, - argvalues, +def _parametrize_plus(argnames=None, + argvalues=None, indirect=False, # type: bool ids=None, # type: Union[Callable, List[str]] idstyle='explicit', # type: str + idgen=_IDGEN, # type: Union[str, Callable] scope=None, # type: str hook=None, # type: Callable[[Callable], Callable] _frame_offset=2, debug=False, # type: bool - **kwargs): - # make sure that we do not destroy the argvalues if it is provided as an iterator - try: - argvalues = list(argvalues) - except TypeError: - raise InvalidParamsList(argvalues) - - # get the param names - initial_argnames = argnames - argnames = get_param_argnames_as_list(argnames) - nb_params = len(argnames) - - # extract all marks and custom ids. - # Do not check consistency of sizes argname/argvalue as a fixture_ref can stand for several argvalues. - marked_argvalues = argvalues - custom_pids, p_marks, argvalues = extract_parameterset_info(argnames, argvalues, check_nb=False) - - # find if there are fixture references in the values provided - fixture_indices = [] - if nb_params == 1: - for i, v in enumerate(argvalues): - if isinstance(v, lazy_value): - # Note: no need to modify the id, it will be ok thanks to the lazy_value class design - # handle marks - _mks = v.get_marks(as_decorators=True) - if len(_mks) > 0: - # merge with the mark decorators possibly already present with pytest.param - if p_marks[i] is None: - p_marks[i] = [] - p_marks[i] = list(p_marks[i]) + _mks - - # update the marked_argvalues - marked_argvalues[i] = ParameterSet(values=(argvalues[i],), id=custom_pids[i], marks=p_marks[i]) - del _mks - - if isinstance(v, fixture_ref): - fixture_indices.append((i, None)) - elif nb_params > 1: - for i, v in enumerate(argvalues): - if isinstance(v, lazy_value): - # a lazy value is used for several parameters at the same time, and is NOT between pytest.param() - argvalues[i] = LazyTuple(v, nb_params) - - # TUPLE usage: we HAVE to set an id to prevent too early access to the value by _idmaker - # note that on pytest 2 we cannot set an id here, so the lazy value wont be too lazy - assert custom_pids[i] is None - _id = v.get_id() - if not has_pytest_param: - warn("The custom id %r in `lazy_value` will be ignored as this version of pytest is too old to" - " support `pytest.param`." % _id) - _id = None - - # handle marks - _mks = v.get_marks(as_decorators=True) - if len(_mks) > 0: - # merge with the mark decorators possibly already present with pytest.param - assert p_marks[i] is None - p_marks[i] = _mks + **args): - # note that here argvalues[i] IS a tuple-like so we do not create a tuple around it - marked_argvalues[i] = ParameterSet(values=argvalues[i], id=_id, marks=_mks) - custom_pids[i] = _id - del _id, _mks + # idgen default + if idgen is _IDGEN: + idgen = AUTO if len(args) > 0 else None - elif isinstance(v, fixture_ref): - # a fixture ref is used for several parameters at the same time - fixture_indices.append((i, None)) + if idgen is AUTO: + # note: we use a "trick" here with mini_idval to get the appropriate result + # TODO support fixture_ref in mini_idval or add __name__ and str() in fixture_ref + idgen = lambda **args: "-".join("%s=%s" % (n, mini_idval(val=v, argname='', idx=v)) for n, v in args.items()) - elif len(v) == 1 and isinstance(v[0], lazy_value): - # same than above but it was in a pytest.mark - # valueref_indices.append((i, None)) - argvalues[i] = LazyTuple(v[0], nb_params) # unpack it - if custom_pids[i] is None: - # force-use the id from the lazy value (do not have pytest request for it, that would unpack it) - custom_pids[i] = v[0].get_id() - # handle marks - _mks = v[0].get_marks(as_decorators=True) - if len(_mks) > 0: - # merge with the mark decorators possibly already present with pytest.param - if p_marks[i] is None: - p_marks[i] = [] - p_marks[i] = list(p_marks[i]) + _mks - del _mks - marked_argvalues[i] = ParameterSet(values=argvalues[i], id=custom_pids[i], marks=p_marks[i]) + # first handle argnames / argvalues (new modes of input) + argnames, argvalues = _get_argnames_argvalues(argnames, argvalues, **args) - elif len(v) == 1 and isinstance(v[0], fixture_ref): - # same than above but it was in a pytest.mark - fixture_indices.append((i, None)) - argvalues[i] = v[0] # unpack it - else: - # check for consistency - if len(v) != len(argnames): - raise ValueError("Inconsistent number of values in pytest parametrize: %s items found while the " - "number of parameters is %s: %s." % (len(v), len(argnames), v)) + # argnames related + initial_argnames = ','.join(argnames) + nb_params = len(argnames) - # let's dig into the tuple - fix_pos_list = [j for j, _pval in enumerate(v) if isinstance(_pval, fixture_ref)] - if len(fix_pos_list) > 0: - # there is at least one fixture ref inside the tuple - fixture_indices.append((i, fix_pos_list)) + # extract all marks and custom ids. + # Do not check consistency of sizes argname/argvalue as a fixture_ref can stand for several argvalues. + marked_argvalues = argvalues + p_ids, p_marks, argvalues, fixture_indices = _process_argvalues(argnames, marked_argvalues, nb_params) - # let's dig into the tuple - # has_val_ref = any(isinstance(_pval, lazy_value) for _pval in v) - # val_pos_list = [j for j, _pval in enumerate(v) if isinstance(_pval, lazy_value)] - # if len(val_pos_list) > 0: - # # there is at least one value ref inside the tuple - # argvalues[i] = tuple_with_value_refs(v, theoreticalsize=nb_params, positions=val_pos_list) - del i + # generate id + if idgen is not None: + if ids is not None: + raise ValueError("Only one of `ids` and `idgen` should be provided") + ids = _gen_ids(argnames, argvalues, idgen) if len(fixture_indices) == 0: if debug: print("No fixture reference found. Calling @pytest.mark.parametrize...") + print(" - argnames: %s" % initial_argnames) + print(" - argvalues: %s" % marked_argvalues) + print(" - ids: %s" % ids) + # no fixture reference: shortcut, do as usual (note that the hook wont be called since no fixture is created) - return pytest.mark.parametrize(initial_argnames, marked_argvalues, indirect=indirect, - ids=ids, scope=scope, **kwargs) + _decorator = pytest.mark.parametrize(initial_argnames, marked_argvalues, indirect=indirect, + ids=ids, scope=scope) + if indirect: + return _decorator + else: + # wrap the decorator to check if the test function has the parameters as arguments + def _apply(test_func): + s = signature(test_func) + for p in argnames: + if p not in s.parameters: + raise ValueError("parameter '%s' not found in test function signature '%s%s'" + "" % (p, test_func.__name__, s)) + return _decorator(test_func) + + return _apply + else: - if len(kwargs) > 0: - warn("Unsupported kwargs for `parametrize_plus`: %r" % kwargs) + if indirect: + raise ValueError("Setting `indirect=True` is not yet supported when at least a `fixure_ref` is present in " + "the `argvalues`.") if debug: print("Fixture references found. Creating fixtures...") - # there are fixture references: we have to create a specific decorator + + # there are fixture references: we will create a specific decorator replacing the params with a "union" fixture caller_module = get_caller_module(frame_offset=_frame_offset) param_names_str = '_'.join(argnames).replace(' ', '') + # First define a few functions that will help us create the various fixtures to use in the final "union" def _make_idfun_for_params(argnames, # noqa nb_positions): """ @@ -701,7 +456,7 @@ def _tmp_make_id(argvals): raise ValueError("Internal error, please report") if len(argnames) <= 1: argvals = (argvals,) - elif isinstance(argvals, LazyTuple): + elif is_lazy(argvals): return argvals.get_id() return mini_idvalset(argnames, argvals, idx=_tmp_make_id.i) @@ -750,22 +505,22 @@ def _create_params_alt(test_func_name, union_name, from_i, to_i, hook): # noqa # If an explicit list of ids was provided, slice it. Otherwise use the provided callable try: - p_ids = ids[from_i:to_i] + param_ids = ids[from_i:to_i] except TypeError: # callable ? otherwise default to a customized id maker that replaces the fixture name # that we use (p_fix_name) with a simpler name in the ids (just the argnames) - p_ids = ids or _make_idfun_for_params(argnames=argnames, nb_positions=(to_i - from_i)) + param_ids = ids or _make_idfun_for_params(argnames=argnames, nb_positions=(to_i - from_i)) # Create the fixture that will take ALL these parameter values (in a single parameter) - # That fixture WILL be parametrized, this is why we propagate the p_ids and use the marked values + # That fixture WILL be parametrized, this is why we propagate the param_ids and use the marked values if nb_params == 1: _argvals = marked_argvalues[from_i:to_i] else: # we have to create a tuple around the vals because we have a SINGLE parameter that is a tuple _argvals = tuple(ParameterSet((vals, ), id=id, marks=marks or ()) for vals, id, marks in zip(argvalues[from_i:to_i], - custom_pids[from_i:to_i], p_marks[from_i:to_i])) - _create_param_fixture(caller_module, argname=p_fix_name, argvalues=_argvals, ids=p_ids, hook=hook) + p_ids[from_i:to_i], p_marks[from_i:to_i])) + _create_param_fixture(caller_module, argname=p_fix_name, argvalues=_argvals, ids=param_ids, hook=hook) # todo put back debug=debug above @@ -796,12 +551,12 @@ def _create_fixture_ref_product(union_name, i, fixture_ref_positions, test_func_ # If an explicit list of ids was provided, slice it. Otherwise use the provided callable try: - p_ids = ids[i] + param_ids = ids[i] except TypeError: - p_ids = ids # callable + param_ids = ids # callable # values to use: - p_values = argvalues[i] + param_values = argvalues[i] # Create a unique fixture name p_fix_name = "%s_%s_P%s" % (test_func_name, param_names_str, i) @@ -811,8 +566,8 @@ def _create_fixture_ref_product(union_name, i, fixture_ref_positions, test_func_ print("Creating fixture %r to handle parameter %s that is a cross-product" % (p_fix_name, i)) # Create the fixture - _make_fixture_product(caller_module, name=p_fix_name, hook=hook, ids=p_ids, fixtures_or_values=p_values, - fixture_positions=fixture_ref_positions, caller=parametrize_plus) + _make_fixture_product(caller_module, name=p_fix_name, hook=hook, caller=parametrize_plus, ids=param_ids, + fixtures_or_values=param_values, fixture_positions=fixture_ref_positions) # Create the corresponding alternative p_fix_alt = ProductParamAlternative(union_name=union_name, alternative_name=p_fix_name, @@ -823,7 +578,7 @@ def _create_fixture_ref_product(union_name, i, fixture_ref_positions, test_func_ return p_fix_name, p_fix_alt - # then create the decorator + # Then create the decorator per se def parametrize_plus_decorate(test_func): """ A decorator that wraps the test function so that instead of receiving the parameter names, it receives the @@ -1001,3 +756,188 @@ def wrapped_test_func(*args, **kwargs): # noqa return wrapped_test_func return parametrize_plus_decorate + + +def _get_argnames_argvalues(argnames=None, argvalues=None, **args): + """ + + :param argnames: + :param argvalues: + :param args: + :return: argnames, argvalues - both guaranteed to be lists + """ + # handle **args - a dict of {argnames: argvalues} + if len(args) > 0: + kw_argnames, kw_argvalues = cart_product_pytest(tuple(args.keys()), tuple(args.values())) + else: + kw_argnames, kw_argvalues = (), () + + if argnames is None: + # (1) all {argnames: argvalues} pairs are provided in **args + if argvalues is not None or len(args) == 0: + raise ValueError("No parameters provided") + + argnames = kw_argnames + argvalues = kw_argvalues + # simplify if needed to comply with pytest.mark.parametrize + if len(argnames) == 1: + argvalues = [l[0] if not is_marked_parameter_value(l) else l for l in argvalues] + + elif isinstance(argnames, string_types): + # (2) argnames + argvalues, as usual. However **args can also be passed and should be added + argnames = get_param_argnames_as_list(argnames) + + if argvalues is None: + raise ValueError("No argvalues provided while argnames are provided") + + # transform argvalues to a list (it can be a generator) + try: + argvalues = list(argvalues) + except TypeError: + raise InvalidParamsList(argvalues) + + # append **args + if len(kw_argnames) > 0: + argnames, argvalues = cart_product_pytest((argnames, kw_argnames), + (argvalues, kw_argvalues)) + + return argnames, argvalues + + +def _gen_ids(argnames, argvalues, idgen): + """ + Generates an explicit test ids list from a non-none `idgen`. + + `idgen` should be either a callable of a string template. + + :param argnames: + :param argvalues: + :param idgen: + :return: + """ + if not callable(idgen): + _formatter = idgen + + def gen_id_using_str_formatter(**params): + try: + return _formatter.format(**params) + except Exception as e: + raise InvalidIdTemplateException(_formatter, params, e) + + idgen = gen_id_using_str_formatter + if len(argnames) > 1: + ids = [idgen(**{n: v for n, v in zip(argnames, _argvals)}) for _argvals in argvalues] + else: + _only_name = argnames[0] + ids = [idgen(**{_only_name: v}) for v in argvalues] + + return ids + + +def _process_argvalues(argnames, marked_argvalues, nb_params): + """Internal method to use in _pytest_parametrize_plus + + Processes the provided marked_argvalues (possibly marked with pytest.param) and returns + p_ids, p_marks, argvalues (not marked with pytest.param), fixture_indices + + Note: `marked_argvalues` is modified in the process if a `lazy_value` is found with a custom id or marks. + + :param argnames: + :param marked_argvalues: + :param nb_params: + :return: + """ + p_ids, p_marks, argvalues = extract_parameterset_info(argnames, marked_argvalues, check_nb=False) + + # find if there are fixture references or lazy values in the values provided + fixture_indices = [] + if nb_params == 1: + for i, v in enumerate(argvalues): + if is_lazy_value(v): + # Note: no need to modify the id, it will be ok thanks to the lazy_value class design + # handle marks + _mks = v.get_marks(as_decorators=True) + if len(_mks) > 0: + # merge with the mark decorators possibly already present with pytest.param + if p_marks[i] is None: + p_marks[i] = [] + p_marks[i] = list(p_marks[i]) + _mks + + # update the marked_argvalues + marked_argvalues[i] = ParameterSet(values=(argvalues[i],), id=p_ids[i], marks=p_marks[i]) + del _mks + + if isinstance(v, fixture_ref): + fixture_indices.append((i, None)) + elif nb_params > 1: + for i, v in enumerate(argvalues): + if is_lazy_value(v): + # a lazy value is used for several parameters at the same time, and is NOT between pytest.param() + argvalues[i] = v.as_lazy_tuple(nb_params) + + # TUPLE usage: we HAVE to set an id to prevent too early access to the value by _idmaker + # note that on pytest 2 we cannot set an id here, the lazy value wont be too lazy + assert p_ids[i] is None + _id = v.get_id() + if not has_pytest_param: + warn("The custom id %r in `lazy_value` will be ignored as this version of pytest is too old to" + " support `pytest.param`." % _id) + _id = None + + # handle marks + _mks = v.get_marks(as_decorators=True) + if len(_mks) > 0: + # merge with the mark decorators possibly already present with pytest.param + assert p_marks[i] is None + p_marks[i] = _mks + + # note that here argvalues[i] IS a tuple-like so we do not create a tuple around it + marked_argvalues[i] = ParameterSet(values=argvalues[i], id=_id, marks=_mks) + p_ids[i] = _id + del _id, _mks + + elif isinstance(v, fixture_ref): + # a fixture ref is used for several parameters at the same time + fixture_indices.append((i, None)) + + elif len(v) == 1 and is_lazy_value(v[0]): + # same than above but it was in a pytest.mark + # valueref_indices.append((i, None)) + argvalues[i] = v[0].as_lazy_tuple(nb_params) # unpack it + if p_ids[i] is None: + # force-use the id from the lazy value (do not have pytest request for it, that would unpack it) + p_ids[i] = v[0].get_id() + # handle marks + _mks = v[0].get_marks(as_decorators=True) + if len(_mks) > 0: + # merge with the mark decorators possibly already present with pytest.param + if p_marks[i] is None: + p_marks[i] = [] + p_marks[i] = list(p_marks[i]) + _mks + del _mks + marked_argvalues[i] = ParameterSet(values=argvalues[i], id=p_ids[i], marks=p_marks[i]) + + elif len(v) == 1 and isinstance(v[0], fixture_ref): + # same than above but it was in a pytest.mark + fixture_indices.append((i, None)) + argvalues[i] = v[0] # unpack it + else: + # check for consistency + if len(v) != len(argnames): + raise ValueError("Inconsistent number of values in pytest parametrize: %s items found while the " + "number of parameters is %s: %s." % (len(v), len(argnames), v)) + + # let's dig into the tuple + fix_pos_list = [j for j, _pval in enumerate(v) if isinstance(_pval, fixture_ref)] + if len(fix_pos_list) > 0: + # there is at least one fixture ref inside the tuple + fixture_indices.append((i, fix_pos_list)) + + # let's dig into the tuple + # has_val_ref = any(isinstance(_pval, lazy_value) for _pval in v) + # val_pos_list = [j for j, _pval in enumerate(v) if isinstance(_pval, lazy_value)] + # if len(val_pos_list) > 0: + # # there is at least one value ref inside the tuple + # argvalues[i] = tuple_with_value_refs(v, theoreticalsize=nb_params, positions=val_pos_list) + + return p_ids, p_marks, argvalues, fixture_indices diff --git a/pytest_cases/plugin.py b/pytest_cases/plugin.py index 1d4cd6e9..99f4a288 100644 --- a/pytest_cases/plugin.py +++ b/pytest_cases/plugin.py @@ -18,16 +18,57 @@ pass from .common_mini_six import string_types +from .common_pytest_lazy_values import get_lazy_args from .common_pytest import get_pytest_nodeid, get_pytest_function_scopenum, is_function_node, get_param_names, \ get_pytest_scopenum, get_param_argnames_as_list from .fixture_core1_unions import NOT_USED, is_fixture_union_params, UnionFixtureAlternative -from .fixture_parametrize_plus import handle_lazy_args _DEBUG = False +# @pytest.hookimpl(hookwrapper=True, tryfirst=True) +# def pytest_pycollect_makeitem(collector, name, obj): +# # custom collection of additional things for Cases for example +# # see also https://hackebrot.github.io/pytest-tricks/customize_class_collection/ +# outcome = yield +# res = outcome.get_result() +# if res is not None: +# return + # nothing was collected elsewhere, let's do it here + # if safe_isclass(obj): + # if collector.istestclass(obj, name): + # outcome.force_result(Class(name, parent=collector)) + # elif collector.istestfunction(obj, name): + # # mock seems to store unbound methods (issue473), normalize it + # obj = getattr(obj, "__func__", obj) + # # We need to try and unwrap the function if it's a functools.partial + # # or a functools.wrapped. + # # We mustn't if it's been wrapped with mock.patch (python 2 only) + # if not (inspect.isfunction(obj) or inspect.isfunction(get_real_func(obj))): + # filename, lineno = getfslineno(obj) + # warnings.warn_explicit( + # message=PytestCollectionWarning( + # "cannot collect %r because it is not a function." % name + # ), + # category=None, + # filename=str(filename), + # lineno=lineno + 1, + # ) + # elif getattr(obj, "__test__", True): + # if is_generator(obj): + # res = Function(name, parent=collector) + # reason = "yield tests were removed in pytest 4.0 - {name} will be ignored".format( + # name=name + # ) + # res.add_marker(MARK_GEN.xfail(run=False, reason=reason)) + # res.warn(PytestCollectionWarning(reason)) + # else: + # res = list(collector._genfunctions(name, obj)) + # outcome.force_result(res) + + @pytest.hookimpl(tryfirst=True, hookwrapper=True) def pytest_runtest_setup(item # type: Function ): @@ -39,7 +80,7 @@ def pytest_runtest_setup(item # type: Function yield # now item.funcargs exists so we can handle it - item.funcargs = {argname: handle_lazy_args(argvalue) for argname, argvalue in item.funcargs.items()} + item.funcargs = {argname: get_lazy_args(argvalue) for argname, argvalue in item.funcargs.items()} # @hookspec(firstresult=True) @@ -691,7 +732,7 @@ def parametrize(metafunc, argnames, argvalues, indirect=False, ids=None, scope=N The resulting `CallsReactor` instance is then able to dynamically behave like the correct list of calls, lazy-creating that list when it is used. """ - # create our special container object if needed + # create our special container object TODO maybe we could be lazy and create it only when a union appears if not isinstance(metafunc._calls, CallsReactor): # noqa # first call: should be an empty list if len(metafunc._calls) > 0: # noqa diff --git a/pytest_cases/tests/cases/doc/cases_doc.py b/pytest_cases/tests/cases/doc/cases_doc.py new file mode 100644 index 00000000..33a430c0 --- /dev/null +++ b/pytest_cases/tests/cases/doc/cases_doc.py @@ -0,0 +1,44 @@ +import pytest + +from pytest_cases import case + + +@case(marks=pytest.mark.skipif(True, reason="hello")) +def two_positive_ints(): + """ Inputs are two positive integers """ + return 1, 2 + + +class CasesFoo: + @classmethod + def case_toto(cls): + return + + @staticmethod + def case_foo(): + return + + @case(id="hello") + def case_blah(self): + """a blah""" + return 0, 0 + + @pytest.mark.skip + def case_skipped(self): + """skipped case""" + return 0 + + def case_two_negative_ints(self): + """ Inputs are two negative integers """ + return -1, -2 + + +@pytest.mark.skip +def case_three_negative_ints(): + """ Inputs are three negative integers """ + return -1, -2, -6 + + +def case_two_negative_ints(): + """ Inputs are two negative integers """ + return -1, -2 diff --git a/pytest_cases/tests/cases/doc/example.py b/pytest_cases/tests/cases/doc/example.py new file mode 100644 index 00000000..704f2620 --- /dev/null +++ b/pytest_cases/tests/cases/doc/example.py @@ -0,0 +1,4 @@ + +def foo(a, b): + """ the function to test ! """ + return a + 1, b + 1 diff --git a/pytest_cases/tests/cases/doc/test_doc.py b/pytest_cases/tests/cases/doc/test_doc.py new file mode 100644 index 00000000..61c11117 --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_doc.py @@ -0,0 +1,184 @@ +import pytest + +from pytest_harvest import get_session_synthesis_dct +from pytest_cases import parametrize_with_cases, AUTO2, fixture, case +from pytest_cases.common_pytest import has_pytest_param + +from . import cases_doc +from .example import foo + + +@pytest.mark.parametrize("a,b", [(1, 2), (-1, -2)]) +def test_foo1(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo1_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo1, test_id_format='function') + assert list(results_dct) == ['test_foo1[1-2]', 'test_foo1[-1--2]'] + + +@parametrize_with_cases("a,b") +def test_foo_default_cases_file(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_default_cases_file_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_default_cases_file, test_id_format='function') + assert list(results_dct) == [ + 'test_foo_default_cases_file[%s]' % ('two_positive_ints' if has_pytest_param else 'two_positive_ints[0]-two_positive_ints[1]'), + 'test_foo_default_cases_file[%s]' % ('two_negative_ints' if has_pytest_param else 'two_negative_ints[0]-two_negative_ints[1]') + ] + + +@parametrize_with_cases("a,b", cases=AUTO2) +def test_foo_alternate_cases_file_and_two_marked_skip(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_alternate_cases_file_and_two_marked_skip_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_alternate_cases_file_and_two_marked_skip, + test_id_format='function') + if has_pytest_param: + assert list(results_dct) == [ + 'test_foo_alternate_cases_file_and_two_marked_skip[hello]', + 'test_foo_alternate_cases_file_and_two_marked_skip[two_negative_ints0]', + 'test_foo_alternate_cases_file_and_two_marked_skip[two_negative_ints1]' + ] + else: + assert list(results_dct) == [ + 'test_foo_alternate_cases_file_and_two_marked_skip[0hello[0]-hello[1]]', + 'test_foo_alternate_cases_file_and_two_marked_skip[2two_negative_ints[0]-two_negative_ints[1]]', + 'test_foo_alternate_cases_file_and_two_marked_skip[4two_negative_ints[0]-two_negative_ints[1]]' + ] + + +def strange_ints(): + """ Inputs are two negative integers """ + return -1, -2 + + +@parametrize_with_cases("a,b", cases=strange_ints) +def test_foo_fun(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_fun_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_fun, test_id_format='function') + if has_pytest_param: + assert list(results_dct) == ['test_foo_fun[strange_ints]'] + else: + assert list(results_dct) == ['test_foo_fun[strange_ints[0]-strange_ints[1]]'] + + +@parametrize_with_cases("a,b", cases=(strange_ints, strange_ints)) +def test_foo_fun_list(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_fun_list_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_fun_list, test_id_format='function') + if has_pytest_param: + assert list(results_dct) == [ + 'test_foo_fun_list[strange_ints0]', + 'test_foo_fun_list[strange_ints1]' + ] + else: + assert list(results_dct) == [ + 'test_foo_fun_list[0strange_ints[0]-strange_ints[1]]', + 'test_foo_fun_list[1strange_ints[0]-strange_ints[1]]' + ] + + +class CasesFoo: + @classmethod + def case_toto(cls): + return + + @staticmethod + def case_foo(): + return + + @pytest.mark.skipif(False, reason="no") + @case(id="hello world") + def case_blah(self): + """a blah""" + return 0, 0 + + @pytest.mark.skip + def case_skipped(self): + """skipped case""" + return 0 + + def case_two_negative_ints(self): + """ Inputs are two negative integers """ + return -1, -2 + + +@parametrize_with_cases("a,b", cases=CasesFoo) +def test_foo_cls(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_cls_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_cls, test_id_format='function') + if has_pytest_param: + assert list(results_dct) == [ + 'test_foo_cls[hello world]', + 'test_foo_cls[two_negative_ints]' + ] + else: + assert list(results_dct) == [ + 'test_foo_cls[hello world[0]-hello world[1]]', + 'test_foo_cls[two_negative_ints[0]-two_negative_ints[1]]' + ] + + +@parametrize_with_cases("a,b", cases=(CasesFoo, strange_ints, cases_doc, CasesFoo, '.test_doc_cases')) +def test_foo_cls_list(a, b): + assert isinstance(foo(a, b), tuple) + + +def test_foo_cls_list_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_cls_list, test_id_format='function') + ref_list = [ + # CasesFoo + 'test_foo_cls_list[hello world0]', + 'test_foo_cls_list[two_negative_ints0]', + # strange_ints + 'test_foo_cls_list[strange_ints]', + # cases_doc.py + 'test_foo_cls_list[hello]', + 'test_foo_cls_list[two_negative_ints1]', + 'test_foo_cls_list[two_negative_ints2]', + # CasesFoo + 'test_foo_cls_list[hello world1]', + 'test_foo_cls_list[two_negative_ints3]', + # test_doc_cases.py + 'test_foo_cls_list[two_positive_ints]', + 'test_foo_cls_list[two_negative_ints4]' + ] + if has_pytest_param: + assert list(results_dct) == ref_list + else: + assert len(results_dct) == len(ref_list) + + +@fixture +@parametrize_with_cases("a,b") +def c(a, b): + return a + b + + +def test_foo_parametrize_fixture(c): + assert isinstance(c, int) + + +def test_foo_parametrize_fixture_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_parametrize_fixture, test_id_format='function') + if has_pytest_param: + assert list(results_dct) == ['test_foo_parametrize_fixture[two_positive_ints]', + 'test_foo_parametrize_fixture[two_negative_ints]'] + else: + assert list(results_dct) == ['test_foo_parametrize_fixture[two_positive_ints[0]-two_positive_ints[1]]', + 'test_foo_parametrize_fixture[two_negative_ints[0]-two_negative_ints[1]]'] diff --git a/pytest_cases/tests/cases/doc/test_doc_cache.py b/pytest_cases/tests/cases/doc/test_doc_cache.py new file mode 100644 index 00000000..a02132cb --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_doc_cache.py @@ -0,0 +1,35 @@ +from pytest_cases import parametrize, parametrize_with_cases, fixture + + +already_read = set() + + +@parametrize(a=range(2)) +def case_dummy(a): + global already_read + if a in already_read: + raise ValueError() + else: + already_read.add(a) + return a + + +@fixture(scope='session') +@parametrize_with_cases("a", cases='.') +def cached_a(a): + return a + + +@parametrize(d=range(2)) +def test_caching(cached_a, d): + assert d < 2 + assert 0 <= cached_a <= 1 + + +def test_synthesis(module_results_dct): + assert list(module_results_dct) == [ + 'test_caching[dummy-a=0-d=0]', + 'test_caching[dummy-a=0-d=1]', + 'test_caching[dummy-a=1-d=0]', + 'test_caching[dummy-a=1-d=1]' + ] diff --git a/pytest_cases/tests/cases/doc/test_doc_cases.py b/pytest_cases/tests/cases/doc/test_doc_cases.py new file mode 100644 index 00000000..1d40d2f0 --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_doc_cases.py @@ -0,0 +1,8 @@ +def case_two_positive_ints(): + """ Inputs are two positive integers """ + return 1, 2 + + +def case_two_negative_ints(): + """ Inputs are two negative integers """ + return -1, -2 diff --git a/pytest_cases/tests/cases/doc/test_doc_filters_n_tags.py b/pytest_cases/tests/cases/doc/test_doc_filters_n_tags.py new file mode 100644 index 00000000..ddc46f2e --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_doc_filters_n_tags.py @@ -0,0 +1,65 @@ +from pytest_harvest import get_session_synthesis_dct + +from pytest_cases.common_pytest_marks import has_pytest_param +from pytest_cases import parametrize_with_cases, case, parametrize + + +def data_a(): + return 'a' + + +@parametrize("hello", [True, False]) +def data_b(hello): + return "hello" if hello else "world" + + +def case_c(): + return dict(name="hi i'm not used") + + +def user_bob(): + return "bob" + + +@parametrize_with_cases("data", cases='.', prefix="data_") +@parametrize_with_cases("user", cases='.', prefix="user_") +def test_with_data(data, user): + assert data in ('a', "hello", "world") + assert user == 'bob' + + +def test_with_data_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_with_data, test_id_format='function') + if has_pytest_param: + assert list(results_dct) == [ + 'test_with_data[bob-a]', + 'test_with_data[bob-b-True]', + 'test_with_data[bob-b-False]' + ] + else: + assert list(results_dct) == [ + 'test_with_data[a-bob]', + 'test_with_data[b-True-bob]', + 'test_with_data[b-False-bob]' + ] + + +class Foo: + def case_two_positive_ints(self): + return 1, 2 + + @case(tags='foo') + def case_one_positive_int(self): + return 1 + + +@parametrize_with_cases("a", cases=Foo, has_tag='foo') +def test_foo(a): + assert a > 0 + + +def test_foo_fixtures_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo, test_id_format='function') + assert list(results_dct) == [ + 'test_foo[one_positive_int]', + ] diff --git a/pytest_cases/tests/cases/doc/test_doc_filters_n_tags2.py b/pytest_cases/tests/cases/doc/test_doc_filters_n_tags2.py new file mode 100644 index 00000000..d7b46401 --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_doc_filters_n_tags2.py @@ -0,0 +1,74 @@ +from math import sqrt +import pytest + +from pytest_cases.common_pytest_marks import has_pytest_param +from pytest_cases import parametrize_with_cases + + +def case_int_success(): + return 1 + + +def case_negative_int_failure(): + # note that we decide to return the expected type of failure to check it + return -1, ValueError, "math domain error" + + +@parametrize_with_cases("data", cases='.', glob="*success") +def test_good_datasets(data): + assert sqrt(data) > 0 + + +@parametrize_with_cases("data, err_type, err_msg", cases='.', glob="*failure") +def test_bad_datasets(data, err_type, err_msg): + with pytest.raises(err_type, match=err_msg): + sqrt(data) + + +def test_synthesis(module_results_dct): + if has_pytest_param: + assert list(module_results_dct) == [ + 'test_good_datasets[int_success]', + 'test_bad_datasets[negative_int_failure]' + ] + else: + assert list(module_results_dct) == [ + 'test_good_datasets[int_success]', + 'test_bad_datasets[negative_int_failure[0]-negative_int_failure[1]-negative_int_failure[2]]' + ] + + +def create_filter(sub_str): + def my_filter(case_func): + return sub_str in case_func._pytestcase.id + return my_filter + + +@parametrize_with_cases("data", cases='.', filter=lambda case_func: "success" in case_func._pytestcase.id) +def test_good_datasets2(data): + assert sqrt(data) > 0 + + +@parametrize_with_cases("data, err_type, err_msg", cases='.', filter=create_filter("failure")) +def test_bad_datasets2(data, err_type, err_msg): + with pytest.raises(err_type, match=err_msg): + sqrt(data) + + +def test_synthesis2(module_results_dct): + if has_pytest_param: + assert list(module_results_dct) == [ + 'test_good_datasets[int_success]', + 'test_bad_datasets[negative_int_failure]', + 'test_synthesis', + 'test_good_datasets2[int_success]', + 'test_bad_datasets2[negative_int_failure]' + ] + else: + assert list(module_results_dct) == [ + 'test_good_datasets[int_success]', + 'test_bad_datasets[negative_int_failure[0]-negative_int_failure[1]-negative_int_failure[2]]', + 'test_synthesis', + 'test_good_datasets2[int_success]', + 'test_bad_datasets2[negative_int_failure[0]-negative_int_failure[1]-negative_int_failure[2]]' + ] diff --git a/pytest_cases/tests/cases/doc/test_fixtures.py b/pytest_cases/tests/cases/doc/test_fixtures.py new file mode 100644 index 00000000..65f1f256 --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_fixtures.py @@ -0,0 +1,32 @@ +from pytest_harvest import get_session_synthesis_dct + +from pytest_cases import parametrize_with_cases, fixture, parametrize + + +@fixture(scope='session') +def db(): + return {0: 'louise', 1: 'bob'} + + +def user_bob(db): + return db[1] + + +@parametrize(id=range(2)) +def user_from_db(db, id): + return db[id] + + +@parametrize_with_cases("a", cases='.', prefix='user_') +def test_users(a, db, request): + print("this is test %r" % request.node.nodeid) + assert a in db.values() + + +def test_users_synthesis(request, db): + results_dct = get_session_synthesis_dct(request, filter=test_users, test_id_format='function') + assert list(results_dct) == [ + 'test_users[a_is_bob]', + 'test_users[a_is_from_db-id=0]', + 'test_users[a_is_from_db-id=1]' + ] diff --git a/pytest_cases/tests/cases/doc/test_generators.py b/pytest_cases/tests/cases/doc/test_generators.py new file mode 100644 index 00000000..efc7d8d9 --- /dev/null +++ b/pytest_cases/tests/cases/doc/test_generators.py @@ -0,0 +1,70 @@ +import sys + +from pytest_harvest import get_session_synthesis_dct + +from pytest_cases import parametrize_with_cases, parametrize +from pytest_cases.common_pytest import has_pytest_param + +from ...utils import skip + + +class CasesFoo: + def case_hello(self): + return "hello world" + + @parametrize(who=('you', skip('there'))) + def case_simple_generator(self, who): + return "hello %s" % who + + +@parametrize_with_cases("msg", cases=CasesFoo) +def test_foo(msg): + assert isinstance(msg, str) and msg.startswith("hello") + + +def test_foo_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo, test_id_format='function') + assert list(results_dct) == [ + 'test_foo[hello]', + 'test_foo[simple_generator-who=you]', + # 'test_foo[simple_generator-who=there]' skipped + ] + + +class CasesFooMulti: + def case_hello(self): + return "hello world", len("hello world") + + @parametrize(who=(skip('you'), 'there'), **{'a,b': [(5, 5), (10, 10)]}) + def case_simple_generator(self, who, a, b): + assert a == b + return "hello %s" % who, len("hello %s" % who) + + +@parametrize_with_cases("msg,score", cases=CasesFooMulti) +def test_foo_multi(msg, score): + assert isinstance(msg, str) and msg.startswith("hello") + assert score == len(msg) + + +def test_foo_multi_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_foo_multi, test_id_format='function') + if sys.version_info >= (3, 6): + if has_pytest_param: + assert list(results_dct) == [ + 'test_foo_multi[hello]', + # 'test_foo_multi[simple_generator-who=you]', skipped + # 'test_foo_multi[simple_generator-who=you]', skipped + 'test_foo_multi[simple_generator-who=there-a=5-b=5]', + 'test_foo_multi[simple_generator-who=there-a=10-b=10]' + ] + else: + assert list(results_dct) == [ + 'test_foo_multi[hello[0]-hello[1]]', + # 'test_foo_multi[simple_generator-who=you]', skipped + # 'test_foo_multi[simple_generator-who=you]', skipped + 'test_foo_multi[simple_generator-who=there-a=5-b=5[0]-simple_generator-who=there-a=5-b=5[1]]', + 'test_foo_multi[simple_generator-who=there-a=10-b=10[0]-simple_generator-who=there-a=10-b=10[1]]' + ] + else: + assert len(results_dct) == 3 diff --git a/pytest_cases/tests/cases/advanced/__init__.py b/pytest_cases/tests/cases/legacy/__init__.py similarity index 100% rename from pytest_cases/tests/cases/advanced/__init__.py rename to pytest_cases/tests/cases/legacy/__init__.py diff --git a/pytest_cases/tests/cases/intermediate/__init__.py b/pytest_cases/tests/cases/legacy/advanced/__init__.py similarity index 100% rename from pytest_cases/tests/cases/intermediate/__init__.py rename to pytest_cases/tests/cases/legacy/advanced/__init__.py diff --git a/pytest_cases/tests/cases/advanced/test_memoize.py b/pytest_cases/tests/cases/legacy/advanced/test_memoize.py similarity index 96% rename from pytest_cases/tests/cases/advanced/test_memoize.py rename to pytest_cases/tests/cases/legacy/advanced/test_memoize.py index 6eace0d2..40c227e7 100644 --- a/pytest_cases/tests/cases/advanced/test_memoize.py +++ b/pytest_cases/tests/cases/legacy/advanced/test_memoize.py @@ -1,5 +1,5 @@ from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, case_tags -from pytest_cases.tests.cases.utils import nb_pytest_parameters, get_pytest_param +from ..utils import nb_pytest_parameters, get_pytest_param try: # python 3.2+ from functools import lru_cache diff --git a/pytest_cases/tests/cases/advanced/test_memoize_generators.py b/pytest_cases/tests/cases/legacy/advanced/test_memoize_generators.py similarity index 88% rename from pytest_cases/tests/cases/advanced/test_memoize_generators.py rename to pytest_cases/tests/cases/legacy/advanced/test_memoize_generators.py index 270adc0a..0ff2c0eb 100644 --- a/pytest_cases/tests/cases/advanced/test_memoize_generators.py +++ b/pytest_cases/tests/cases/legacy/advanced/test_memoize_generators.py @@ -1,5 +1,5 @@ -from pytest_cases import cases_data, THIS_MODULE, cases_generator, CaseDataGetter, get_all_cases -from pytest_cases.tests.cases.utils import nb_pytest_parameters, get_pytest_param +from pytest_cases import cases_data, THIS_MODULE, cases_generator, CaseDataGetter, get_all_cases_legacy +from ..utils import nb_pytest_parameters, get_pytest_param try: # python 3+: type hints from pytest_cases import CaseData @@ -40,7 +40,7 @@ def test_b(case_data # type: CaseDataGetter def test_assert_cases_are_here(): """Asserts that the 3 cases are generated""" import sys - cases = get_all_cases(module=sys.modules[case_gen.__module__]) + cases = get_all_cases_legacy(module=sys.modules[case_gen.__module__]) assert len(cases) == 3 diff --git a/pytest_cases/tests/cases/advanced/test_parameters.py b/pytest_cases/tests/cases/legacy/advanced/test_parameters.py similarity index 92% rename from pytest_cases/tests/cases/advanced/test_parameters.py rename to pytest_cases/tests/cases/legacy/advanced/test_parameters.py index faa2df27..963ddd10 100644 --- a/pytest_cases/tests/cases/advanced/test_parameters.py +++ b/pytest_cases/tests/cases/legacy/advanced/test_parameters.py @@ -1,7 +1,7 @@ import pytest -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test -from pytest_cases.tests.cases.utils import nb_pytest_parameters, get_pytest_param +from ..example_code import super_function_i_want_to_test +from ..utils import nb_pytest_parameters, get_pytest_param from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, cases_generator try: diff --git a/pytest_cases/tests/cases/advanced/test_stackoverflow.py b/pytest_cases/tests/cases/legacy/advanced/test_stackoverflow.py similarity index 97% rename from pytest_cases/tests/cases/advanced/test_stackoverflow.py rename to pytest_cases/tests/cases/legacy/advanced/test_stackoverflow.py index a89cbb1b..d8175833 100644 --- a/pytest_cases/tests/cases/advanced/test_stackoverflow.py +++ b/pytest_cases/tests/cases/legacy/advanced/test_stackoverflow.py @@ -1,4 +1,4 @@ -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test from pytest_cases import cases_data, CaseDataGetter try: from pytest_cases import CaseData diff --git a/pytest_cases/tests/cases/advanced/test_suite_parametrized_cases.py b/pytest_cases/tests/cases/legacy/advanced/test_suite_parametrized_cases.py similarity index 100% rename from pytest_cases/tests/cases/advanced/test_suite_parametrized_cases.py rename to pytest_cases/tests/cases/legacy/advanced/test_suite_parametrized_cases.py diff --git a/pytest_cases/tests/cases/advanced/test_suite_shared_dict_cases.py b/pytest_cases/tests/cases/legacy/advanced/test_suite_shared_dict_cases.py similarity index 100% rename from pytest_cases/tests/cases/advanced/test_suite_shared_dict_cases.py rename to pytest_cases/tests/cases/legacy/advanced/test_suite_shared_dict_cases.py diff --git a/pytest_cases/tests/cases/example_code.py b/pytest_cases/tests/cases/legacy/example_code.py similarity index 100% rename from pytest_cases/tests/cases/example_code.py rename to pytest_cases/tests/cases/legacy/example_code.py diff --git a/pytest_cases/tests/cases/simple/__init__.py b/pytest_cases/tests/cases/legacy/intermediate/__init__.py similarity index 100% rename from pytest_cases/tests/cases/simple/__init__.py rename to pytest_cases/tests/cases/legacy/intermediate/__init__.py diff --git a/pytest_cases/tests/cases/intermediate/test_filtering.py b/pytest_cases/tests/cases/legacy/intermediate/test_filtering.py similarity index 79% rename from pytest_cases/tests/cases/intermediate/test_filtering.py rename to pytest_cases/tests/cases/legacy/intermediate/test_filtering.py index b4da9571..33a42b32 100644 --- a/pytest_cases/tests/cases/intermediate/test_filtering.py +++ b/pytest_cases/tests/cases/legacy/intermediate/test_filtering.py @@ -1,4 +1,4 @@ -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, case_tags try: # python 3.5+ @@ -23,7 +23,12 @@ def case_a(): return ins, outs, None -@cases_data(module=THIS_MODULE, filter=lambda tags: 'a' in tags or 'b' in tags) +def my_tag_filter(case_func): + tags = case_func._pytestcase.tags + return 'a' in tags or 'b' in tags + + +@cases_data(module=THIS_MODULE, filter=my_tag_filter) def test_with_cases_a_or_b(case_data # type: CaseDataGetter ): @@ -36,7 +41,7 @@ def test_with_cases_a_or_b(case_data # type: CaseDataGetter assert outs == expected_o -@cases_data(module=THIS_MODULE, filter=lambda tags: 'a' in tags and 'b' in tags) +@cases_data(module=THIS_MODULE, filter=my_tag_filter) def test_with_cases_a_and_b(case_data # type: CaseDataGetter ): diff --git a/pytest_cases/tests/cases/intermediate/test_shared.py b/pytest_cases/tests/cases/legacy/intermediate/test_shared.py similarity index 92% rename from pytest_cases/tests/cases/intermediate/test_shared.py rename to pytest_cases/tests/cases/legacy/intermediate/test_shared.py index 918e57fb..7a76a8d8 100644 --- a/pytest_cases/tests/cases/intermediate/test_shared.py +++ b/pytest_cases/tests/cases/legacy/intermediate/test_shared.py @@ -1,6 +1,6 @@ from pytest_cases import CaseDataGetter, cases_data -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test, super_function_i_want_to_test2 +from ..example_code import super_function_i_want_to_test, super_function_i_want_to_test2 # the file with case functions from . import test_shared_cases diff --git a/pytest_cases/tests/cases/intermediate/test_shared_cases.py b/pytest_cases/tests/cases/legacy/intermediate/test_shared_cases.py similarity index 85% rename from pytest_cases/tests/cases/intermediate/test_shared_cases.py rename to pytest_cases/tests/cases/legacy/intermediate/test_shared_cases.py index 115d0e0d..e57092f9 100644 --- a/pytest_cases/tests/cases/intermediate/test_shared_cases.py +++ b/pytest_cases/tests/cases/legacy/intermediate/test_shared_cases.py @@ -1,4 +1,4 @@ -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test, super_function_i_want_to_test2 +from ..example_code import super_function_i_want_to_test, super_function_i_want_to_test2 from pytest_cases import test_target, case_tags try: # python 3.5+ diff --git a/pytest_cases/tests/cases/intermediate/test_two_modules.py b/pytest_cases/tests/cases/legacy/intermediate/test_two_modules.py similarity index 100% rename from pytest_cases/tests/cases/intermediate/test_two_modules.py rename to pytest_cases/tests/cases/legacy/intermediate/test_two_modules.py diff --git a/pytest_cases/tests/cases/legacy/simple/__init__.py b/pytest_cases/tests/cases/legacy/simple/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/pytest_cases/tests/cases/simple/test_fixtures.py b/pytest_cases/tests/cases/legacy/simple/test_fixtures.py similarity index 92% rename from pytest_cases/tests/cases/simple/test_fixtures.py rename to pytest_cases/tests/cases/legacy/simple/test_fixtures.py index 9b188c48..f5247a7c 100644 --- a/pytest_cases/tests/cases/simple/test_fixtures.py +++ b/pytest_cases/tests/cases/legacy/simple/test_fixtures.py @@ -4,7 +4,7 @@ from pytest_cases import unfold_expected_err, cases_fixture, cases_data, pytest_fixture_plus -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test @cases_fixture(module=test_main_cases) @@ -28,7 +28,7 @@ def test_with_cases_decorated_legacy(my_case_fixture_legacy): else: # **** Error test **** # First see what we need to assert - err_type, err_inst, err_checker = unfold_expected_err(expected_e) + err_type, err_ptrn, err_inst, err_checker = unfold_expected_err(expected_e) # Run with exception capture and type check with pytest.raises(err_type) as err_info: @@ -75,7 +75,7 @@ def test_with_cases_decorated(my_case_fixture): else: # **** Error test **** # First see what we need to assert - err_type, err_inst, err_checker = unfold_expected_err(expected_e) + err_type, err_ptrn, err_inst, err_checker = unfold_expected_err(expected_e) # Run with exception capture and type check with pytest.raises(err_type) as err_info: diff --git a/pytest_cases/tests/cases/simple/test_generators.py b/pytest_cases/tests/cases/legacy/simple/test_generators.py similarity index 95% rename from pytest_cases/tests/cases/simple/test_generators.py rename to pytest_cases/tests/cases/legacy/simple/test_generators.py index 72e108bc..24578e0b 100644 --- a/pytest_cases/tests/cases/simple/test_generators.py +++ b/pytest_cases/tests/cases/legacy/simple/test_generators.py @@ -1,4 +1,4 @@ -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE, cases_generator try: diff --git a/pytest_cases/tests/cases/simple/test_main.py b/pytest_cases/tests/cases/legacy/simple/test_main.py similarity index 84% rename from pytest_cases/tests/cases/simple/test_main.py rename to pytest_cases/tests/cases/legacy/simple/test_main.py index 611494cf..f6e76269 100644 --- a/pytest_cases/tests/cases/simple/test_main.py +++ b/pytest_cases/tests/cases/legacy/simple/test_main.py @@ -1,7 +1,7 @@ import pytest -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test -from pytest_cases import cases_data, CaseDataGetter, unfold_expected_err, get_all_cases, get_pytest_parametrize_args +from pytest_cases import cases_data, CaseDataGetter, unfold_expected_err, get_all_cases_legacy, get_pytest_parametrize_args_legacy from ..simple import test_main_cases @@ -23,7 +23,7 @@ def test_with_cases_decorated(case_data # type: CaseDataGetter else: # **** Error test **** # First see what we need to assert - err_type, err_inst, err_checker = unfold_expected_err(expected_e) + err_type, err_ptrn, err_inst, err_checker = unfold_expected_err(expected_e) # Run with exception capture and type check with pytest.raises(err_type) as err_info: @@ -39,9 +39,9 @@ def test_with_cases_decorated(case_data # type: CaseDataGetter # ----------------- Advanced: Manual way: ------------- -cases = get_all_cases(module=test_main_cases) +cases = get_all_cases_legacy(module=test_main_cases) # apply the pytest marks -marked_cases, cases_ids = get_pytest_parametrize_args(cases) +marked_cases, cases_ids = get_pytest_parametrize_args_legacy(cases) @pytest.mark.parametrize('case_data', marked_cases, ids=cases_ids) @@ -61,7 +61,7 @@ def test_with_cases_manual(case_data # type: CaseDataGetter else: # **** Error test **** # First see what we need to assert - err_type, err_inst, err_checker = unfold_expected_err(expected_e) + err_type, err_ptrn, err_inst, err_checker = unfold_expected_err(expected_e) # Run with exception capture and type check with pytest.raises(err_type) as err_info: diff --git a/pytest_cases/tests/cases/simple/test_main_cases.py b/pytest_cases/tests/cases/legacy/simple/test_main_cases.py similarity index 95% rename from pytest_cases/tests/cases/simple/test_main_cases.py rename to pytest_cases/tests/cases/legacy/simple/test_main_cases.py index aea0fc2b..d16f6eb3 100644 --- a/pytest_cases/tests/cases/simple/test_main_cases.py +++ b/pytest_cases/tests/cases/legacy/simple/test_main_cases.py @@ -12,7 +12,7 @@ pass -from pytest_cases.tests.cases.example_code import InfiniteInput +from ..example_code import InfiniteInput def case_simple(): diff --git a/pytest_cases/tests/cases/legacy/simple/test_pytest_marks.py b/pytest_cases/tests/cases/legacy/simple/test_pytest_marks.py new file mode 100644 index 00000000..9edad63d --- /dev/null +++ b/pytest_cases/tests/cases/legacy/simple/test_pytest_marks.py @@ -0,0 +1,26 @@ +import pytest + +from pytest_cases.case_parametrizer_legacy import get_pytest_marks_on_function, make_marked_parameter_value + + +def test_get_pytest_marks(): + """ + Tests that we are able to correctly retrieve the marks on case_func + :return: + """ + skip_mark = pytest.mark.skipif(True, reason="why") + + @skip_mark + def case_func(): + pass + + # extract the marks from a case function + marks = get_pytest_marks_on_function(case_func, as_decorators=True) + + # check that the mark is the same than a manually made one + assert len(marks) == 1 + assert str(marks[0]) == str(skip_mark) + + # transform a parameter into a marked parameter + dummy_case = (1, 2, 3) + marked_param = make_marked_parameter_value((dummy_case,), marks=marks) diff --git a/pytest_cases/tests/cases/simple/test_same_file.py b/pytest_cases/tests/cases/legacy/simple/test_same_file.py similarity index 90% rename from pytest_cases/tests/cases/simple/test_same_file.py rename to pytest_cases/tests/cases/legacy/simple/test_same_file.py index a44b4b88..0c7bdb7c 100644 --- a/pytest_cases/tests/cases/simple/test_same_file.py +++ b/pytest_cases/tests/cases/legacy/simple/test_same_file.py @@ -1,4 +1,4 @@ -from pytest_cases.tests.cases.example_code import super_function_i_want_to_test +from ..example_code import super_function_i_want_to_test from pytest_cases import cases_data, CaseDataGetter, THIS_MODULE try: # python 3.5+ diff --git a/pytest_cases/tests/cases/simple/test_so.py b/pytest_cases/tests/cases/legacy/simple/test_so.py similarity index 100% rename from pytest_cases/tests/cases/simple/test_so.py rename to pytest_cases/tests/cases/legacy/simple/test_so.py diff --git a/pytest_cases/tests/cases/simple/test_so2.py b/pytest_cases/tests/cases/legacy/simple/test_so2.py similarity index 100% rename from pytest_cases/tests/cases/simple/test_so2.py rename to pytest_cases/tests/cases/legacy/simple/test_so2.py diff --git a/pytest_cases/tests/cases/utils.py b/pytest_cases/tests/cases/legacy/utils.py similarity index 100% rename from pytest_cases/tests/cases/utils.py rename to pytest_cases/tests/cases/legacy/utils.py diff --git a/pytest_cases/tests/cases/simple/test_pytest_marks.py b/pytest_cases/tests/cases/simple/test_pytest_marks.py deleted file mode 100644 index 2c7f0ac9..00000000 --- a/pytest_cases/tests/cases/simple/test_pytest_marks.py +++ /dev/null @@ -1,27 +0,0 @@ -import pytest - -from pytest_cases.common_pytest import transform_marks_into_decorators -from pytest_cases.case_parametrizer import get_pytest_marks_on_function, make_marked_parameter_value - - -@pytest.mark.skipif(True, reason="why") -def case_func(): - pass - - -def test_get_pytest_marks(): - """ - Tests that we are able to correctly retrieve the marks on case_func - :return: - """ - # extract the marks from a case function - marks = get_pytest_marks_on_function(case_func) - # transform them into decorators - marks = transform_marks_into_decorators(marks) - # check that the mark is the same than a manually made one - assert len(marks) == 1 - assert str(marks[0]) == str(pytest.mark.skipif(True, reason="why")) - - # transform a parameter into a marked parameter - dummy_case = (1, 2, 3) - marked_param = make_marked_parameter_value(dummy_case, marks=marks) diff --git a/pytest_cases/tests/pytest_extension/others/__init__.py b/pytest_cases/tests/pytest_extension/others/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/pytest_cases/tests/pytest_extension/others/test_assert_exception.py b/pytest_cases/tests/pytest_extension/others/test_assert_exception.py new file mode 100644 index 00000000..7b0be51e --- /dev/null +++ b/pytest_cases/tests/pytest_extension/others/test_assert_exception.py @@ -0,0 +1,43 @@ +import pytest + +from pytest_cases import assert_exception + + +def test_assert_exception(): + # good type + with assert_exception(ValueError): + raise ValueError() + + # good type - inherited + class MyErr(ValueError): + pass + with assert_exception(ValueError): + raise MyErr() + + # no exception + with pytest.raises(AssertionError, match="DID NOT RAISE any BaseException"): + with assert_exception(ValueError): + pass + + # wrong type + with pytest.raises(AssertionError, match=r"Caught exception TypeError\(\) " + "is not an instance of expected type.*ValueError"): + with assert_exception(ValueError): + raise TypeError() + + # good repr pattern + with assert_exception(r"ValueError\('hello'[,]?\)"): + raise ValueError("hello") + + # good instance - equality check + class MyExc(Exception): + def __eq__(self, other): + return vars(self) == vars(other) + + with assert_exception(MyExc('hello')): + raise MyExc("hello") + + # good equality but wrong type + with pytest.raises(AssertionError, match=r"is not an instance of expected type.*MyExc"): + with assert_exception(MyExc('hello')): + raise Exception("hello") diff --git a/pytest_cases/tests/pytest_extension/parametrize_plus/test_basics_misc.py b/pytest_cases/tests/pytest_extension/parametrize_plus/test_basics_misc.py new file mode 100644 index 00000000..3a69dec7 --- /dev/null +++ b/pytest_cases/tests/pytest_extension/parametrize_plus/test_basics_misc.py @@ -0,0 +1,193 @@ +import sys +from distutils.version import LooseVersion + +import pytest + +from pytest_cases import parametrize_plus, lazy_value +from pytest_harvest import get_session_synthesis_dct + +from pytest_cases.common_pytest import has_pytest_param, cart_product_pytest, get_marked_parameter_values, \ + extract_parameterset_info +from pytest_cases.common_pytest_lazy_values import is_lazy +from pytest_cases.fixture_parametrize_plus import _get_argnames_argvalues +from ...utils import skip + + +def test_cart_product_pytest(): + + # simple + names_lst, values = cart_product_pytest(('a', 'b'), ([True], [1, 2])) + assert names_lst == ['a', 'b'] + assert values == [(True, 1), (True, 2)] + + # multi + names_lst, values = cart_product_pytest(('a,b', 'c'), ([(True, 1)], [1, 2])) + assert names_lst == ['a', 'b', 'c'] + assert values == [(True, 1, 1), (True, 1, 2)] + + # marks + names_lst, values = cart_product_pytest(('a,b', 'c'), ([(True, 1)], [skip(1), 2])) + assert names_lst == ['a', 'b', 'c'] + assert get_marked_parameter_values(values[0]) == (True, 1, 1) + assert values[1] == (True, 1, 2) + + # lazy values + def get_tuple(): + return 3, 4 + names_lst, values = cart_product_pytest(('a', 'b,c'), ([True], [lazy_value(get_tuple, marks=skip), (1, 2)])) + assert names_lst == ['a', 'b', 'c'] + assert values[0][0] is True + assert is_lazy(values[0][1]) + assert is_lazy(values[0][2]) + assert values[0][1].get_id() == 'get_tuple[0]' + assert values[0][2].get_id() == 'get_tuple[1]' + assert values[1] == (True, 1, 2) + + +def test_argname_error(): + with pytest.raises(ValueError, match="parameter 'a' not found in test function signature"): + @parametrize_plus("a", [True]) + def test_foo(b): + pass + + +PY36 = sys.version_info >= (3, 6) + + +@pytest.mark.parametrize("tuple_around_single", [False, True]) +def test_get_argnames_argvalues(tuple_around_single): + + # legacy way + # -- 1 argname + argnames, argvalues = _get_argnames_argvalues('a', (True, 1.25)) + assert argnames == ['a'] + assert argvalues == [True, 1.25] + # -- several argnames + argnames, argvalues = _get_argnames_argvalues('a,b', ((True, 1.25), (True, 0))) + assert argnames == ['a', 'b'] + assert argvalues == [(True, 1.25), (True, 0)] + + # **args only + # -- 1 argname + argnames, argvalues = _get_argnames_argvalues(b=[1.25, 0]) + assert argnames == ['b'] + assert argvalues == [1.25, 0] + # -- several argnames + argnames, argvalues = _get_argnames_argvalues(a=[True], b=[1.25, 0]) + if PY36: + assert argnames == ['a', 'b'] + else: + assert set(argnames) == {'a', 'b'} + if argnames[-1] == 'b': + assert argvalues == [(True, 1.25), (True, 0)] + else: + assert argvalues == [(1.25, True), (0, True)] + + # --dict version + # -- 1 argname + argnames, argvalues = _get_argnames_argvalues(**{'b': [1.25, 0]}) + assert argnames == ['b'] + assert argvalues == [1.25, 0] + # -- several argnames at once + argnames, argvalues = _get_argnames_argvalues(**{'a,b': ((True, 1.25), (True, 0))}) + assert argnames == ['a', 'b'] + assert argvalues == [(True, 1.25), (True, 0)] + # -- several argnames in two entries + argnames, argvalues = _get_argnames_argvalues(**{'a,b': ((True, 1.25), (True, 0)), 'c': [-1, 2]}) + if not PY36: + # order is lost + assert set(argnames) == {'a', 'b', 'c'} + else: + assert argnames == ['a', 'b', 'c'] + if argnames[-1] == 'c': + assert argvalues == [(True, 1.25, -1), (True, 1.25, 2), (True, 0, -1), (True, 0, 2)] + else: + # for python < 3.6 + assert argvalues == [(-1, True, 1.25), (-1, True, 0), (2, True, 1.25), (2, True, 0)] + + # a mark on any of them + argnames, argvalues = _get_argnames_argvalues(**{'a,b': (skip(True, 1.25), (True, 0)), 'c': [-1, 2]}) + if has_pytest_param: + assert argvalues[0].id is None + assert argvalues[0].marks[0].name == 'skip' + assert argvalues[0].values == (True, 1.25, -1) if argnames[-1] == 'c' else (-1, True, 1.25) + + # hybrid + # -- several argnames in two entries + argnames, argvalues = _get_argnames_argvalues('c', (-1, 2), **{'a,b': ((True, 1.25), (True, 0))}) + assert argnames == ['c', 'a', 'b'] + assert argvalues == [(-1, True, 1.25), (-1, True, 0), (2, True, 1.25), (2, True, 0)] + # -- several argnames in two entries with marks + argnames, argvalues = _get_argnames_argvalues('c,d', ((True, -1), skip('hey', 2)), **{'a,b': ((True, 1.25), (True, 0))}) + assert argnames == ['c', 'd', 'a', 'b'] + custom_pids, p_marks, p_values = extract_parameterset_info(argnames, argvalues, check_nb=True) + assert all(p is None for p in custom_pids) + assert p_values == [(True, -1, True, 1.25), (True, -1, True, 0), ('hey', 2, True, 1.25), ('hey', 2, True, 0)] + assert p_marks[0:2] == [None, None] + if has_pytest_param: + assert len(p_marks[2]) == 1 + assert p_marks[2][0].name == 'skip' + assert len(p_marks[3]) == 1 + assert p_marks[3][0].name == 'skip' + + +def format_me(**kwargs): + if 'a' in kwargs: + return "a={a},b={b:3d}".format(**kwargs) + else: + return "{d}yes".format(**kwargs) + + +@parametrize_plus("a,b", [(True, -1), (False, 3)], idgen=format_me) +@parametrize_plus("c", [2.1, 0.], idgen="c{c:.1f}") +@parametrize_plus("d", [10], idgen=format_me) +def test_idgen1(a, b, c, d): + pass + + +def test_idgen1_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_idgen1, test_id_format='function') + if sys.version_info >= (3, 6): + if LooseVersion(pytest.__version__) >= LooseVersion('3.0.0'): + assert list(results_dct) == [ + 'test_idgen1[10yes-c2.1-a=True,b= -1]', + 'test_idgen1[10yes-c2.1-a=False,b= 3]', + 'test_idgen1[10yes-c0.0-a=True,b= -1]', + 'test_idgen1[10yes-c0.0-a=False,b= 3]' + ] + else: + # the order seems not guaranteed or at least quite different in pytest 2 + assert len(results_dct) == 4 + else: + assert len(results_dct) == 4 + + +@parametrize_plus(idgen="a={a},b={b:.1f} and {c:4d}", **{'a,b': ((True, 1.25), (True, 0.)), 'c': [-1, 2]}) +def test_alt_usage1(a, b, c): + pass + + +def test_alt_usage1_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_alt_usage1, test_id_format='function') + if sys.version_info > (3, 6): + assert list(results_dct) == [ + 'test_alt_usage1[a=True,b=1.2 and -1]', + 'test_alt_usage1[a=True,b=1.2 and 2]', + 'test_alt_usage1[a=True,b=0.0 and -1]', + 'test_alt_usage1[a=True,b=0.0 and 2]' + ] + else: + assert len(results_dct) == 4 + + +@parametrize_plus(idgen="b{b:.1}", **{'b': (1.25, 0.)}) +def test_alt_usage2(b): + pass + + +def test_alt_usage2_synthesis(request): + results_dct = get_session_synthesis_dct(request, filter=test_alt_usage2, test_id_format='function') + assert list(results_dct) == [ + 'test_alt_usage2[b1e+00]', + 'test_alt_usage2[b0e+00]' + ] diff --git a/pytest_cases/tests/pytest_extension/parametrize_plus/test_getcallspecs.py b/pytest_cases/tests/pytest_extension/parametrize_plus/test_getcallspecs.py new file mode 100644 index 00000000..ca24397b --- /dev/null +++ b/pytest_cases/tests/pytest_extension/parametrize_plus/test_getcallspecs.py @@ -0,0 +1,51 @@ +import pytest + +from pytest_cases import parametrize +from pytest_cases.common_pytest import get_callspecs, has_pytest_param + +if not has_pytest_param: + @pytest.mark.parametrize('new_style', [False, True]) + def test_getcallspecs(new_style): + if new_style: + parametrizer = parametrize(a=[1, pytest.mark.skipif(True)(('12',))], idgen="a={a}") + else: + parametrizer = parametrize('a', [1, pytest.mark.skipif(True)(('12',))], ids=['oh', 'my']) + + @parametrizer + def test_foo(a): + pass + + calls = get_callspecs(test_foo) + + assert len(calls) == 2 + assert calls[0].funcargs == dict(a=1) + assert calls[0].id == 'a=1' if new_style else 'oh' + assert calls[0].marks == [] + + assert calls[1].funcargs == dict(a='12') + ref_id = "a=12" if new_style else 'my' + assert calls[1].id == ref_id + assert calls[1].marks[0].name == 'skipif' + +else: + @pytest.mark.parametrize('new_style', [False, True]) + def test_getcallspecs(new_style): + if new_style: + parametrizer = parametrize(a=[1, pytest.param('12', marks=pytest.mark.skip)], idgen="a={a}") + else: + parametrizer = parametrize('a', [1, pytest.param('12', marks=pytest.mark.skip, id='hey')], ids=['oh', 'my']) + + @parametrizer + def test_foo(a): + pass + + calls = get_callspecs(test_foo) + + assert len(calls) == 2 + assert calls[0].funcargs == dict(a=1) + assert calls[0].id == 'a=1' if new_style else 'oh' + assert calls[0].marks == [] + + assert calls[1].funcargs == dict(a='12') + assert calls[1].id == 'a=12' if new_style else 'hey' + assert calls[1].marks[0].name == 'skip' diff --git a/pytest_cases/tests/pytest_extension/parametrize_plus/test_lazy_value.py b/pytest_cases/tests/pytest_extension/parametrize_plus/test_lazy_value.py index de4e4144..b71fd0eb 100644 --- a/pytest_cases/tests/pytest_extension/parametrize_plus/test_lazy_value.py +++ b/pytest_cases/tests/pytest_extension/parametrize_plus/test_lazy_value.py @@ -38,7 +38,7 @@ def test_foo_multi(a, b): def test_synthesis(module_results_dct): assert list(module_results_dct) == ['test_foo_single[val]', 'test_foo_single[A]', - 'test_foo_multi[a0-b0]', # normal: lazy_value is used for the whole tuple + 'test_foo_multi[valtuple[0]-valtuple[1]]', # normal: lazy_value is used for the whole tuple # AND we cannot use pytest.param in this version # AND there are no fixtures so we pass to normal @parametrize 'test_foo_multi[1-val]'] diff --git a/pytest_cases/tests/utils.py b/pytest_cases/tests/utils.py new file mode 100644 index 00000000..33225e2b --- /dev/null +++ b/pytest_cases/tests/utils.py @@ -0,0 +1,10 @@ +import pytest + +from pytest_cases.common_pytest import has_pytest_param + +if has_pytest_param: + def skip(*argvals): + return pytest.param(*argvals, marks=pytest.mark.skip) +else: + def skip(*argvals): + return pytest.mark.skip(argvals)