New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

supplying filterwarnings overrides default filters #4013

Closed
jaraco opened this Issue Sep 21, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@jaraco
Contributor

jaraco commented Sep 21, 2018

I'm really happy about pytest 3.8 and the work in #2908.

And now that I'm seeing warnings again, I realize how blissful ignorance is.

So I've gone through one project and fixed all of the deprecation warnings for the project under test (there are still many coming from other projects). In order to assert that the project no longer has any warnings, I added this to pytest.ini:

filterwarnings = 
    error:::mylib[.*]

Now tests pass, but all of the deprecation warnings (from the dependencies) are suppressed. That wasn't what I'd intended by setting the project to error on warnings.

I don't think that's the right behavior. Instead, pytest should configure the filters for DeprecationWarnings and PendingDeprecationWarnings and then allow the user to override those filters but not suppress the defaults with any value.

If the project must suppress the defaults, the docs should at least describe what the defaults are so they can be copied into the configuration wishing to define other filters. I had to traverse to the source to determine what the defaults are:

filterwarnings =
	always::DeprecationWarning
	always::PendingDeprecationWarning

Is there any reason not to always configure the defaults and allow overrides?

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Sep 21, 2018

Hi @jaraco,

Now tests pass, but all of the deprecation warnings (from the dependencies) are suppressed. That wasn't what I'd intended by setting the project to error on warnings.

Indeed. When we introduced the warnings plugin a few versions back we had some serious regressions, so this time around I decided to be really conservative to minimize disruption.

The unittest module only changes the warnings filters to diplay deprecations if sys.warnoptions is set (usually by PYTHONWARNINGS). I decided to go with the same route with our warnings plugin:

with warnings.catch_warnings(record=True) as log:
filters_configured = args or inifilters or sys.warnoptions
for arg in args:
warnings._setoption(arg)
for arg in inifilters:
_setoption(warnings, arg)
if item is not None:
for mark in item.iter_markers(name="filterwarnings"):
for arg in mark.args:
_setoption(warnings, arg)
filters_configured = True
if not filters_configured:
# if user is not explicitly configuring warning filters, show deprecation warnings by default (#2908)
warnings.filterwarnings("always", category=DeprecationWarning)
warnings.filterwarnings("always", category=PendingDeprecationWarning)

So basically we only install the filters if the user did not specify any filter configuration themselves, either using pytest's mechanisms or python's own PYTHONWARNINGS.

But I agree with you that this is not ideal, because as you point out any customization effectively silences deprecation warnings again.

I believe that for 3.9 we can install the deprecation filters before all other filters, right after catch_warnings() in the previous snippet. This shows deprecations by default, but any other filter that comes after it will take precedence.

@nicoddemus nicoddemus self-assigned this Sep 21, 2018

@nicoddemus nicoddemus added this to the 3.9 milestone Sep 21, 2018

@jaraco

This comment has been minimized.

Contributor

jaraco commented Sep 22, 2018

Wonderful. Yes, I completely understand the desire to be cautious... and I concur with your assessment and plan. Most importantly, thank you for all the time and effort you put into this project.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Sep 22, 2018

Thanks @jaraco! 😊

PR opened at #4019 if you want to provide feedback. 👍

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Sep 22, 2018

@nicoddemus nicoddemus closed this Sep 22, 2018

peterbe added a commit to mozilla-services/tecken that referenced this issue Oct 25, 2018

Update pytest to 3.9.2 (#1264)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.8.2** to **3.9.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.9.2
   ```
   =========================

Bug Fixes
---------

- `2909 &lt;https://github.com/pytest-dev/pytest/issues/2909&gt;`_: Improve error message when a recursive dependency between fixtures is detected.


- `3340 &lt;https://github.com/pytest-dev/pytest/issues/3340&gt;`_: Fix logging messages not shown in hooks ``pytest_sessionstart()`` and ``pytest_sessionfinish()``.


- `3533 &lt;https://github.com/pytest-dev/pytest/issues/3533&gt;`_: Fix unescaped XML raw objects in JUnit report for skipped tests


- `3691 &lt;https://github.com/pytest-dev/pytest/issues/3691&gt;`_: Python 2: safely format warning message about passing unicode strings to ``warnings.warn``, which may cause
  surprising ``MemoryError`` exception when monkey patching ``warnings.warn`` itself.


- `4026 &lt;https://github.com/pytest-dev/pytest/issues/4026&gt;`_: Improve error message when it is not possible to determine a function&#39;s signature.


- `4177 &lt;https://github.com/pytest-dev/pytest/issues/4177&gt;`_: Pin ``setuptools&gt;=40.0`` to support ``py_modules`` in ``setup.cfg``


- `4179 &lt;https://github.com/pytest-dev/pytest/issues/4179&gt;`_: Restore the tmpdir behaviour of symlinking the current test run.


- `4192 &lt;https://github.com/pytest-dev/pytest/issues/4192&gt;`_: Fix filename reported by ``warnings.warn`` when using ``recwarn`` under python2.
   ```
   
  
  
   ### 3.9.1
   ```
   =========================

Features
--------

- `4159 &lt;https://github.com/pytest-dev/pytest/issues/4159&gt;`_: For test-suites containing test classes, the information about the subclassed
  module is now output only if a higher verbosity level is specified (at least
  &quot;-vv&quot;).
   ```
   
  
  
   ### 3.9.0
   ```
   =========================================================================

Deprecations
------------

- `3616 &lt;https://github.com/pytest-dev/pytest/issues/3616&gt;`_: The following accesses have been documented as deprecated for years, but are now actually emitting deprecation warnings.

  * Access of ``Module``, ``Function``, ``Class``, ``Instance``, ``File`` and ``Item`` through ``Node`` instances. Now
    users will this warning::

          usage of Function.Module is deprecated, please use pytest.Module instead

    Users should just ``import pytest`` and access those objects using the ``pytest`` module.

  * ``request.cached_setup``, this was the precursor of the setup/teardown mechanism available to fixtures. You can
    consult `funcarg comparison section in the docs &lt;https://docs.pytest.org/en/latest/funcarg_compare.html&gt;`_.

  * Using objects named ``&quot;Class&quot;`` as a way to customize the type of nodes that are collected in ``Collector``
    subclasses has been deprecated. Users instead should use ``pytest_collect_make_item`` to customize node types during
    collection.

    This issue should affect only advanced plugins who create new collection types, so if you see this warning
    message please contact the authors so they can change the code.

  * The warning that produces the message below has changed to ``RemovedInPytest4Warning``::

          getfuncargvalue is deprecated, use getfixturevalue


- `3988 &lt;https://github.com/pytest-dev/pytest/issues/3988&gt;`_: Add a Deprecation warning for pytest.ensuretemp as it was deprecated since a while.



Features
--------

- `2293 &lt;https://github.com/pytest-dev/pytest/issues/2293&gt;`_: Improve usage errors messages by hiding internal details which can be distracting and noisy.

  This has the side effect that some error conditions that previously raised generic errors (such as
  ``ValueError`` for unregistered marks) are now raising ``Failed`` exceptions.


- `3332 &lt;https://github.com/pytest-dev/pytest/issues/3332&gt;`_: Improve the error displayed when a ``conftest.py`` file could not be imported.

  In order to implement this, a new ``chain`` parameter was added to ``ExceptionInfo.getrepr``
  to show or hide chained tracebacks in Python 3 (defaults to ``True``).


- `3849 &lt;https://github.com/pytest-dev/pytest/issues/3849&gt;`_: Add ``empty_parameter_set_mark=fail_at_collect`` ini option for raising an exception when parametrize collects an empty set.


- `3964 &lt;https://github.com/pytest-dev/pytest/issues/3964&gt;`_: Log messages generated in the collection phase are shown when
  live-logging is enabled and/or when they are logged to a file.


- `3985 &lt;https://github.com/pytest-dev/pytest/issues/3985&gt;`_: Introduce ``tmp_path`` as a fixture providing a Path object.


- `4013 &lt;https://github.com/pytest-dev/pytest/issues/4013&gt;`_: Deprecation warnings are now shown even if you customize the warnings filters yourself. In the previous version
  any customization would override pytest&#39;s filters and deprecation warnings would fall back to being hidden by default.


- `4073 &lt;https://github.com/pytest-dev/pytest/issues/4073&gt;`_: Allow specification of timeout for ``Testdir.runpytest_subprocess()`` and ``Testdir.run()``.


- `4098 &lt;https://github.com/pytest-dev/pytest/issues/4098&gt;`_: Add returncode argument to pytest.exit() to exit pytest with a specific return code.


- `4102 &lt;https://github.com/pytest-dev/pytest/issues/4102&gt;`_: Reimplement ``pytest.deprecated_call`` using ``pytest.warns`` so it supports the ``match=&#39;...&#39;`` keyword argument.

  This has the side effect that ``pytest.deprecated_call`` now raises ``pytest.fail.Exception`` instead
  of ``AssertionError``.


- `4149 &lt;https://github.com/pytest-dev/pytest/issues/4149&gt;`_: Require setuptools&gt;=30.3 and move most of the metadata to ``setup.cfg``.



Bug Fixes
---------

- `2535 &lt;https://github.com/pytest-dev/pytest/issues/2535&gt;`_: Improve error message when test functions of ``unittest.TestCase`` subclasses use a parametrized fixture.


- `3057 &lt;https://github.com/pytest-dev/pytest/issues/3057&gt;`_: ``request.fixturenames`` now correctly returns the name of fixtures created by ``request.getfixturevalue()``.


- `3946 &lt;https://github.com/pytest-dev/pytest/issues/3946&gt;`_: Warning filters passed as command line options using ``-W`` now take precedence over filters defined in ``ini``
  configuration files.


- `4066 &lt;https://github.com/pytest-dev/pytest/issues/4066&gt;`_: Fix source reindenting by using ``textwrap.dedent`` directly.


- `4102 &lt;https://github.com/pytest-dev/pytest/issues/4102&gt;`_: ``pytest.warn`` will capture previously-warned warnings in Python 2. Previously they were never raised.


- `4108 &lt;https://github.com/pytest-dev/pytest/issues/4108&gt;`_: Resolve symbolic links for args.

  This fixes running ``pytest tests/test_foo.py::test_bar``, where ``tests``
  is a symlink to ``project/app/tests``:
  previously ``project/app/conftest.py`` would be ignored for fixtures then.


- `4132 &lt;https://github.com/pytest-dev/pytest/issues/4132&gt;`_: Fix duplicate printing of internal errors when using ``--pdb``.


- `4135 &lt;https://github.com/pytest-dev/pytest/issues/4135&gt;`_: pathlib based tmpdir cleanup now correctly handles symlinks in the folder.


- `4152 &lt;https://github.com/pytest-dev/pytest/issues/4152&gt;`_: Display the filename when encountering ``SyntaxWarning``.



Improved Documentation
----------------------

- `3713 &lt;https://github.com/pytest-dev/pytest/issues/3713&gt;`_: Update usefixtures documentation to clarify that it can&#39;t be used with fixture functions.


- `4058 &lt;https://github.com/pytest-dev/pytest/issues/4058&gt;`_: Update fixture documentation to specify that a fixture can be invoked twice in the scope it&#39;s defined for.


- `4064 &lt;https://github.com/pytest-dev/pytest/issues/4064&gt;`_: According to unittest.rst, setUpModule and tearDownModule were not implemented, but it turns out they are. So updated the documentation for unittest.


- `4151 &lt;https://github.com/pytest-dev/pytest/issues/4151&gt;`_: Add tempir testing example to CONTRIBUTING.rst guide



Trivial/Internal Changes
------------------------

- `2293 &lt;https://github.com/pytest-dev/pytest/issues/2293&gt;`_: The internal ``MarkerError`` exception has been removed.


- `3988 &lt;https://github.com/pytest-dev/pytest/issues/3988&gt;`_: Port the implementation of tmpdir to pathlib.


- `4063 &lt;https://github.com/pytest-dev/pytest/issues/4063&gt;`_: Exclude 0.00 second entries from ``--duration`` output unless ``-vv`` is passed on the command-line.


- `4093 &lt;https://github.com/pytest-dev/pytest/issues/4093&gt;`_: Fixed formatting of string literals in internal tests.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: https://docs.pytest.org/en/latest/
</details>

bors bot added a commit to rehandalal/therapist that referenced this issue Oct 30, 2018

Merge #43
43: Update pytest to 3.9.3 r=rehandalal a=pyup-bot


This PR updates [pytest](https://pypi.org/project/pytest) from **3.8.2** to **3.9.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.9.3
   ```
   =========================

Bug Fixes
---------

- `4174 &lt;https://github.com/pytest-dev/pytest/issues/4174&gt;`_: Fix &quot;ValueError: Plugin already registered&quot; with conftest plugins via symlink.


- `4181 &lt;https://github.com/pytest-dev/pytest/issues/4181&gt;`_: Handle race condition between creation and deletion of temporary folders.


- `4221 &lt;https://github.com/pytest-dev/pytest/issues/4221&gt;`_: Fix bug where the warning summary at the end of the test session was not showing the test where the warning was originated.


- `4243 &lt;https://github.com/pytest-dev/pytest/issues/4243&gt;`_: Fix regression when ``stacklevel`` for warnings was passed as positional argument on python2.



Improved Documentation
----------------------

- `3851 &lt;https://github.com/pytest-dev/pytest/issues/3851&gt;`_: Add reference to ``empty_parameter_set_mark`` ini option in documentation of ``pytest.mark.parametrize``



Trivial/Internal Changes
------------------------

- `4028 &lt;https://github.com/pytest-dev/pytest/issues/4028&gt;`_: Revert patching of ``sys.breakpointhook`` since it appears to do nothing.


- `4233 &lt;https://github.com/pytest-dev/pytest/issues/4233&gt;`_: Apply an import sorter (``reorder-python-imports``) to the codebase.


- `4248 &lt;https://github.com/pytest-dev/pytest/issues/4248&gt;`_: Remove use of unnecessary compat shim, six.binary_type
   ```
   
  
  
   ### 3.9.2
   ```
   =========================

Bug Fixes
---------

- `2909 &lt;https://github.com/pytest-dev/pytest/issues/2909&gt;`_: Improve error message when a recursive dependency between fixtures is detected.


- `3340 &lt;https://github.com/pytest-dev/pytest/issues/3340&gt;`_: Fix logging messages not shown in hooks ``pytest_sessionstart()`` and ``pytest_sessionfinish()``.


- `3533 &lt;https://github.com/pytest-dev/pytest/issues/3533&gt;`_: Fix unescaped XML raw objects in JUnit report for skipped tests


- `3691 &lt;https://github.com/pytest-dev/pytest/issues/3691&gt;`_: Python 2: safely format warning message about passing unicode strings to ``warnings.warn``, which may cause
  surprising ``MemoryError`` exception when monkey patching ``warnings.warn`` itself.


- `4026 &lt;https://github.com/pytest-dev/pytest/issues/4026&gt;`_: Improve error message when it is not possible to determine a function&#39;s signature.


- `4177 &lt;https://github.com/pytest-dev/pytest/issues/4177&gt;`_: Pin ``setuptools&gt;=40.0`` to support ``py_modules`` in ``setup.cfg``


- `4179 &lt;https://github.com/pytest-dev/pytest/issues/4179&gt;`_: Restore the tmpdir behaviour of symlinking the current test run.


- `4192 &lt;https://github.com/pytest-dev/pytest/issues/4192&gt;`_: Fix filename reported by ``warnings.warn`` when using ``recwarn`` under python2.
   ```
   
  
  
   ### 3.9.1
   ```
   =========================

Features
--------

- `4159 &lt;https://github.com/pytest-dev/pytest/issues/4159&gt;`_: For test-suites containing test classes, the information about the subclassed
  module is now output only if a higher verbosity level is specified (at least
  &quot;-vv&quot;).
   ```
   
  
  
   ### 3.9.0
   ```
   =========================================================================

Deprecations
------------

- `3616 &lt;https://github.com/pytest-dev/pytest/issues/3616&gt;`_: The following accesses have been documented as deprecated for years, but are now actually emitting deprecation warnings.

  * Access of ``Module``, ``Function``, ``Class``, ``Instance``, ``File`` and ``Item`` through ``Node`` instances. Now
    users will this warning::

          usage of Function.Module is deprecated, please use pytest.Module instead

    Users should just ``import pytest`` and access those objects using the ``pytest`` module.

  * ``request.cached_setup``, this was the precursor of the setup/teardown mechanism available to fixtures. You can
    consult `funcarg comparison section in the docs &lt;https://docs.pytest.org/en/latest/funcarg_compare.html&gt;`_.

  * Using objects named ``&quot;Class&quot;`` as a way to customize the type of nodes that are collected in ``Collector``
    subclasses has been deprecated. Users instead should use ``pytest_collect_make_item`` to customize node types during
    collection.

    This issue should affect only advanced plugins who create new collection types, so if you see this warning
    message please contact the authors so they can change the code.

  * The warning that produces the message below has changed to ``RemovedInPytest4Warning``::

          getfuncargvalue is deprecated, use getfixturevalue


- `3988 &lt;https://github.com/pytest-dev/pytest/issues/3988&gt;`_: Add a Deprecation warning for pytest.ensuretemp as it was deprecated since a while.



Features
--------

- `2293 &lt;https://github.com/pytest-dev/pytest/issues/2293&gt;`_: Improve usage errors messages by hiding internal details which can be distracting and noisy.

  This has the side effect that some error conditions that previously raised generic errors (such as
  ``ValueError`` for unregistered marks) are now raising ``Failed`` exceptions.


- `3332 &lt;https://github.com/pytest-dev/pytest/issues/3332&gt;`_: Improve the error displayed when a ``conftest.py`` file could not be imported.

  In order to implement this, a new ``chain`` parameter was added to ``ExceptionInfo.getrepr``
  to show or hide chained tracebacks in Python 3 (defaults to ``True``).


- `3849 &lt;https://github.com/pytest-dev/pytest/issues/3849&gt;`_: Add ``empty_parameter_set_mark=fail_at_collect`` ini option for raising an exception when parametrize collects an empty set.


- `3964 &lt;https://github.com/pytest-dev/pytest/issues/3964&gt;`_: Log messages generated in the collection phase are shown when
  live-logging is enabled and/or when they are logged to a file.


- `3985 &lt;https://github.com/pytest-dev/pytest/issues/3985&gt;`_: Introduce ``tmp_path`` as a fixture providing a Path object.


- `4013 &lt;https://github.com/pytest-dev/pytest/issues/4013&gt;`_: Deprecation warnings are now shown even if you customize the warnings filters yourself. In the previous version
  any customization would override pytest&#39;s filters and deprecation warnings would fall back to being hidden by default.


- `4073 &lt;https://github.com/pytest-dev/pytest/issues/4073&gt;`_: Allow specification of timeout for ``Testdir.runpytest_subprocess()`` and ``Testdir.run()``.


- `4098 &lt;https://github.com/pytest-dev/pytest/issues/4098&gt;`_: Add returncode argument to pytest.exit() to exit pytest with a specific return code.


- `4102 &lt;https://github.com/pytest-dev/pytest/issues/4102&gt;`_: Reimplement ``pytest.deprecated_call`` using ``pytest.warns`` so it supports the ``match=&#39;...&#39;`` keyword argument.

  This has the side effect that ``pytest.deprecated_call`` now raises ``pytest.fail.Exception`` instead
  of ``AssertionError``.


- `4149 &lt;https://github.com/pytest-dev/pytest/issues/4149&gt;`_: Require setuptools&gt;=30.3 and move most of the metadata to ``setup.cfg``.



Bug Fixes
---------

- `2535 &lt;https://github.com/pytest-dev/pytest/issues/2535&gt;`_: Improve error message when test functions of ``unittest.TestCase`` subclasses use a parametrized fixture.


- `3057 &lt;https://github.com/pytest-dev/pytest/issues/3057&gt;`_: ``request.fixturenames`` now correctly returns the name of fixtures created by ``request.getfixturevalue()``.


- `3946 &lt;https://github.com/pytest-dev/pytest/issues/3946&gt;`_: Warning filters passed as command line options using ``-W`` now take precedence over filters defined in ``ini``
  configuration files.


- `4066 &lt;https://github.com/pytest-dev/pytest/issues/4066&gt;`_: Fix source reindenting by using ``textwrap.dedent`` directly.


- `4102 &lt;https://github.com/pytest-dev/pytest/issues/4102&gt;`_: ``pytest.warn`` will capture previously-warned warnings in Python 2. Previously they were never raised.


- `4108 &lt;https://github.com/pytest-dev/pytest/issues/4108&gt;`_: Resolve symbolic links for args.

  This fixes running ``pytest tests/test_foo.py::test_bar``, where ``tests``
  is a symlink to ``project/app/tests``:
  previously ``project/app/conftest.py`` would be ignored for fixtures then.


- `4132 &lt;https://github.com/pytest-dev/pytest/issues/4132&gt;`_: Fix duplicate printing of internal errors when using ``--pdb``.


- `4135 &lt;https://github.com/pytest-dev/pytest/issues/4135&gt;`_: pathlib based tmpdir cleanup now correctly handles symlinks in the folder.


- `4152 &lt;https://github.com/pytest-dev/pytest/issues/4152&gt;`_: Display the filename when encountering ``SyntaxWarning``.



Improved Documentation
----------------------

- `3713 &lt;https://github.com/pytest-dev/pytest/issues/3713&gt;`_: Update usefixtures documentation to clarify that it can&#39;t be used with fixture functions.


- `4058 &lt;https://github.com/pytest-dev/pytest/issues/4058&gt;`_: Update fixture documentation to specify that a fixture can be invoked twice in the scope it&#39;s defined for.


- `4064 &lt;https://github.com/pytest-dev/pytest/issues/4064&gt;`_: According to unittest.rst, setUpModule and tearDownModule were not implemented, but it turns out they are. So updated the documentation for unittest.


- `4151 &lt;https://github.com/pytest-dev/pytest/issues/4151&gt;`_: Add tempir testing example to CONTRIBUTING.rst guide



Trivial/Internal Changes
------------------------

- `2293 &lt;https://github.com/pytest-dev/pytest/issues/2293&gt;`_: The internal ``MarkerError`` exception has been removed.


- `3988 &lt;https://github.com/pytest-dev/pytest/issues/3988&gt;`_: Port the implementation of tmpdir to pathlib.


- `4063 &lt;https://github.com/pytest-dev/pytest/issues/4063&gt;`_: Exclude 0.00 second entries from ``--duration`` output unless ``-vv`` is passed on the command-line.


- `4093 &lt;https://github.com/pytest-dev/pytest/issues/4093&gt;`_: Fixed formatting of string literals in internal tests.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: https://docs.pytest.org/en/latest/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment