Deprecation warnings introduced in #10850 can't be filtered by pytest

[a-recknagel]

A number of deprecation warnings were introduced with #10850 that can not be suppressed by pytest's warnings-capture options and will clutter the output.
I ran into the issue because sklearn still uses the deprecated functions, so I can't fix the issue in my own code. My repro-example will feature sklearn, but it shouldn't be necessary to isolate the issue.

Reproducing code example:

environment: python3.7.0, numpy==1.15.1, pytest==3.7.4, scikit-learn==0.19.2, scipy==1.1.0

Files: test.py

def test_umath_tests_warning():
    from sklearn.ensemble import weight_boosting  # <-- imports numpy.core.umath_tests

Run from command line:

$ python -m pytest test.py

Error message:

=============================== warnings summary ===============================
test.py::test_umath_tests_warning
  /home/user/env/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
    from numpy.core.umath_tests import inner1d

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 0.33 seconds =====================

Possible fix:

Changing the stacklevel of all warnings from #10850 to 1 (i.e. the default value) solves the problem on my end.

[charris]

OK, confirmed. The stacklevel=2 warnings can be suppressed in pytest.ini, but not in _pytesttester. The latter works by adding them to the pytest arguments.

[charris]

Changing my mind, the -W options work for me, but only if pytest.ini is missing. The reason is that currently the pytest.ini file has precedence in pytest.

[a-recknagel]

Short rundown of the scenarios I tried:

Setup

environment dependencies: same as in the top post

test.py:

def test_warning_stacklevel_1():
  from warnings import warn
  warn('foo', DeprecationWarning)

def test_warning_stacklevel_2():
  from warnings import warn
  warn('bar', DeprecationWarning, stacklevel=2)

def test_umath_warning():
  from sklearn.ensemble import weight_boosting

Warning message when running $ python -m pytest test.py with no arguments and no pytest.ini:

=============================== warnings summary ===============================
/home/user/tmp/test.py:3: DeprecationWarning: foo
  warn('foo', DeprecationWarning)

/home/user/env/lib/python3.7/site-packages/_pytest/python.py:197: DeprecationWarning: bar
  testfunction(**testargs)

/home/user/env/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 3 passed, 3 warnings in 0.36 seconds =====================

1) - no pytest.ini, suppress on command line

run command: $ python -m pytest test.py -W ignore:DeprecationWarning
warnings:

=============================== warnings summary ===============================
/home/user/env/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 3 passed, 1 warnings in 0.35 seconds =====================

I'm pretty stumped by this result to be honest. I expected test_warning_stacklevel_2 and test_umath_warning to behave the same way.

2) - pytest.ini with explicit DeprecationWarning suppression

pytest.ini:

[pytest]
filterwarnings =
    ignore::DeprecationWarning

run command: $ python -m pytest test.py
warnings:

=============================== warnings summary ===============================
/home/user/env/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 3 passed, 1 warnings in 0.34 seconds =====================

Same result as in 1).

3) - pytest.ini with global no-warnings setting

pytest.ini:

[pytest]
addopts = -p no:warnings

run command: $ python -m pytest test.py
warnings: No warnings!

But using this setting is discouraged by the pytest docu unless you absolutely have to use it.

4) - decorating test functions to drop warnings, based on substring matching

test.py:

@pytest.mark.filterwarnings('ignore:foo')
def test_warning_stacklevel_1():
  from warnings import warn
  warn('foo', DeprecationWarning)

@pytest.mark.filterwarnings('ignore:bar')
def test_warning_stacklevel_2():
  from warnings import warn
  warn('bar', DeprecationWarning, stacklevel=2)

@pytest.mark.filterwarnings('ignore:numpy.core.umath_tests')
def test_umath_warning():
  from sklearn.ensemble import weight_boosting

run command: $ python -m pytest test.py
warnings:

=============================== warnings summary ===============================
/home/user/env/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================== 3 passed, 1 warnings in 0.34 seconds =====================

Same as 1) and 2).

---

I would say that 1), 2), and 4) would be valid workarounds. I took your comments to mean that these options do work for you, can you confirm that? If yes, something spooky might be afoot regarding environments. That being said, my gitlab CI, which runs on alpine, shows the same behavior as these scenarios here. So it's at least somewhat consistent.

[charris]

We just ran into a problem with pytest 3.8.0, released yesterday, and in the process of resolving the issue discovered that the pytest.ini file was read in after the command line, thus had precedence. There is a new pytest release out today that reverses that.

[charris]

Might want to escape the . s in the last example.

r'ignore:numpy\.core\.umath_tests'

although I don't think it will make a difference.

[charris]

I think it possible that sklearn is messing up the warnings stack, easy to do. We ended up rolling our own suppression filter and, IIRC, there was some comment about pytest not having it quite right at some time in the past, but possibly fixed (or not).

EDIT: @seberg Thoughts?

[charris]

IIRC, the problem is that some (import) warnings are raised during pytest collection time and so it is not possible to filter them with a runtime filter, which I suspect the warnings filter is.

[seberg]

@charris yeah, I don't think pytest tries to be as clean as numpy, but since this is python 3.7, I also do not think there should be any issue (plus the big issues we had is too little warning, not too many, basically why the pytest global filters are discouraged probably).

I am a bit stumped as well... I mean all the warnings give the right line of code, etc. so the stacklevel seems completely fine (for a minute I thought maybe the level is just somewhere strange). Maybe we should ask pytest people?

[seberg]

Tried to reproduce it by also having a module that calls a module that gives the warning and even __init__.py imports, nothing at first, but then it gets stranger :)...

I build the same import through numpy, and then decorated the numpy function with the ignore filter of another one, and the warning disappeared, if I ignore the correct one, it doesn't disappear, believe it or not, this weird thing runs without any warnings:

import pytest

@pytest.mark.filterwarnings('ignore:.*blob')  # warning actually includes "blob"
def test_warnings_import_stacklevel2_():
    from warnmod import getswarning

@pytest.mark.filterwarnings('ignore:asdfasdf')  # I assume asdfasdf is nonsense, but I the decorator
def test_warnings_import_stacklevel2_numpy():
    from warnmod import numpywarn

#@pytest.mark.filterwarnings('ignore:.*numpy')
def test_umath_warning():
  from sklearn.ensemble import weight_boosting

The rest of the setup is warnmod, its __init__.py imports from . import getswarning (I doubt that matters). And then there is getswarning which imports giveswarning which gives a warning at stacklevel 2. warnmod.numpywarn just imports umath_tests.

[a-recknagel]

Do you know who'd be the right person to ping? Otherwise I'd just drop a fyi mail at pytest-dev@python.org

[charris]

I think the problem arises because the pytest.ini file currently overrides the command line. Pytest will change that in a future release.

[a-recknagel]

Alright, I'll try to check every now and then whether this issue can be closed, then. From what you said, I understand that a stacklevel of 2 is the right way to handle this in numpy and shouldn't change, right?

On another note, I found a different workaround for this issue that for some reason hadn't occurred to me before:

import warnings

def test_umath_warning():
  with warnings.catch_warnings():
    warnings.filterwarnings('ignore')
    from sklearn.ensemble import weight_boosting

This is what I'm currently doing.

[charris]

@a-recknagel Does latest pytest (3.10, released 20 hrs ago) fix this for you?