New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest 'slow' not a registered marker with `--strict` flag on other projects #16680

Closed
kmodry opened this Issue Jun 12, 2017 · 14 comments

Comments

Projects
None yet
4 participants
@kmodry

kmodry commented Jun 12, 2017

Create a file test_pandas.py with import pandas

$ echo 'import pandas' > test_pandas.py

run pytest with --strict flag

$ pytest --strict test_pandas.py
================================================ test session starts ================================================
platform darwin -- Python 3.6.1, pytest-3.1.0, py-1.4.33, pluggy-0.4.0
rootdir: /private/tmp, inifile:
plugins: xdist-1.15.0, repeat-0.4.1, cov-2.4.0
collected 0 items / 1 errors

====================================================== ERRORS =======================================================
__________________________________________ ERROR collecting test_pandas.py __________________________________________
test_pandas.py:1: in <module>
    import pandas
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/__init__.py:60: in <module>
    import pandas.testing
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/testing.py:7: in <module>
    from pandas.util.testing import (
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/util/testing.py:55: in <module>
    slow = pytest.mark.slow
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================== 1 error in 1.10 seconds ==============================================

(original post below)


Importing panda make my pytest tests fail because of one marker "slow" is required in
/usr/local/lib/python2.7/dist-packages/pandas/util/testing.py:55: in
slow = pytest.mark.slow

I don't use slow marker and have to add it to run my test even though it's just for panda

complete stack trace :

tests/_lib/createfullteststoresem.py:54: in createFullTestStoreSem
from oxlib.sem.history import get_ad_product_history
oxlib/sem/history.py:5: in
import pandas as pd
/usr/local/lib/python2.7/dist-packages/pandas/init.py:60: in
import pandas.testing
/usr/local/lib/python2.7/dist-packages/pandas/testing.py:7: in
from pandas.util.testing import (
/usr/local/lib/python2.7/dist-packages/pandas/util/testing.py:55: in
slow = pytest.mark.slow
/usr/local/lib/python2.7/dist-packages/_pytest/mark.py:184: in getattr
self._check(name)


@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Jun 12, 2017

Contributor

you would have to show more information.

Contributor

jreback commented Jun 12, 2017

you would have to show more information.

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Jun 13, 2017

Contributor

@kmodry can you provide more information? What version of pandas is installed? Is your project publicly available?

Contributor

TomAugspurger commented Jun 13, 2017

@kmodry can you provide more information? What version of pandas is installed? Is your project publicly available?

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 27, 2017

Hi, I just ran into the exact same problem while making a pull request on github for pyjson_tricks . This pullrequest caused travis to run the continues integration test, whose results you can see at: https://travis-ci.org/mverleg/pyjson_tricks/jobs/246885590 . According to that webpage the error occured with pandas version 0.20.2. An older test of the same project with pandas 0.19.2 at https://travis-ci.org/mverleg/pyjson_tricks/jobs/222351028 did not produce these problems.

koffie commented Jun 27, 2017

Hi, I just ran into the exact same problem while making a pull request on github for pyjson_tricks . This pullrequest caused travis to run the continues integration test, whose results you can see at: https://travis-ci.org/mverleg/pyjson_tricks/jobs/246885590 . According to that webpage the error occured with pandas version 0.20.2. An older test of the same project with pandas 0.19.2 at https://travis-ci.org/mverleg/pyjson_tricks/jobs/222351028 did not produce these problems.

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 27, 2017

To recreate the problem: clone https://github.com/koffie/pyjson_tricks (commit 4f5e734b70ffcb...) and run the tests as described in: https://github.com/koffie/pyjson_tricks/blob/master/docs/index.rst . I.e. just run either tox or detox from the root directory. Changing tox.ini to require pandas==0.19.2 doesn't produce the error, while requiring pandas==0.20.0rc1 or any higher version triggers the error.

koffie commented Jun 27, 2017

To recreate the problem: clone https://github.com/koffie/pyjson_tricks (commit 4f5e734b70ffcb...) and run the tests as described in: https://github.com/koffie/pyjson_tricks/blob/master/docs/index.rst . I.e. just run either tox or detox from the root directory. Changing tox.ini to require pandas==0.19.2 doesn't produce the error, while requiring pandas==0.20.0rc1 or any higher version triggers the error.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Jun 28, 2017

Contributor

@koffie still not entirely sure what this has to do with pandas. pls show a reproducible example with pandas code itself.

I would guess you are picking up the pandas conftest

Contributor

jreback commented Jun 28, 2017

@koffie still not entirely sure what this has to do with pandas. pls show a reproducible example with pandas code itself.

I would guess you are picking up the pandas conftest

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Jun 28, 2017

Contributor

Hmm, looking through @koffie's example, pandas fails to import with

(Pdb) import pandas.testing
*** AttributeError: module 'pandas' has no attribute 'plotting'

I'm not sure how that would cause the test error. Even install matplotlib in that env causes the same error.

Contributor

TomAugspurger commented Jun 28, 2017

Hmm, looking through @koffie's example, pandas fails to import with

(Pdb) import pandas.testing
*** AttributeError: module 'pandas' has no attribute 'plotting'

I'm not sure how that would cause the test error. Even install matplotlib in that env causes the same error.

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 28, 2017

Yes we are indeed picking up parts of pandas conftest and this is happening because there is the statement: import pandas.testing in the /pandas/__init__.py file. This means that you cannot import pandas without also implicitly importing the test code for pandas.

Now the error occurs because we like to test our code with the --strict option of pytest. And according to the documentation

Note
It is recommended to explicitly register markers so that:

  • there is one place in your test suite defining your markers
  • asking for existing markers via pytest --markers gives good output
  • typos in function markers are treated as an error if you use the --strict option. Future versions of pytest are probably going to start treating non-registered markers as errors at some point.

Now we don't want to be forced to register all markers that are used in pandas.testing . So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas? This would solve all our problems and make it much easier to run tests in other projects that depend on pandas, but only want to run their own tests, and not of pandas.

koffie commented Jun 28, 2017

Yes we are indeed picking up parts of pandas conftest and this is happening because there is the statement: import pandas.testing in the /pandas/__init__.py file. This means that you cannot import pandas without also implicitly importing the test code for pandas.

Now the error occurs because we like to test our code with the --strict option of pytest. And according to the documentation

Note
It is recommended to explicitly register markers so that:

  • there is one place in your test suite defining your markers
  • asking for existing markers via pytest --markers gives good output
  • typos in function markers are treated as an error if you use the --strict option. Future versions of pytest are probably going to start treating non-registered markers as errors at some point.

Now we don't want to be forced to register all markers that are used in pandas.testing . So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas? This would solve all our problems and make it much easier to run tests in other projects that depend on pandas, but only want to run their own tests, and not of pandas.

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 28, 2017

Another solution if you really want to keep exposing all your testing functions is to write a pandas plugin for pytest as explained on https://docs.pytest.org/en/latest/writing_plugins.html#writing-plugins and register all your custom markers as explained on https://docs.pytest.org/en/latest/example/markers.html#adding-a-custom-marker-from-a-plugin .

koffie commented Jun 28, 2017

Another solution if you really want to keep exposing all your testing functions is to write a pandas plugin for pytest as explained on https://docs.pytest.org/en/latest/writing_plugins.html#writing-plugins and register all your custom markers as explained on https://docs.pytest.org/en/latest/example/markers.html#adding-a-custom-marker-from-a-plugin .

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 28, 2017

By the way, a minimal way to reproduce the problem is the following:
create a file test_pandas.py containing only the line of code import pandas and then running pytest --strict test_pandas.py. It would be nice to be able to import pandas from a file on which you want to run pytest --strict.

koffie commented Jun 28, 2017

By the way, a minimal way to reproduce the problem is the following:
create a file test_pandas.py containing only the line of code import pandas and then running pytest --strict test_pandas.py. It would be nice to be able to import pandas from a file on which you want to run pytest --strict.

@TomAugspurger TomAugspurger changed the title from pytest marker "slow" to pytest 'slow' not a registered marker with `--strict` flag on other projects Jun 28, 2017

@TomAugspurger TomAugspurger added this to the 0.20.3 milestone Jun 28, 2017

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Jun 28, 2017

Contributor

@koffie thanks for the minimal example. Added to the original post.

So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas?

Don't think so. That's part of the public API, and we provide pandas.test() to run the unit tests.

I think you're suggesting to write a plugin is the way to go.

Contributor

TomAugspurger commented Jun 28, 2017

@koffie thanks for the minimal example. Added to the original post.

So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas?

Don't think so. That's part of the public API, and we provide pandas.test() to run the unit tests.

I think you're suggesting to write a plugin is the way to go.

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 28, 2017

@TomAugspurger One could move the import pandas.testing statement inside the pandas.test() function. But I agree that a plugin is the better way to go.

koffie commented Jun 28, 2017

@TomAugspurger One could move the import pandas.testing statement inside the pandas.test() function. But I agree that a plugin is the better way to go.

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 28, 2017

By the way just running pytest --strict pandas also gives the same error. So maybe just registering the mark slow in setup.cfg would solve the problem. I don't know how fancy the marker registration framework of pytest is.

pytest --strict pandas
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 2.7.10, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/mderickx/code/pandas, inifile: setup.cfg
collected 11472 items / 2 errors / 4 skipped 

================================================================================================== ERRORS ==================================================================================================
______________________________________________________________________________ ERROR collecting pandas/tests/test_sorting.py _______________________________________________________________________________
pandas/tests/test_sorting.py:19: in <module>
    class TestSorting(object):
pandas/tests/test_sorting.py:21: in TestSorting
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
________________________________________________________________________ ERROR collecting pandas/tests/groupby/test_value_counts.py ________________________________________________________________________
pandas/tests/groupby/test_value_counts.py:10: in <module>
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 4 skipped, 2 error in 5.83 seconds ====================================================================================
bt-nac-h149:pandas mderickx$

koffie commented Jun 28, 2017

By the way just running pytest --strict pandas also gives the same error. So maybe just registering the mark slow in setup.cfg would solve the problem. I don't know how fancy the marker registration framework of pytest is.

pytest --strict pandas
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 2.7.10, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/mderickx/code/pandas, inifile: setup.cfg
collected 11472 items / 2 errors / 4 skipped 

================================================================================================== ERRORS ==================================================================================================
______________________________________________________________________________ ERROR collecting pandas/tests/test_sorting.py _______________________________________________________________________________
pandas/tests/test_sorting.py:19: in <module>
    class TestSorting(object):
pandas/tests/test_sorting.py:21: in TestSorting
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
________________________________________________________________________ ERROR collecting pandas/tests/groupby/test_value_counts.py ________________________________________________________________________
pandas/tests/groupby/test_value_counts.py:10: in <module>
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 4 skipped, 2 error in 5.83 seconds ====================================================================================
bt-nac-h149:pandas mderickx$
@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Jun 28, 2017

Contributor

Ha, I think that does it, at least for the simple example. Here's the diff if anyone wants to submit a PR before I have a chance tomorrow

diff --git a/pandas/util/testing.py b/pandas/util/testing.py
index 17e09b38b..d6ba95613 100644
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -50,13 +50,6 @@ from pandas import (bdate_range, CategoricalIndex, Categorical, IntervalIndex,
 
 from pandas._libs import testing as _testing
 from pandas.io.common import urlopen
-try:
-    import pytest
-    slow = pytest.mark.slow
-except ImportError:
-    # Should be ok to just ignore. If you actually need
-    # slow then you'll hit an import error long before getting here.
-    pass
 
 
 N = 30
diff --git a/setup.cfg b/setup.cfg
index 8b32f0f62..e0d5f274a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -27,3 +27,4 @@ split_penalty_logical_operator = 30
 testpaths = pandas
 markers =
     single: mark a test as single cpu only
+    slow: mark a test as slow

Thanks for the help @koffie

Contributor

TomAugspurger commented Jun 28, 2017

Ha, I think that does it, at least for the simple example. Here's the diff if anyone wants to submit a PR before I have a chance tomorrow

diff --git a/pandas/util/testing.py b/pandas/util/testing.py
index 17e09b38b..d6ba95613 100644
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -50,13 +50,6 @@ from pandas import (bdate_range, CategoricalIndex, Categorical, IntervalIndex,
 
 from pandas._libs import testing as _testing
 from pandas.io.common import urlopen
-try:
-    import pytest
-    slow = pytest.mark.slow
-except ImportError:
-    # Should be ok to just ignore. If you actually need
-    # slow then you'll hit an import error long before getting here.
-    pass
 
 
 N = 30
diff --git a/setup.cfg b/setup.cfg
index 8b32f0f62..e0d5f274a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -27,3 +27,4 @@ split_penalty_logical_operator = 30
 testpaths = pandas
 markers =
     single: mark a test as single cpu only
+    slow: mark a test as slow

Thanks for the help @koffie

@koffie

This comment has been minimized.

Show comment
Hide comment
@koffie

koffie Jun 30, 2017

Hi,

Thanks for the pull request. I just tested this with 16797 applied. And it did fix the problem with running pytest --strict pandas since it reads the setup.cfg file. But if one installs this version of pandas with pip, and then tries pytest --strict test_pandas.py one still gets the same error about the marker slow not being registered. Apparently one really needs to write a plugin if one wants the markers to be registered even when running tests in a directory not containing your setup.cfg file.

koffie commented Jun 30, 2017

Hi,

Thanks for the pull request. I just tested this with 16797 applied. And it did fix the problem with running pytest --strict pandas since it reads the setup.cfg file. But if one installs this version of pandas with pip, and then tries pytest --strict test_pandas.py one still gets the same error about the marker slow not being registered. Apparently one really needs to write a plugin if one wants the markers to be registered even when running tests in a directory not containing your setup.cfg file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment