TST: Use pytest #13856

TomAugspurger · 2016-07-31T13:32:45Z

closes TST: Start using py.test #13097
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

Just a WIP for now, will need to iterate on the CI stuff to ensure that's all good.

What can I provide to make this easier to review? Refactoring tests is scary.
I'll give verbose outputs of nose on master and pytest on this branch.
junit.xml output too. Anything else?

Here are some notes I took along the way

Kinds of Changes

Nosetests Generators to fixtures

See TestPickle, TestMsgPack and this SO post. We switched to using parametrized fixtures (yay) and dependency injection instead of setUp + a for loop inside the test.

Different Discover Rules

See the sql tests. py.test discovers _Test* class (if they inherit from TestCase?), while unittest does not.
Solution: move TestCase inheritance down a level

Assert Rewriting

See e.g. pandas/src/testing.py

solution: rewrite assert cond, msg as if not cond; raise AssertionError(msg)
looking into just excluding these files... They aren't actual tests, just test helpers.
CI work:
- updating nose to pytest

jreback · 2016-07-31T13:45:00Z

factoring asserts can be in a later PR

pytest will work with nose tools (though we mostly use our own tm.* ones)

sinhrks · 2016-07-31T23:37:23Z

pandas/io/tests/test_packers.py

    """
-    How to add msgpack tests:
-
-    1. Install pandas version intended to output the msgpack.


Shouldn't this procedure remain?

max-sixty · 2016-08-01T16:53:36Z

Thanks a lot @TomAugspurger !

One note on something above:

py.test discovers _Test* class (if they inherit from TestCase?), while unittest does not.

My understanding is that the only way to use pytest fixtures within a class is to not inherit from TestCase. This can make it a bit more difficult to gradually transition, but you can use autouse=True on existing setUp methods, and the inheritance then isn't necessary.

And pytest discovers tests in Test* named classes, not *Test, unless they inherit from TestCase.

Let me know if that's unclear.

TomAugspurger · 2016-08-01T17:02:48Z

factoring asserts can be in a later PR

Agreed, the only asserts I've manually rewritten were in src/testing.pyx to get the tests to pass. I believe this is due to pytest-dev/pytest#645, which has been fixed and will be included in pytest 3.0 which is coming out soonish. They were targeting Europython, but pushed it back.
I'll revert those changes and wait for pytest3 I think.

@MaximilianR thanks for the heads about autouse=True, I'll check that out. My initial goal was to have the test-suite runnable by either nose or pytest runners. Then followup PRs could make use of pytest features.
Doesn't look like I'll be able to achieve that though due to the nose generator tests.

TomAugspurger · 2016-08-20T01:02:38Z

pytest 3 is out now. Do we want to make the switch just after 0.19 is released?

codecov-io · 2016-11-03T12:33:28Z

Codecov Report

❗ No coverage uploaded for pull request base (master@3d6fcdc). Click here to learn what that means.

@@            Coverage Diff            @@
##             master   #13856   +/-   ##
=========================================
  Coverage          ?   87.88%           
=========================================
  Files             ?      142           
  Lines             ?    51106           
  Branches          ?        0           
=========================================
  Hits              ?    44913           
  Misses            ?     6193           
  Partials          ?        0

Impacted Files	Coverage Δ
pandas/init.py	`91.42% <100%> (ø)`
pandas/util/testing.py	`82.79% <100%> (ø)`
pandas/util/_tester.py	`46.15% <46.15%> (ø)`
pandas/conftest.py	`83.33% <83.33%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d6fcdc...59e2be9. Read the comment docs.

jreback · 2016-11-11T12:55:56Z

.travis.yml

@@ -44,7 +44,7 @@ matrix:
    - python: 2.7
      env:
        - JOB_NAME: "27_slow_nnet_LOCALE"
-        - NOSE_ARGS="slow and not network and not disabled"
+        - NOSE_ARGS="--skip-network"


could just rename these

jreback · 2016-11-11T12:56:51Z

ci/install_appveyor.ps1

@@ -126,8 +126,8 @@ function UpdateConda ($python_home) {
 function main () {
    InstallMiniconda $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
    UpdateConda $env:PYTHON
-    InstallCondaPackages $env:PYTHON "pip setuptools nose"
+    InstallCondaPackages $env:PYTHON "pip setuptools nose pytest"


I would remove nose (so that it doesn't accidently run)

@jreback I think we should keep nose for now for things like SkipTest. I'll make a followup issue to clean all this up though (changing skips, removing generator-based tests, and using more fixtures are the big ones).

jreback · 2016-11-11T12:57:18Z

.travis.yml

@@ -82,7 +82,7 @@ matrix:
    - python: 3.5
      env:
        - JOB_NAME: "35_nslow"
-        - NOSE_ARGS="not slow and not network and not disabled"
+        - NOSE_ARGS="--skip-slow --skip-network"


need to remove nose from any ci/requirements* files

jreback · 2016-11-11T12:57:59Z

pandas/io/tests/test_pickle.py

@@ -32,6 +33,7 @@ class TestPickle():
    """
    _multiprocess_can_split_ = True

+    @pytest.fixture(autouse=True)


don't use autouse its a bit magical, create explict fixtures (we can come back and fix this later though if easier)

I think this might have been my suggestion.

Without this, I'm not sure setUp will run unless the class inherits from unittest.TestCase.

I think we could change setUp to [setup_class](http://doc.pytest.org/en/latest/xunit_setup.html) and it would run pytest. (Or setup_method if we need access to self rather than class)

@MaximilianR thanks, changed to use setup_class and all the tests run / pass. Made the same change in test_packers to remove autouse.

jreback · 2017-02-08T00:41:00Z

doc/source/contributing.rst

-    nosetests pandas/tests/[test-module].py:[TestClass]
-    nosetests pandas/tests/[test-module].py:[TestClass].[test_method]
+    pytest pandas/tests/[test-module].py
+    pytest pandas/tests/[test-module].py::[TestClass]


I would instead recommend using -k, in fact this is one of the main reasons I really like pytest, e.g.

pytest pandas/[tests-module].py -k cool_test IMHO is much easiest then finding class names and the exact names of the methods.

jreback · 2017-02-08T01:24:36Z

@TomAugspurger all merged. rebase and let's get this in!

jreback · 2017-02-08T01:32:42Z

doc/source/contributing.rst

+
+For more, see the `pytest<http://doc.pytest.org/en/latest/>`_ documentation.
+
+    .. versionadded:: 0.18.0


prob should take out this tag

jreback · 2017-02-08T01:33:58Z

pandas/conftest.py

+    if 'slow' not in item.keywords and item.config.getoption("--only-slow"):
+        pytest.skip("skipping due to --only-slow")
+
+    if 'skip' in item.keywords and item.config.getoption("--skip-network"):


on the todo list, let's just make @network an actual marker then this becomes trival

jreback · 2017-02-08T01:35:41Z

pandas/util/testing.py

@@ -2549,9 +2550,7 @@ def assert_produces_warning(expected_warning=Warning, filter_level="always",
                                    % extra_warnings)


-def disabled(t):
-    t.disabled = True


I think you might need to fix TestCase, IOW the setUpClass and such.

jreback · 2017-02-10T00:07:27Z

can you rebase and we will try for tomorrow for merging?

Pytest wouldn't discover the tests in the same way as nose when 1. The base class inherits from `TestCase`, and 2. The base class doesn't start with `Test` (e.g. PandasSQLTest) 3. The parent class with test methods doesn't start with `Test` ( e.g. _TestSQLAPI) We solved this by moving the class inheriting from `TestCase` down a level from `PandasSQLTest` to the many `TestSQL*` methods.

Pytest won't use setUp unless you inherit from TestCase. These classes don't because they use nose-style generator tests. We change to setup_class becuase that is called by the pytest runner.

Replaced a __init__ method with a setUp method; Makes the XML generated by nose more standard. TST: Move staticmethod to top-level Makes the XML generated by nose more standard.

update ci DOC Fixup runners added only-slow appveyor Move conftest

Removes all pandas.util.nosetester as interactive's no longer needed

On some travis runs, `locale.getlocale()` returned (None, None). When the `tm.set_locale` context manager exists, we set it to `locale.getlocale(locale.LC_ALL, None)`, which defauls to utf-8, so when we compare afterwards, we get a failure.

TomAugspurger · 2017-02-10T15:03:34Z

Rebased and travis is green. I'm under a deadline for Wednesday, so won't be able to start on followup issues till later next week.

jreback · 2017-02-10T15:19:57Z

ok thanks @TomAugspurger let me play a bit with this. I think we can just merge and fix anything else later.

jreback · 2017-02-10T15:52:38Z

thanks @TomAugspurger awesome job!

jreback · 2017-02-10T15:53:36Z

I made a couple of minor changes: 7713f29 and added a bit to the 'list' of TODOs, but otherwise merged very nicely!

Prevent locale from affecting Stata file date creation, which must be en_US. xref pandas-dev#13856 Author: Kevin Sheppard <kevin.k.sheppard@gmail.com> Closes pandas-dev#15208 from bashtage/stata-locale-date-fix and squashes the following commits: a55bfd8 [Kevin Sheppard] BUG: Remove locale conversion from Stata file date

xref pandas-dev#13856 (comment) Author: Jeff Reback <jeff@reback.net> Closes pandas-dev#15333 from jreback/mcs and squashes the following commits: 2edc842 [Jeff Reback] TST/STYLE: remove multiprocess nose flags and slight PEP fixes

closes pandas-dev#13097 Author: Tom Augspurger <tom.augspurger88@gmail.com> Closes pandas-dev#13856 from TomAugspurger/pytest and squashes the following commits: 59e2be9 [Tom Augspurger] NOSE_ARGS -> TEST_ARGS 03695aa [Tom Augspurger] TST: Remove disabled marks 42790ae [Tom Augspurger] TST: Remove test_multi.sh 40d7336 [Tom Augspurger] PKG: redo pd.test import 9ba1f12 [Tom Augspurger] TST: Skip if getlocale is None 14c447c [Tom Augspurger] TST: pd.test uses pytest c4f6008 [Tom Augspurger] TST/CI: Use pytest b268d89 [Tom Augspurger] TST: Change method to make reporting more standard a638390 [Tom Augspurger] TST: Test consistency change c8dc927 [Tom Augspurger] TST: Refactor to use setup_class 9b5f2b2 [Tom Augspurger] TST: Refactor sql test inheritance

TomAugspurger added Testing pandas testing functions or related to the test suite CI Continuous Integration labels Jul 31, 2016

TomAugspurger added this to the 0.20.0 milestone Jul 31, 2016

TomAugspurger force-pushed the pytest branch from db097d6 to 7f735b8 Compare July 31, 2016 14:55

sinhrks reviewed Jul 31, 2016
View reviewed changes

TomAugspurger force-pushed the pytest branch 6 times, most recently from 6985526 to 2d1d91f Compare August 3, 2016 21:25

TomAugspurger force-pushed the pytest branch from 2d1d91f to ae2e146 Compare September 10, 2016 00:26

TomAugspurger mentioned this pull request Sep 18, 2016

TST: Some nose generator tests not running #14244

Closed

TomAugspurger force-pushed the pytest branch 2 times, most recently from cccc82c to cd9bf8f Compare October 26, 2016 13:05

TomAugspurger force-pushed the pytest branch 3 times, most recently from 0e63f0f to ab3d85e Compare November 3, 2016 12:33

TomAugspurger force-pushed the pytest branch 3 times, most recently from f95208f to 312f4e5 Compare November 8, 2016 22:14

jreback reviewed Nov 11, 2016

View reviewed changes

jreback reviewed Feb 8, 2017

View reviewed changes

TomAugspurger force-pushed the pytest branch from 7046dff to 1be6ea5 Compare February 8, 2017 01:28

jreback changed the title ~~[WIP] TST: Use pytest~~ TST: Use pytest Feb 8, 2017

jreback reviewed Feb 8, 2017

View reviewed changes

TomAugspurger mentioned this pull request Feb 8, 2017

pytest followup #15341

Closed

17 tasks

jreback reviewed Feb 8, 2017

View reviewed changes

TomAugspurger added 11 commits February 9, 2017 19:22

TST: Refactor to use setup_class

c8dc927

Pytest won't use setUp unless you inherit from TestCase. These classes don't because they use nose-style generator tests. We change to setup_class becuase that is called by the pytest runner.

TST: Test consistency change

a638390

TST: Change method to make reporting more standard

b268d89

Replaced a __init__ method with a setUp method; Makes the XML generated by nose more standard. TST: Move staticmethod to top-level Makes the XML generated by nose more standard.

TST/CI: Use pytest

c4f6008

update ci DOC Fixup runners added only-slow appveyor Move conftest

TST: pd.test uses pytest

14c447c

Removes all pandas.util.nosetester as interactive's no longer needed

TST: Skip if getlocale is None

9ba1f12

On some travis runs, `locale.getlocale()` returned (None, None). When the `tm.set_locale` context manager exists, we set it to `locale.getlocale(locale.LC_ALL, None)`, which defauls to utf-8, so when we compare afterwards, we get a failure.

PKG: redo pd.test import

40d7336

TST: Remove test_multi.sh

42790ae

TST: Remove disabled marks

03695aa

NOSE_ARGS -> TEST_ARGS

59e2be9

TomAugspurger force-pushed the pytest branch from 1be6ea5 to 59e2be9 Compare February 10, 2017 01:25

jreback closed this in ab8822a Feb 10, 2017

TomAugspurger deleted the pytest branch April 5, 2017 02:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: Use pytest #13856

TST: Use pytest #13856

TomAugspurger commented Jul 31, 2016 •

edited

Loading

jreback commented Jul 31, 2016

sinhrks Jul 31, 2016 •

edited

Loading

max-sixty commented Aug 1, 2016

TomAugspurger commented Aug 1, 2016

TomAugspurger commented Aug 20, 2016 •

edited

Loading

codecov-io commented Nov 3, 2016 •

edited

Loading

jreback Nov 11, 2016

jreback Nov 11, 2016

TomAugspurger Nov 12, 2016

jreback Nov 11, 2016

jreback Nov 11, 2016

max-sixty Nov 11, 2016

TomAugspurger Nov 12, 2016

jreback Feb 8, 2017

jreback commented Feb 8, 2017

jreback Feb 8, 2017

jreback Feb 8, 2017

jreback Feb 8, 2017

jreback commented Feb 10, 2017

TomAugspurger commented Feb 10, 2017

jreback commented Feb 10, 2017

jreback commented Feb 10, 2017

jreback commented Feb 10, 2017


		For more, see the `pytest<http://doc.pytest.org/en/latest/>`_ documentation.

		.. versionadded:: 0.18.0

TST: Use pytest #13856

TST: Use pytest #13856

Conversation

TomAugspurger commented Jul 31, 2016 • edited Loading

Kinds of Changes

jreback commented Jul 31, 2016

sinhrks Jul 31, 2016 • edited Loading

Choose a reason for hiding this comment

max-sixty commented Aug 1, 2016

TomAugspurger commented Aug 1, 2016

TomAugspurger commented Aug 20, 2016 • edited Loading

codecov-io commented Nov 3, 2016 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 10, 2017

TomAugspurger commented Feb 10, 2017

jreback commented Feb 10, 2017

jreback commented Feb 10, 2017

jreback commented Feb 10, 2017

TomAugspurger commented Jul 31, 2016 •

edited

Loading

sinhrks Jul 31, 2016 •

edited

Loading

TomAugspurger commented Aug 20, 2016 •

edited

Loading

codecov-io commented Nov 3, 2016 •

edited

Loading