BUG: adding numpy errstate to dataframe/series broadcasting (GH16378) #16433

AndrewArcher · 2017-05-22T19:48:26Z

chris-b1 · 2017-05-22T20:18:42Z

Looks like this may also close #16306 - could you add some confirming tests?

chris-b1 · 2017-05-22T20:23:24Z

pandas/tests/frame/test_convert_to.py

+        df = DataFrame({'A': [np.nan, 2.0, np.nan]})
+        s = Series([1, 1, 1])
+
+        df.clip_lower(s, axis=0)


you can use with tm.assert_produces_warning(None): around this to assert no warning raised rather than reading stderr

TomAugspurger · 2017-05-22T20:22:49Z

pandas/tests/frame/test_convert_to.py

+        df = DataFrame({'A': [np.nan, 2.0, np.nan]})
+        s = Series([1, 1, 1])
+
+        df.clip_lower(s, axis=0)


Could you write this as

with warnings.catch_warnings(record=True) as w: df.clip_lower(s, axis=0) assert len(w) == 0

And you shouldnt need the capsys then (may have to import warnings).

nvm, do what @chris-b1 said, that's shorter than catching and asserting 😄

TomAugspurger · 2017-05-22T20:23:50Z

pandas/tests/frame/test_convert_to.py

@@ -229,3 +229,12 @@ def test_to_records_datetimeindex_with_tz(self, tz):

        # both converted to UTC, so they are equal
        tm.assert_numpy_array_equal(result, expected)
+
+    def test_to_series(self, capsys):


Could you move this to pandas/tests/frame/test_analytics.py

codecov · 2017-05-22T22:20:19Z

Codecov Report

Merging #16433 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #16433      +/-   ##
==========================================
- Coverage   90.42%   90.42%   -0.01%     
==========================================
  Files         161      161              
  Lines       51023    51024       +1     
==========================================
  Hits        46138    46138              
- Misses       4885     4886       +1

Flag	Coverage Δ
#multiple	`88.26% <100%> (-0.01%)`	⬇️
#single	`40.17% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/ops.py	`91.69% <100%> (+0.01%)`	⬆️
pandas/core/common.py	`91.05% <0%> (-0.34%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49ec31b...2a1bee3. Read the comment docs.

codecov · 2017-05-22T22:20:24Z

Codecov Report

Merging #16433 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #16433      +/-   ##
==========================================
+ Coverage   90.42%   90.42%   +<.01%     
==========================================
  Files         161      161              
  Lines       51027    51028       +1     
==========================================
+ Hits        46142    46143       +1     
  Misses       4885     4885

Flag	Coverage Δ
#multiple	`88.26% <100%> (ø)`	⬆️
#single	`40.17% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/ops.py	`91.69% <100%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04356a8...c78ee39. Read the comment docs.

jreback · 2017-05-22T22:32:44Z

doc/source/whatsnew/v0.20.2.txt

@@ -42,6 +42,7 @@ Conversion
 ^^^^^^^^^^

 - Bug in ``pd.to_numeric()`` in which empty data inputs were causing Python to crash (:issue:`16302`)
+- Bug when broadcasting dataframe to series silencing numpy warning messages in common use case (:issue:`16378`)


Silence numpy warnings when broadcasting DataFrame to Series with comparison ops.

add 16306 as well

jreback · 2017-05-22T22:33:30Z

pandas/tests/frame/test_analytics.py

@@ -2073,3 +2073,14 @@ def test_n_duplicate_index(self, df_duplicates, n, order):
        result = df.nlargest(n, order)
        expected = df.sort_values(order, ascending=False).head(n)
        tm.assert_frame_equal(result, expected)
+
+    def test_series_broadcasting(self):
+        df = DataFrame({'A': [np.nan, 2.0, np.nan]})


can you add the issue number as a comment here. (and a 1-line comment that this is a smoke test for numpy warnings)

jreback

minor doc comments. ping on green.

TomAugspurger · 2017-05-23T02:03:02Z

@AndrewArcher small lint failure here: https://travis-ci.org/pandas-dev/pandas/jobs/235018875#L1380 Mind removing the extra newline at the end of the file? Then we can get it merged.

Actually I'll just push a commit fixing that quick. I'm guessing your editor may be setup to add them automatically, and I don't want you to have to fiddle with settings.

jreback · 2017-05-23T10:37:53Z

pandas/tests/frame/test_analytics.py

+
+        with tm.assert_produces_warning(None):
+            df.clip_lower(s, axis=0)
+            df.gt(s, axis=0)


pd.DataFrame([0,0]).gt(pd.Series([0, np.nan]), axis=0)

can you also add this test (the Series has the nan)

maybe just loop over these e.g.

for op in ['lt', 'le', 'ge', 'lt', 'eq', 'ne']: getattr(df, op)(s, axis=0)

TomAugspurger · 2017-05-23T19:56:04Z

pandas/tests/frame/test_analytics.py

+        with tm.assert_produces_warning(None):
+            df_nan.clip_lower(s, axis=0)
+            for op in ['lt', 'le', 'gt', 'ge', 'eq', 'ne']:
+                getattr(df, op)(s_nan, axis=0)


Hah, now we'll have opposite problem.

pandas/tests/frame/test_analytics.py:2088:47: W292 no newline at end of file

Basically line 2088 should end with a newline, but there shouldn't be a line 2089 that's just a newline. Want me to push a fix for this, or do you want to take care of it?

I think I updated it correctly this time - if not, could you please push the correct fix?

I think that looks good.

…h comparison ops (GH16378, GH16306) TST: test for fix of GH16378, GH16306

TomAugspurger · 2017-05-25T02:08:59Z

Merging (the Appveyor failure was fixed in master this morning).

Thanks @AndrewArcher!

…, GH16306) (pandas-dev#16433) TST: test for fix of GH16378, GH16306(cherry picked from commit 96f3e7c)

…, GH16306) (#16433) TST: test for fix of GH16378, GH16306(cherry picked from commit 96f3e7c)

…, GH16306) (pandas-dev#16433) TST: test for fix of GH16378, GH16306

chris-b1 reviewed May 22, 2017

View reviewed changes

TomAugspurger reviewed May 22, 2017

View reviewed changes

jreback added Bug Numeric Operations Arithmetic, Comparison, and Logical operations Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels May 22, 2017

jreback reviewed May 22, 2017

View reviewed changes

jreback approved these changes May 22, 2017

View reviewed changes

jreback added this to the 0.20.2 milestone May 22, 2017

AndrewArcher force-pushed the dataframe-series-broadcasting-bug-16378 branch 2 times, most recently from c72dc40 to fe1ab74 Compare May 22, 2017 22:54

jreback requested changes May 23, 2017

View reviewed changes

TomAugspurger reviewed May 23, 2017

View reviewed changes

BUG: Silence numpy warnings when broadcasting DataFrame to Series wit…

c78ee39

…h comparison ops (GH16378, GH16306) TST: test for fix of GH16378, GH16306

AndrewArcher force-pushed the dataframe-series-broadcasting-bug-16378 branch from 3d41393 to c78ee39 Compare May 23, 2017 21:26

jreback approved these changes May 24, 2017

View reviewed changes

TomAugspurger merged commit 96f3e7c into pandas-dev:master May 25, 2017

TomAugspurger pushed a commit to TomAugspurger/pandas that referenced this pull request May 29, 2017

BUG: Silence numpy warnings when broadcasting comparison ops (GH16378…

07ae2e0

…, GH16306) (pandas-dev#16433) TST: test for fix of GH16378, GH16306(cherry picked from commit 96f3e7c)

TomAugspurger added Backported and removed Needs Backport labels May 30, 2017

TomAugspurger pushed a commit that referenced this pull request May 30, 2017

BUG: Silence numpy warnings when broadcasting comparison ops (GH16378…

0ffce83

…, GH16306) (#16433) TST: test for fix of GH16378, GH16306(cherry picked from commit 96f3e7c)

stangirala pushed a commit to stangirala/pandas that referenced this pull request Jun 11, 2017

BUG: Silence numpy warnings when broadcasting comparison ops (GH16378…

77a8581

…, GH16306) (pandas-dev#16433) TST: test for fix of GH16378, GH16306

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: adding numpy errstate to dataframe/series broadcasting (GH16378) #16433

BUG: adding numpy errstate to dataframe/series broadcasting (GH16378) #16433

AndrewArcher commented May 22, 2017 •

edited by jreback

chris-b1 commented May 22, 2017

chris-b1 May 22, 2017

TomAugspurger May 22, 2017

TomAugspurger May 22, 2017

TomAugspurger May 22, 2017

TomAugspurger May 22, 2017

codecov bot commented May 22, 2017

codecov bot commented May 22, 2017 •

edited

jreback May 22, 2017

jreback May 22, 2017

jreback May 22, 2017

jreback left a comment

TomAugspurger commented May 23, 2017 •

edited

jreback May 23, 2017

TomAugspurger May 23, 2017

AndrewArcher May 23, 2017

TomAugspurger May 23, 2017

TomAugspurger commented May 25, 2017

BUG: adding numpy errstate to dataframe/series broadcasting (GH16378) #16433

BUG: adding numpy errstate to dataframe/series broadcasting (GH16378) #16433

Conversation

AndrewArcher commented May 22, 2017 • edited by jreback

chris-b1 commented May 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 22, 2017

Codecov Report

codecov bot commented May 22, 2017 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

TomAugspurger commented May 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger commented May 25, 2017

AndrewArcher commented May 22, 2017 •

edited by jreback

codecov bot commented May 22, 2017 •

edited

TomAugspurger commented May 23, 2017 •

edited