Melting with not present column does not produce error #23575

michaelsilverstein · 2018-11-08T18:08:30Z

closes Melting with not present column does not produce error #23570
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

pep8speaks · 2018-11-08T18:08:32Z

Hello @michaelsilverstein! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/core/reshape/melt.py !
There are no PEP8 issues in the file pandas/tests/reshape/test_melt.py !

Comment last updated on November 15, 2018 at 22:12 Hours UTC

codecov · 2018-11-08T21:11:16Z

Codecov Report

Merging #23575 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #23575      +/-   ##
==========================================
+ Coverage   92.25%   92.29%   +0.03%     
==========================================
  Files         161      161              
  Lines       51383    51500     +117     
==========================================
+ Hits        47404    47531     +127     
+ Misses       3979     3969      -10

Flag	Coverage Δ
#multiple	`90.68% <100%> (+0.04%)`	⬆️
#single	`42.31% <10%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/reshape/melt.py	`97.54% <100%> (+0.21%)`	⬆️
pandas/core/arrays/datetimelike.py	`95.96% <0%> (-0.19%)`	⬇️
pandas/io/formats/format.py	`97.76% <0%> (-0.12%)`	⬇️
pandas/core/indexes/timedeltas.py	`89.18% <0%> (-0.08%)`	⬇️
pandas/core/arrays/integer.py	`95.47% <0%> (-0.02%)`	⬇️
pandas/tseries/frequencies.py	`97.06% <0%> (-0.02%)`	⬇️
pandas/tseries/offsets.py	`96.98% <0%> (-0.01%)`	⬇️
pandas/core/arrays/period.py	`98.44% <0%> (-0.01%)`	⬇️
pandas/core/util/hashing.py	`98.4% <0%> (ø)`	⬆️
pandas/plotting/_core.py	`83.63% <0%> (ø)`	⬆️
... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a23f901...0db8838. Read the comment docs.

TomAugspurger

This will need tests and a release note.

pandas/core/reshape/melt.py

jreback

pls add some tests and a whatsnew entry

michaelsilverstein · 2018-11-13T14:14:22Z

@jreback would you be able to point me to some documentation on how to properly document tests and a whatsnew entry?

TomAugspurger · 2018-11-13T14:35:56Z

All the contributing docs are at http://pandas-docs.github.io/pandas-docs-travis/contributing.html

release notes: http://pandas-docs.github.io/pandas-docs-travis/contributing.html#documenting-your-code

doc/source/whatsnew/v0.24.0.txt

pandas/core/reshape/melt.py

pandas/tests/reshape/test_melt.py

michaelsilverstein

@jreback I have added tests and a whatsnew entry

doc/source/whatsnew/v0.24.0.txt

pandas/core/reshape/melt.py

pandas/tests/reshape/test_melt.py

TomAugspurger · 2018-11-14T02:41:42Z

You may need to escape the `[` and `]` in the match.

…

On Tue, Nov 13, 2018 at 4:30 PM Michael Silverstein < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pandas/tests/reshape/test_melt.py <#23575 (comment)>: > @@ -661,3 +661,36 @@ def test_col_substring_of_stubname(self): i=['node_id', 'A'], j='time') tm.assert_frame_equal(result, expected) + + def test_melt_missing_columns(self): + # GH-23575 + # This test is to ensure that pandas raises an error if melting is + # attempted with column names absent from the dataframe + + # Generate data + df = pd.DataFrame(np.random.randn(5, 4), columns=list('abcd')) + + # Try to melt with missing `value_vars` column name + with pytest.raises(KeyError, match="The following 'value_vars' are not" Regex wasn't working and now when I try pytest with match equal to the exact output of the error message (as before) I get this frustrating error: AssertionError: Pattern 'The following 'value_vars' are not present in the DataFrame: ['C']' not found in '"The following 'value_vars' are not present in the DataFrame: ['C']"' — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23575 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIn5i7rcFplFXCToXnbwsh-Nf_wxuks5uu0fygaJpZM4YVNGG> .

# Conflicts: # doc/source/whatsnew/v0.24.0.rst

…v_melt_column_check # Conflicts: # doc/source/whatsnew/v0.24.0.rst

michaelsilverstein · 2018-11-15T19:01:19Z

@jreback CircleCI and Azure have passed. Failure in TravisCI seems unrelated (@TomAugspurger agrees it seems).

TomAugspurger · 2018-11-15T19:26:31Z

This failure is real: https://travis-ci.org/pandas-dev/pandas/jobs/455623394#L2875

It seems there's an issue with MultiIndex in the columns. I'm surprised there aren't tests for that outside of the docstring.

michaelsilverstein · 2018-11-15T20:37:09Z

I'll add the tests from the docstring explicitly

michaelsilverstein · 2018-11-16T05:18:13Z

I think this is the only error from TravisCI now?
https://travis-ci.org/pandas-dev/pandas/jobs/455723256#L2807

jreback · 2018-11-16T14:19:14Z

yeah this looks like a ci/code_checks.sh error, run it locally and see

TomAugspurger · 2018-11-16T14:58:42Z

Specifically https://travis-ci.org/pandas-dev/pandas/jobs/455723256#L2745 and https://travis-ci.org/pandas-dev/pandas/jobs/455723256#L2727

michaelsilverstein · 2018-11-16T15:30:15Z

What is the proper order of imports? I had to import Index

TomAugspurger · 2018-11-16T15:33:10Z

http://pandas-docs.github.io/pandas-docs-travis/contributing.html#import-formatting

SO something like isort pandas/core/reshape/melt.py

michaelsilverstein · 2018-11-16T19:48:41Z

I think the failure from TravisCI passed here:
https://travis-ci.org/michaelsilverstein/pandas/builds/456028319

TomAugspurger · 2018-11-16T19:59:49Z

pandas/core/reshape/melt.py

@@ -24,6 +25,10 @@
 def melt(frame, id_vars=None, value_vars=None, var_name=None,
         value_name='value', col_level=None):
    # TODO: what about the existing index?
+    if isinstance(frame.columns, ABCMultiIndex):


I'm not especially familiar with melt and multi-index columns, but I don't think this is quite right.

It seems like you need to specify col_level when you have a MI in the columns, so you should probably just be checks against frame.columns.levels[col_level] when you have a MI.

However, it doesn't quite seem that a col_level is required when there's a MI in the columns. The default of pd.melt(df) seems to work, but any time I specified an id_vars or value_vars without col_level I get an uninformative error message. I'm not sure what's going on.

I think you need to provide col_level for MI when only melting on one level like this example from the docstring (that I added a new test for):

pd.melt(df, col_level=0, id_vars=['A'], value_vars=['B'])

But you don't need to specify col_level when using all levels of MI:
pd.melt(df, id_vars=[('A', 'D')], value_vars=[('B', 'E')])

All I am doing at L28 is gathering column names from all levels. There are other checks to make sure that melting is performed properly, this will just check to make sure that whatever you pass, it is in your df

yeah i think this is ok, can you provdie a comment on what is going on.

doc/source/whatsnew/v0.24.0.rst

jreback · 2018-11-19T20:29:47Z

pandas/core/reshape/melt.py

@@ -24,6 +25,10 @@
 def melt(frame, id_vars=None, value_vars=None, var_name=None,
         value_name='value', col_level=None):
    # TODO: what about the existing index?
+    if isinstance(frame.columns, ABCMultiIndex):


yeah i think this is ok, can you provdie a comment on what is going on.

jreback · 2018-11-19T20:30:58Z

pandas/tests/reshape/test_melt.py

+                KeyError,
+                match=msg.format(Var='id_vars',
+                                 Col="\\['not_here', 'or_there'\\]")):
+            df.melt(['a', 'b', 'not_here', 'or_there'], ['c', 'd'])


can you do an example with an MI and columns that are not in the top level of the MI, ideally try with and w/o col_level as well.

jreback · 2018-11-21T16:57:17Z

lgtm. ping on green.

jreback · 2018-11-21T17:16:13Z

thanks!

)

check for columns in dataframe

855985d

check for columns in dataframe

40fdb05

TomAugspurger reviewed Nov 8, 2018

View reviewed changes

pandas/core/reshape/melt.py Outdated Show resolved Hide resolved

pandas/core/reshape/melt.py Outdated Show resolved Hide resolved

datapythonista added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Error Reporting Incorrect or improved errors from pandas labels Nov 9, 2018

jreback changed the title ~~check for columns in dataframe~~ Melting with not present column does not produce error Nov 11, 2018

jreback requested changes Nov 11, 2018

View reviewed changes

michaelsilverstein added 2 commits November 13, 2018 09:07

check difference with Index; use {} str formatting

9670da2

missing.any()

3ffc870

started test

8139f78

michaelsilverstein added 5 commits November 13, 2018 09:42

added to whatsnew

0a94650

PEP criteria

d0f6d23

missing.empty to accommodate MultiIndex

6c76161

rm *

ad3d926

rm comment

e097a87

TomAugspurger reviewed Nov 13, 2018

View reviewed changes

michaelsilverstein added 6 commits November 13, 2018 12:50

add test for id_var and multiple missing

5ff3a32

reformat error statement; Value->KeyError

fcbda15

simplified test

3175b34

Issue -> GH

515fb9f

PEP criteria

c7d6fcf

PEP criteria

5911cc3

michaelsilverstein commented Nov 13, 2018

View reviewed changes

TomAugspurger reviewed Nov 13, 2018

View reviewed changes

test not working now

47ca7fc

michaelsilverstein added 4 commits November 14, 2018 17:09

Merge branch 'master' into dev_melt_column_check

479b761

# Conflicts: # doc/source/whatsnew/v0.24.0.rst

resolving conflicts

01e8d74

Merge branch 'master' of https://github.com/pandas-dev/pandas into de…

6762b21

…v_melt_column_check # Conflicts: # doc/source/whatsnew/v0.24.0.rst

Merge branch 'master' of https://github.com/pandas-dev/pandas into de…

eae7716

…v_melt_column_check # Conflicts: # doc/source/whatsnew/v0.24.0.rst

handle multiindex columns

fba641f

michaelsilverstein added 2 commits November 15, 2018 17:09

test single var melt with multiindex

06b7cdb

test single var melt with multiindex

39c746b

pep8 and index sorting

af170e1

TomAugspurger reviewed Nov 16, 2018

View reviewed changes

jreback requested changes Nov 19, 2018

View reviewed changes

jreback reviewed Nov 19, 2018

View reviewed changes

michaelsilverstein added 3 commits November 21, 2018 11:00

rm extra description

4c9bc9f

add comment

c59d29f

add MI tests

0db8838

jreback approved these changes Nov 21, 2018

View reviewed changes

jreback merged commit 3e01c38 into pandas-dev:master Nov 21, 2018

michaelsilverstein deleted the dev_melt_column_check branch November 21, 2018 17:54

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Melting with not present column does not produce error (pandas-dev#23575

88f9b80

)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Melting with not present column does not produce error (pandas-dev#23575

378aebc

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Melting with not present column does not produce error #23575

Melting with not present column does not produce error #23575

michaelsilverstein commented Nov 8, 2018 •

edited

pep8speaks commented Nov 8, 2018 •

edited

codecov bot commented Nov 8, 2018 •

edited

TomAugspurger left a comment

jreback left a comment

michaelsilverstein commented Nov 13, 2018

TomAugspurger commented Nov 13, 2018

michaelsilverstein left a comment •

edited

TomAugspurger commented Nov 14, 2018 via email

michaelsilverstein commented Nov 15, 2018

TomAugspurger commented Nov 15, 2018

michaelsilverstein commented Nov 15, 2018

michaelsilverstein commented Nov 16, 2018

jreback commented Nov 16, 2018

TomAugspurger commented Nov 16, 2018

michaelsilverstein commented Nov 16, 2018

TomAugspurger commented Nov 16, 2018

michaelsilverstein commented Nov 16, 2018

TomAugspurger Nov 16, 2018

michaelsilverstein Nov 16, 2018

michaelsilverstein Nov 16, 2018

michaelsilverstein Nov 19, 2018

jreback Nov 19, 2018

jreback Nov 19, 2018

jreback Nov 19, 2018

jreback commented Nov 21, 2018

jreback commented Nov 21, 2018

Melting with not present column does not produce error #23575

Melting with not present column does not produce error #23575

Conversation

michaelsilverstein commented Nov 8, 2018 • edited

pep8speaks commented Nov 8, 2018 • edited

Comment last updated on November 15, 2018 at 22:12 Hours UTC

codecov bot commented Nov 8, 2018 • edited

Codecov Report

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

michaelsilverstein commented Nov 13, 2018

TomAugspurger commented Nov 13, 2018

michaelsilverstein left a comment • edited

Choose a reason for hiding this comment

TomAugspurger commented Nov 14, 2018 via email

michaelsilverstein commented Nov 15, 2018

TomAugspurger commented Nov 15, 2018

michaelsilverstein commented Nov 15, 2018

michaelsilverstein commented Nov 16, 2018

jreback commented Nov 16, 2018

TomAugspurger commented Nov 16, 2018

michaelsilverstein commented Nov 16, 2018

TomAugspurger commented Nov 16, 2018

michaelsilverstein commented Nov 16, 2018

TomAugspurger Nov 16, 2018

Choose a reason for hiding this comment

michaelsilverstein Nov 16, 2018

Choose a reason for hiding this comment

michaelsilverstein Nov 16, 2018

Choose a reason for hiding this comment

michaelsilverstein Nov 19, 2018

Choose a reason for hiding this comment

jreback Nov 19, 2018

Choose a reason for hiding this comment

jreback Nov 19, 2018

Choose a reason for hiding this comment

jreback Nov 19, 2018

Choose a reason for hiding this comment

jreback commented Nov 21, 2018

jreback commented Nov 21, 2018

michaelsilverstein commented Nov 8, 2018 •

edited

pep8speaks commented Nov 8, 2018 •

edited

codecov bot commented Nov 8, 2018 •

edited

michaelsilverstein left a comment •

edited