ENH: Accept list as level for groupby in non-MultiIndexed objects #13907

agraboso · 2016-08-04T17:44:14Z

closes Groupby doesn't accept level=[0] for Index. #13901
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

sinhrks · 2016-08-04T20:31:55Z

pandas/core/groupby.py

@@ -2368,6 +2368,13 @@ def _get_grouper(obj, key=None, axis=0, level=None, sort=True,
    # axis of the object
    if level is not None:
        if not isinstance(group_axis, MultiIndex):
+            if isinstance(level, list) or isinstance(level, range):
+                if len(level) > 1:


handle empty case.

Only checking the length doesn't meet MI behavior. Following works in MI.

pd.DataFrame([[1, 1], [2, 2], [3, 3]], index=pd.MultiIndex.from_arrays([[1, 1, 1], [1, 2, 3]])).groupby(level=[0, 0]).sum()

I think it should raise if level is not scalar.

Only checking the length doesn't meet MI behavior.

Done by checking len(set(level)) instead of len(level).

handle empty case.

Done by handling len(set(level)) == 0.

I think it should raise if level is not scalar.

But that's the whole point of this PR and the issue it stems from!

Now that I think about it: should we raise an error when given an empty iterable for level instead of making it level=0?

jreback · 2016-08-04T20:46:35Z

pandas/tests/test_groupby.py

+        # GH 13901
+        a = Series([1, 2, 3, 10, 4, 5, 20, 6], Index([1, 2, 3, 1,
+                                                      4, 5, 2, 6], name='foo'))
+


use s = Series(....)

create an expected result and use tm.assert_series_equal

Done. I took the liberty of modifying the test right above (which is the one I based mine off) to use s = Series(...) and assert_series_equal.

codecov-io · 2016-08-04T21:52:05Z

Current coverage is 85.25% (diff: 100%)

Merging #13907 into master will increase coverage by <.01%

@@             master     #13907   diff @@
==========================================
  Files           140        140          
  Lines         50568      50575     +7   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43112      43119     +7   
  Misses         7456       7456          
  Partials          0          0

Powered by Codecov. Last update a7469cf...fcbb724

sinhrks · 2016-08-05T20:47:31Z

pandas/tests/test_groupby.py

+        s = Series([1, 2, 3, 10, 4, 5, 20, 6],
+                   Index([1, 2, 3, 1, 4, 5, 2, 6], name='foo'))
+
+        result = s.groupby(level=[0]).sum()


also add:

[0, 0], [-1]

agraboso · 2016-08-05T21:01:53Z

@sinhrks Made all changes and additions.

jreback · 2016-08-06T23:08:40Z

pandas/core/groupby.py

@@ -2368,11 +2369,21 @@ def _get_grouper(obj, key=None, axis=0, level=None, sort=True,
    # axis of the object
    if level is not None:
        if not isinstance(group_axis, MultiIndex):
+            if is_list_like(level):


add a comment here why this is needed.

you are setting level, but below we reset to None (see L2389), so something wrong.

add a comment here why this is needed.

Done.

you are setting level, but below we reset to None (see L2389), so something wrong.

L2368-L2369 says that this chunk is indeed just doing validation: if the passed level validates, it is reset to None; if it doesn't, it raises a ValueError. This PR only adds to the validation logic.

agraboso · 2016-08-09T21:26:03Z

@jreback @sinhrks Check out the latest amendment. How does it look to you?

jreback · 2016-08-09T21:50:37Z

doc/source/whatsnew/v0.19.0.txt

@@ -437,8 +437,7 @@ API changes
 - ``pd.Timedelta(None)`` is now accepted and will return ``NaT``, mirroring ``pd.Timestamp`` (:issue:`13687`)
 - ``Timestamp``, ``Period``, ``DatetimeIndex``, ``PeriodIndex`` and ``.dt`` accessor have gained a ``.is_leap_year`` property to check whether the date belongs to a leap year. (:issue:`13727`)
 - ``pd.read_hdf`` will now raise a ``ValueError`` instead of ``KeyError``, if a mode other than ``r``, ``r+`` and ``a`` is supplied. (:issue:`13623`)
-
-
+- ``groupby()`` now accepts ``level=[0]`` (in addition to ``level=0``), ``level=-1`` and ``level=[-1]`` for non-``MultiIndex``ed objects. (:issue:`13907`)


too complicated. a user is reading this. just say .groupby() will accept a scalar and a single element list for specifyig level on an Index grouper.

agraboso · 2016-08-11T15:53:37Z

@jreback There is a test that fails, but it has nothing to do with groupby and I cannot reproduce it on my machine. Thoughts?

agraboso · 2016-08-23T21:29:09Z

@jreback

jreback · 2016-09-09T22:32:46Z

can you rebase / update?

agraboso · 2016-09-20T11:57:30Z

Rebased and all green.

jreback · 2016-09-20T12:35:12Z

pandas/tests/test_groupby.py

@@ -2039,6 +2039,23 @@ def test_mulitindex_passthru(self):
        result = df.groupby(axis=1, level=[0, 1]).first()
        assert_frame_equal(result, df)

+    def test_multiindex_negative_level(self):
+        result = self.mframe.groupby(level=-1).sum()


add the issue reference as a comment

jreback · 2016-09-20T12:36:01Z

ok lgtm. @jorisvandenbossche

Closes #13901

jreback · 2016-09-22T10:34:50Z

thanks!

sinhrks reviewed Aug 4, 2016
View reviewed changes

sinhrks added Groupby Bug labels Aug 4, 2016

jreback reviewed Aug 4, 2016
View reviewed changes

agraboso changed the title ~~ENH: Accept list and range as levels for groupby in non-MultiIndexed objects~~ ENH: Accept list as level for groupby in non-MultiIndexed objects Aug 4, 2016

sinhrks reviewed Aug 5, 2016
View reviewed changes

jreback reviewed Aug 6, 2016
View reviewed changes

jreback reviewed Aug 9, 2016
View reviewed changes

jreback mentioned this pull request Aug 11, 2016

API: Expanded resample #13961

Closed

4 tasks

jreback reviewed Sep 20, 2016

View reviewed changes

jreback added this to the 0.19.0 milestone Sep 20, 2016

ENH: Accept list as level for groupby in non-MultiIndexed objects

fcbb724

Closes #13901

jreback closed this in ebc4ac1 Sep 22, 2016

agraboso deleted the fix-13901 branch September 23, 2016 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Accept list as level for groupby in non-MultiIndexed objects #13907

ENH: Accept list as level for groupby in non-MultiIndexed objects #13907

agraboso commented Aug 4, 2016

sinhrks Aug 4, 2016

sinhrks Aug 4, 2016 •

edited

Loading

agraboso Aug 4, 2016

agraboso Aug 5, 2016

jreback Aug 4, 2016

agraboso Aug 4, 2016

codecov-io commented Aug 4, 2016 •

edited

Loading

sinhrks Aug 5, 2016

agraboso commented Aug 5, 2016

jreback Aug 6, 2016

agraboso Aug 6, 2016

agraboso commented Aug 9, 2016

jreback Aug 9, 2016

agraboso Aug 9, 2016

agraboso commented Aug 11, 2016

agraboso commented Aug 23, 2016

jreback commented Sep 9, 2016

agraboso commented Sep 20, 2016

jreback Sep 20, 2016

agraboso Sep 20, 2016

jreback commented Sep 20, 2016

jreback commented Sep 22, 2016

ENH: Accept list as level for groupby in non-MultiIndexed objects #13907

ENH: Accept list as level for groupby in non-MultiIndexed objects #13907

Conversation

agraboso commented Aug 4, 2016

Choose a reason for hiding this comment

sinhrks Aug 4, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Aug 4, 2016 • edited Loading

Current coverage is 85.25% (diff: 100%)

Choose a reason for hiding this comment

agraboso commented Aug 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agraboso commented Aug 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agraboso commented Aug 11, 2016

agraboso commented Aug 23, 2016

jreback commented Sep 9, 2016

agraboso commented Sep 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Sep 20, 2016

jreback commented Sep 22, 2016

sinhrks Aug 4, 2016 •

edited

Loading

codecov-io commented Aug 4, 2016 •

edited

Loading