Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Groupby all levels except the specified ones #15475

Closed
harisbal opened this issue Feb 22, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@harisbal
Copy link
Contributor

commented Feb 22, 2017

Inspired by

http://stackoverflow.com/questions/25606478/grouping-by-everything-except-for-one-index-column-in-pandas

I was thinking if it would be useful to add a parameter to the groupby function (e.g. allbut=True) so that pandas will group all levels except the specified ones) .

I suggest so because in a multilevel index with several levels, eliminating just one level requires a quite verbose approach

@harisbal harisbal changed the title Groupby all levels except Groupby all levels except the specified ones Feb 22, 2017

@jreback

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2017

Actually this would be quite trivial to implement. We currently support a union (set ops) for FrozenList, e.g. df.index.names + ['foo'], so supporting difference is easy.

This might break other things, so need some tests. Also would be nice to support intersection, & as well.

In [3]: df.groupby(level=df.index.names-['name']).mean()
Out[3]: 
                 accuracy  velocity
condition trial                    
a         1      0.661597  0.277299
          2      0.953187 -0.875767
b         1     -0.836886  0.944629
          2     -0.251443  0.799134

In [4]: quit()
(pandas) bash-3.2$ git diff
diff --git a/pandas/core/base.py b/pandas/core/base.py
index 92ec6bb..b8a4c1d 100644
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -711,6 +711,12 @@ class FrozenList(PandasObject, list):
 
     __iadd__ = __add__
 
+    def __sub__(self, other):
+        if isinstance(other, tuple):
+            other = list(other)
+        return self.__class__([ e for e in self if e not in set(other)])
+
+
     # Python 2 compat
     def __getslice__(self, i, j):
         return self.__class__(super(FrozenList, self).__getslice__(i, j))

@jreback jreback added this to the Next Major Release milestone Feb 22, 2017

@harisbal

This comment has been minimized.

Copy link
Contributor Author

commented Feb 22, 2017

Great idea. Operations on Frozenlists would be really helpful indeed.
Thanks for the quick reply

@jreback

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2017

going to merge this soon: #15477

should provide a bit more runway for expanding / testing these structures.

@bthayer2365 bthayer2365 referenced this issue Feb 25, 2017

Closed

Added FrozenList subtraction #15506

4 of 4 tasks complete

@jreback jreback closed this in 37fe2c4 Mar 2, 2017

@jreback jreback modified the milestones: 0.20.0, Next Major Release Mar 2, 2017

mcocdawc added a commit to mcocdawc/pandas that referenced this issue Mar 2, 2017

ENH: Added FrozenList difference setop
closes pandas-dev#15475

Author: Ben Thayer <benthayer2365@gmail.com>
Author: bthayer2365 <bthayer2365@users.noreply.github.com>

Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits:

428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew
84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index
8dbde1e [Ben Thayer] Rebased to upstream/master
6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
66b3b91 [Ben Thayer] Fixed issue number
3d6cee5 [Ben Thayer] Depricated __add__ in favor of union
ccd75c7 [Ben Thayer] Changed __sub__ to difference
cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs
79dd958 [Ben Thayer] Updated whatsnew to reflect changes
0fc7e19 [Ben Thayer] Removed whitespace
73564ab [Ben Thayer] Added FrozenList subtraction
fee7a7d [bthayer2365] Merge branch 'master' into frozen-index
6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
2ab85cb [Ben Thayer] Fixed issue number
cb95089 [Ben Thayer] Depricated __add__ in favor of union
2e43849 [Ben Thayer] Changed __sub__ to difference
fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs
cd73faa [Ben Thayer] Updated whatsnew to reflect changes
f6381a8 [Ben Thayer] Removed whitespace
ada7cda [Ben Thayer] Added FrozenList subtraction

@jreback jreback reopened this Mar 4, 2017

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2017

reverted in #15566

jreback added a commit to mcocdawc/pandas that referenced this issue Mar 6, 2017

ENH: Added FrozenList difference setop
closes pandas-dev#15475

Author: Ben Thayer <benthayer2365@gmail.com>
Author: bthayer2365 <bthayer2365@users.noreply.github.com>

Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits:

428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew
84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index
8dbde1e [Ben Thayer] Rebased to upstream/master
6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
66b3b91 [Ben Thayer] Fixed issue number
3d6cee5 [Ben Thayer] Depricated __add__ in favor of union
ccd75c7 [Ben Thayer] Changed __sub__ to difference
cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs
79dd958 [Ben Thayer] Updated whatsnew to reflect changes
0fc7e19 [Ben Thayer] Removed whitespace
73564ab [Ben Thayer] Added FrozenList subtraction
fee7a7d [bthayer2365] Merge branch 'master' into frozen-index
6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
2ab85cb [Ben Thayer] Fixed issue number
cb95089 [Ben Thayer] Depricated __add__ in favor of union
2e43849 [Ben Thayer] Changed __sub__ to difference
fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs
cd73faa [Ben Thayer] Updated whatsnew to reflect changes
f6381a8 [Ben Thayer] Removed whitespace
ada7cda [Ben Thayer] Added FrozenList subtraction

AnkurDedania added a commit to AnkurDedania/pandas that referenced this issue Mar 21, 2017

ENH: Added FrozenList difference setop
closes pandas-dev#15475

Author: Ben Thayer <benthayer2365@gmail.com>
Author: bthayer2365 <bthayer2365@users.noreply.github.com>

Closes pandas-dev#15506 from bthayer2365/frozen-index and squashes the following commits:

428a1b3 [Ben Thayer] Added __iadd__ test, fixed whatsnew
84ba405 [Ben Thayer] Merge branch 'master' of github.com:pandas-dev/pandas into frozen-index
8dbde1e [Ben Thayer] Rebased to upstream/master
6f6c140 [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
66b3b91 [Ben Thayer] Fixed issue number
3d6cee5 [Ben Thayer] Depricated __add__ in favor of union
ccd75c7 [Ben Thayer] Changed __sub__ to difference
cd7de26 [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
0ea8d21 [Ben Thayer] Added __isub__ and groupby example to docs
79dd958 [Ben Thayer] Updated whatsnew to reflect changes
0fc7e19 [Ben Thayer] Removed whitespace
73564ab [Ben Thayer] Added FrozenList subtraction
fee7a7d [bthayer2365] Merge branch 'master' into frozen-index
6a2b48d [Ben Thayer] Added docstrings, depricated __iadd__, changed __add__ to use self.union()
2ab85cb [Ben Thayer] Fixed issue number
cb95089 [Ben Thayer] Depricated __add__ in favor of union
2e43849 [Ben Thayer] Changed __sub__ to difference
fdcfbbb [Ben Thayer] Added versionadded tag in docs and renamed test_inplace to test_inplace_add for consistency
2fad2f7 [Ben Thayer] Added __isub__ and groupby example to docs
cd73faa [Ben Thayer] Updated whatsnew to reflect changes
f6381a8 [Ben Thayer] Removed whitespace
ada7cda [Ben Thayer] Added FrozenList subtraction

@jreback jreback modified the milestones: 0.20.0, 0.21.0 Mar 23, 2017

@jreback jreback modified the milestones: 0.21.0, Next Major Release Oct 2, 2017

gfyoung added a commit to forking-repos/pandas that referenced this issue Oct 28, 2018

@jreback jreback modified the milestones: Contributions Welcome, 0.24.0 Nov 1, 2018

gfyoung added a commit to forking-repos/pandas that referenced this issue Nov 1, 2018

gfyoung added a commit to forking-repos/pandas that referenced this issue Nov 1, 2018

jreback added a commit that referenced this issue Nov 3, 2018

JustinZhengBC added a commit to JustinZhengBC/pandas that referenced this issue Nov 14, 2018

brute4s99 added a commit to brute4s99/pandas that referenced this issue Nov 19, 2018

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this issue Feb 28, 2019

Pingviinituutti added a commit to Pingviinituutti/pandas that referenced this issue Feb 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.