Repository navigation
Generalize categorical aggregation and support span in _colorize - #875
Conversation
|
This looks great! Thanks for working on this. We've had an issue open for years now to generalize count_cat (#140), but the limitation has never affected our own work, so it's never gotten addressed. And I don't think we'd ever noticed that span wasn't implemented for categoricals! Some notes:
|
|
Thanks James,
* I'll review the tests that are failing, but I don't believe they are due to my changes because the master branch appears to fail tests without my changes included.
* I'll take a look at refactoring into a more generic categorical aggregator. I don't think it will be that difficult.
* I'll take a stab at helping with the documentation, but will greatly appreciate your assistance as well.
|
|
It looks like the failing tests on master are due to API changes for Pandas 1.0, which we'll address separately. For now we've pinned Pandas <1.0, so can you please rebase this PR (or patch in the changes from #876) so that the tests can run properly? |
5c6535a to
2d64820
Compare
2d64820 to
20f00f3
Compare
|
James,
I've fixed up the tests and my branch now passes without error. I'm in the
process of trying to make the more generic categorical reduction, but I
figured that might be best handled as a separate PR.
~Michael
…On Fri, Feb 14, 2020 at 3:31 PM James A. Bednar ***@***.***> wrote:
It looks like the failing tests on master are due to API changes for
Pandas 1.0, which we'll address separately. For now we've pinned Pandas
<1.0, so can you please rebase this PR (or patch in the changes from #876
<#876>) so that the tests can
run properly?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#875?email_source=notifications&email_token=AABYEIX5KKPEOG4CAHAWUI3RC35TZA5CNFSM4KVAC4WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL2LB2Q#issuecomment-586461418>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYEIVI3RPL52CGE5XVSTTRC35TZANCNFSM4KVAC4WA>
.
|
|
Sounds good! I'll leave this open until then, as it's already useful as-is, but can close it as soon as the more general one is available. Thanks so much! |
|
James, take a look at the most recent push of a generic categorical
reduction. If this is what you were looking for, I can work on some
additional unit tests.
Thanks,
Michael
…On Sun, Feb 16, 2020 at 11:50 PM James A. Bednar ***@***.***> wrote:
Sounds good! I'll leave this open until then, as it's already useful
as-is, but can close it as soon as the more general one is available.
Thanks so much!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#875?email_source=notifications&email_token=AABYEIQKYYBMWDN2GKGBO3DRDIJR3A5CNFSM4KVAC4WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5BVWQ#issuecomment-586816218>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYEIU5NOBVQ5YP3DC5TTDRDIJR3ANCNFSM4KVAC4WA>
.
|
|
Cool, thanks! Presumably this implementation would now replace count_cat and sum_cat? Presumably we'd leave a wrapper to support count_cat for old code, but now simply calling the new code, and we'd delete sum_cat entirely. For the API for the new functionality, I think we should probably call it "by" as suggested in #140, because even though it is currently limited to a categorical axis, in the future it doesn't seem terribly difficult to generalize this approach to perform 3D aggregation by handling numerical axes as well (by first binning the value then treating it like a category). So for now it would be: |
sum_cat aggregation and support for span in _colorize|
That all sounds good. I didn't want to go to the hassle of replacing
count_cat/sum_cat and such until I had confirmation that I was on the right
track. I will finish up the implementation and change the name to 'by' and
update the PR.
…On Fri, Feb 21, 2020 at 3:26 PM James A. Bednar ***@***.***> wrote:
Cool, thanks! Presumably this implementation would now replace count_cat
and sum_cat? Presumably we'd leave a wrapper to support count_cat for old
code, but now simply calling the new code, and we'd delete sum_cat entirely.
For the API for the new functionality, I think we should probably call it
"by" as suggested in #140
<#140>, because even though
it is currently limited to a categorical axis, in the future it doesn't
seem terribly difficult to generalize this approach to handle numerical
axes as well (by first binning the value then treating it like a category).
So for now it would be:
class by(Reduction):
"""Apply the provided reduction separately per categorical ``column`` value.
Parameters
----------
column : str
Name of the column to aggregate over. Column data type must be
categorical. Resulting aggregate has a outer dimension axis along the
categories present.
reduction : Reduction
Per-category reduction function.
"""
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#875?email_source=notifications&email_token=AABYEIVMFIRGGATPFYFUFZ3REA2GNA5CNFSM4KVAC4WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMT7NDY#issuecomment-589821583>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABYEIR74UP35HU3QY7KAIDREA2GNANCNFSM4KVAC4WA>
.
|
e52179f to
47370f3
Compare
|
I've just pushed up an update the renames the reduction to `by()`, removes
`sum_cat`, refactors `count_cat` to use the new `by`, and adds some
additional unit-tests.
On Fri, Feb 21, 2020 at 3:38 PM Michael Ihde <mike.ihde@randomwalking.com>
wrote:
… That all sounds good. I didn't want to go to the hassle of replacing
count_cat/sum_cat and such until I had confirmation that I was on the right
track. I will finish up the implementation and change the name to 'by' and
update the PR.
On Fri, Feb 21, 2020 at 3:26 PM James A. Bednar ***@***.***>
wrote:
> Cool, thanks! Presumably this implementation would now replace count_cat
> and sum_cat? Presumably we'd leave a wrapper to support count_cat for old
> code, but now simply calling the new code, and we'd delete sum_cat entirely.
>
> For the API for the new functionality, I think we should probably call it
> "by" as suggested in #140
> <#140>, because even though
> it is currently limited to a categorical axis, in the future it doesn't
> seem terribly difficult to generalize this approach to handle numerical
> axes as well (by first binning the value then treating it like a category).
> So for now it would be:
>
> class by(Reduction):
> """Apply the provided reduction separately per categorical ``column`` value.
> Parameters
> ----------
> column : str
> Name of the column to aggregate over. Column data type must be
> categorical. Resulting aggregate has a outer dimension axis along the
> categories present.
> reduction : Reduction
> Per-category reduction function.
> """
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#875?email_source=notifications&email_token=AABYEIVMFIRGGATPFYFUFZ3REA2GNA5CNFSM4KVAC4WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMT7NDY#issuecomment-589821583>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABYEIR74UP35HU3QY7KAIDREA2GNANCNFSM4KVAC4WA>
> .
>
|
Fixes holoviz#140 by providing the `by()` reduction which allows generic categorical reductions.
47370f3 to
8e78486
Compare
|
@jbednar any thoughts on this PR? I think I have finished everything and this is ready to merge into master. |
jbednar
left a comment
There was a problem hiding this comment.
Looks great to me! It's so great to have this functionality at last; good job! I made a few tiny suggestions that you should be able to just merge in place, and then I can merge this PR. I've played with it a little, but once this PR is merged I'll try it out some more when I update the docs. Welcome to the Datashader developers club! :-)
fix: address minor changes Co-Authored-By: James A. Bednar <jbednar@users.noreply.github.com>
|
@jbednar I merged in the minor changes and it should be ready to merge into master. |
|
Thanks!!! |
I have categorical data where I want to sum a column, but grouped by category. The
count_catfunctionality simply counts the number of rows of each category. This PR creates a new aggregation calledsum_cat.In addition, I often need to produce multiple datashader visualizations that are colorized where the alpha range is distributed across a fixed span. This PR enables support for using
spanon categorical data.Please provide comments or suggestions on how to improve the PR if necessary.
Practical examples can be found here:
https://github.com/spectriclabs/jupyter-notebooks/blob/master/Datashader-Improvements.ipynb