Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Deprecate combos in favor of lazy single table verbs #22871

Closed
bramtayl opened this issue Jul 19, 2017 · 7 comments
Closed

Proposal: Deprecate combos in favor of lazy single table verbs #22871

bramtayl opened this issue Jul 19, 2017 · 7 comments

Comments

@bramtayl
Copy link
Contributor

bramtayl commented Jul 19, 2017

The canary in the coalmine is mapreducedims, which is a combo of at least three semantic operations: mapping, reducing, and slicing along dimensions. Base seems to be slowly collecting lazy implementations: Base.Generator, Iterators.filter, etc. Of the top of my head, we would want to additionally include reduce and slicedims as single table verbs. Going all the way would mean that the code currently in mapreducedims would simply be a specialized method of collect. This would sweep up #3893. Additionally, we might want to make verbs like sum return a lazy reduce.

@bramtayl
Copy link
Contributor Author

Extensions to lazy operations on multiple tables seems trickier, but having a lazy broadcast might also be useful.

@JeffBezanson
Copy link
Sponsor Member

Why do you want sum to be lazy?

@JeffBezanson
Copy link
Sponsor Member

Also, IIRC mapreduce was added for convenience; some people feel it is much easier to read and write than two nested calls.

@bramtayl
Copy link
Contributor Author

bramtayl commented Jul 19, 2017

Well, it doesn't have to be, but I think it would (at least at the moment) improve performance, because mapslices is currently a lot slower for functions like sum than reducedims is for +.

@simonbyrne
Copy link
Contributor

I've been meaning to write up a Julep and proof-of-concept, but I would like to get rid of mapslices, mapreducedims (and many of the slice-specific functionality) in lieu of a Slice iterator (which is then used with the usual functions).

@mschauer
Copy link
Contributor

@simonbyrne I am curious, I thought this would need a new trait Homogenous() which states that the elements of the iterator have the same shape to avoid the ugly lookahead I had in #16708.

@bramtayl
Copy link
Contributor Author

bramtayl commented Aug 20, 2017

@simonbyrne I started work on a slicing iterator package. So far it just handles mapslices. I'm still not quite there with performance but I'm not too far off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants