Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hail] [streams] more complicated streams #7229

Merged
merged 25 commits into from Dec 5, 2019
Merged

Conversation

@iitalics
Copy link
Contributor

iitalics commented Oct 8, 2019

Stacked on #7228 (transitively, #7207)

Implements the following pull based streams:

  • ArrayFlatMap
  • ArrayLeftJoinDistinct
  • ArrayScan
  • ArrayAggScan
@iitalics iitalics force-pushed the iitalics:jvm-streams branch 2 times, most recently from 7d35043 to 5e6db04 Oct 9, 2019
@tpoterba

This comment has been minimized.

Copy link
Collaborator

tpoterba commented Oct 10, 2019

nice work, Milo!

@iitalics iitalics force-pushed the iitalics:jvm-streams branch from 5e6db04 to 76e57cf Oct 11, 2019
Copy link
Collaborator

patrick-schultz left a comment

This is really great. I'm excited to get it merged and start using it!

A few questions and requests while I finish working through it:

@iitalics iitalics force-pushed the iitalics:jvm-streams branch from 5a10ab8 to 3a1c1b2 Oct 30, 2019
@patrick-schultz

This comment has been minimized.

Copy link
Collaborator

patrick-schultz commented Nov 5, 2019

This isn't WIP anymore, is it? Also, do you just want to delete the first stream PR? It's gotten a bit out of sync with the changes in this one. If you're happy with this, I'm ready to approve.

@iitalics iitalics force-pushed the iitalics:jvm-streams branch from e4f5e6d to 7c41579 Nov 8, 2019
@tpoterba

This comment has been minimized.

Copy link
Collaborator

tpoterba commented Dec 5, 2019

What do we need to do to get this in?

iitalics added 14 commits Oct 8, 2019
standalone emitstream missing impl

make NA case trivially easy

simplify range impl
fix the Let test
add Skip

other streams

add tests

delete all but map/filter

delete tests
simplify makeArray implementation
delete Empty; rename dummyState => emptyState

get rid of Empty in toArrayIterator

 dummyState to emptyState in missing/range

get rid of Empty in test suite

delete Empty in map, filter

rename dummyState in make
leftjoin test

clean up compose impl

less code dup in EmitStream.compose by doing Skip(..dummyState..)

flatmap new emptyState version
leftjoin new emptyState version
get rid of Empty in compose

get rid of empty in compose (2)
get rid of Empty in leftjoin
@iitalics iitalics force-pushed the iitalics:jvm-streams branch from 7c41579 to 853d491 Dec 5, 2019
@iitalics

This comment has been minimized.

Copy link
Contributor Author

iitalics commented Dec 5, 2019

just rebased and fixed something. should be ready to merge after it passes CI

@tpoterba

This comment has been minimized.

Copy link
Collaborator

tpoterba commented Dec 5, 2019

😱 🎉

@danking danking merged commit 9f2e22e into hail-is:master Dec 5, 2019
1 check passed
1 check passed
ci-test success
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.