Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hail] [streams] more complicated streams #7229

merged 25 commits into from Dec 5, 2019


Copy link

@iitalics iitalics commented Oct 8, 2019

Stacked on #7228 (transitively, #7207)

Implements the following pull based streams:

  • ArrayFlatMap
  • ArrayLeftJoinDistinct
  • ArrayScan
  • ArrayAggScan

@iitalics iitalics force-pushed the jvm-streams branch 2 times, most recently from 7d35043 to 5e6db04 Compare Oct 10, 2019
Copy link

@tpoterba tpoterba commented Oct 10, 2019

nice work, Milo!

Copy link

@patrick-schultz patrick-schultz left a comment

This is really great. I'm excited to get it merged and start using it!

A few questions and requests while I finish working through it:

patrick-schultz previously requested changes Oct 30, 2019
hail/src/test/scala/is/hail/expr/ir/EmitStreamSuite.scala Outdated Show resolved Hide resolved
Copy link

@patrick-schultz patrick-schultz commented Nov 5, 2019

This isn't WIP anymore, is it? Also, do you just want to delete the first stream PR? It's gotten a bit out of sync with the changes in this one. If you're happy with this, I'm ready to approve.

@iitalics iitalics added infrastructure and removed WIP stacked PR labels Nov 14, 2019
Copy link

@tpoterba tpoterba commented Dec 5, 2019

What do we need to do to get this in?

iitalics added 14 commits Dec 5, 2019
standalone emitstream missing impl

make NA case trivially easy

simplify range impl
fix the Let test
add Skip

other streams

add tests

delete all but map/filter

delete tests
simplify makeArray implementation
delete Empty; rename dummyState => emptyState

get rid of Empty in toArrayIterator

 dummyState to emptyState in missing/range

get rid of Empty in test suite

delete Empty in map, filter

rename dummyState in make
leftjoin test

clean up compose impl

less code dup in EmitStream.compose by doing Skip(..dummyState..)

flatmap new emptyState version
leftjoin new emptyState version
get rid of Empty in compose

get rid of empty in compose (2)
get rid of Empty in leftjoin
Copy link
Contributor Author

@iitalics iitalics commented Dec 5, 2019

just rebased and fixed something. should be ready to merge after it passes CI

Copy link

@tpoterba tpoterba commented Dec 5, 2019

😱 🎉

@danking danking merged commit 9f2e22e into hail-is:master Dec 5, 2019
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

None yet

4 participants