Repository navigation
[query] fixes hail jar for dataproc notebooks - #13759
Conversation
|
my strategy here was just to undo everything added in this commit, except the version upgrades, so i'm not sure if there are some changes i'm making here that are unnecessary or produce weird results as far as what all ends up in the jar or anything from running |
patrick-schultz
left a comment
patrick-schultz
left a comment
There was a problem hiding this comment.
Approving because @iris-garden was very thorough on testing the effects of this change, but @danking you might want to look when you have a chance and see if this interferes with any important changes from #13551.
closes hail-is#13690. to test that this works, i've been running these commands from the root of my clone of the hail repo: ```bash make -C hail install-editable make -C hail install-hailctl hailctl dataproc start notebook-slowdown-repro --region us-central1 hailctl dataproc connect notebook-slowdown-repro notebook ``` and then running this minimal example in the notebook: ```python import hail hail.utils.range_table(10).show() ``` and making sure it outputs a visual of the table, instead of getting stuck displaying `Stage 0:> (0+X)/Y` and not progressing.
bfcd973 to
61584fe
Compare
|
Wow, really excellent find @iris-garden . I must admit, I don't really understand how this helps. I'm not sure what It makes sense to me that we want to exclude Spark & Scala, but I don't understand how we're pulling those in? Anyway, I'm glad it works. |
|
@danking i'm also not really certain why this works haha, i pretty much just did a bunch of trial and error it would probably be worth taking a deeper look at how we manage our jvm dependencies (especially spark) at some point, but i figured it made sense to just revert a couple lines to fix the bug in the short term |
3 participants
closes #13690.
to test that this works, i've been running these commands from the root of my clone of the hail repo:
and then running this minimal example in the notebook:
and making sure it outputs a visual of the table, instead of getting stuck displaying
Stage 0:> (0+X)/Yand not progressing.