[SPARK-13578] [core] Modify launch scripts to not use assemblies. #11591
Conversation
Instead of looking for a specially-named assembly, the scripts now will blindly add all jars under the libs directory to the classpath. This libs directory is still currently the old assembly dir, so things should keep working the same way as before until we make more packaging changes. The only lost feature is the detection of multiple assemblies; I consider that a minor nicety that only really affects few developers, so it's probably ok. Tested locally by running spark-shell; also did some minor Win32 testing (just made sure spark-shell started).
|
Test build #52700 has finished for PR 11591 at commit
|
|
Ugh, looks like you're running into the similar issues that I saw with MiMa, so maybe my assembly-free MiMa patch will have to go in first? |
|
if you have a fix for it, great. I'm trying to reproduce this locally but so far no luck. |
|
yeah, I can't seem to reproduce this locally, even running the same commands as the build. @JoshRosen did you figure this out? Do you know what's the root cause? |
|
Seems OK to me, fWIW |
|
Jenkins, retest this please |
|
The MiMa tests are currently disabled due to the DataFrame -> DataSet[Row] migration, so this should no longer be blocked on that for now. #11178 has decoupled |
|
Test build #52895 has finished for PR 11591 at commit
|
|
Jenkins, retest this please (the previous run failed due to known flaky test) |
|
Jenkins, retest this please. |
|
Test build #52943 has finished for PR 11591 at commit
|
This allows pyspark tests to run.
|
Test build #52970 has finished for PR 11591 at commit
|
|
Jenkins, retest this please. |
|
Test build #53039 has finished for PR 11591 at commit
|
|
This seems fine to me, so I'm going to merge it into master. |
Instead of looking for a specially-named assembly, the scripts now will blindly add all jars under the libs directory to the classpath. This libs directory is still currently the old assembly dir, so things should keep working the same way as before until we make more packaging changes. The only lost feature is the detection of multiple assemblies; I consider that a minor nicety that only really affects few developers, so it's probably ok. Tested locally by running spark-shell; also did some minor Win32 testing (just made sure spark-shell started). Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#11591 from vanzin/SPARK-13578.
Instead of looking for a specially-named assembly, the scripts now will blindly add all jars under the libs directory to the classpath. This libs directory is still currently the old assembly dir, so things should keep working the same way as before until we make more packaging changes. The only lost feature is the detection of multiple assemblies; I consider that a minor nicety that only really affects few developers, so it's probably ok. Tested locally by running spark-shell; also did some minor Win32 testing (just made sure spark-shell started). Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#11591 from vanzin/SPARK-13578.
Instead of looking for a specially-named assembly, the scripts now will
blindly add all jars under the libs directory to the classpath. This
libs directory is still currently the old assembly dir, so things should
keep working the same way as before until we make more packaging changes.
The only lost feature is the detection of multiple assemblies; I consider
that a minor nicety that only really affects few developers, so it's probably
ok.
Tested locally by running spark-shell; also did some minor Win32 testing
(just made sure spark-shell started).