[SPARK-3619] Part 2. Upgrade to Mesos 0.21 to work around MESOS-1688 #4361
Conversation
jongyoul
commented
Feb 4, 2015
jongyoul
commented
- MESOS_NATIVE_LIBRARY become deprecated
- Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
|
Test build #26739 has started for PR 4361 at commit
|
|
Test build #26739 has finished for PR 4361 at commit
|
|
Test PASSed. |
|
You also need to update the docs with this change. |
|
@tnachen I've already changed docs/running_on_mesos.md. Did you talk about "Known issues"? I think that doesn't need anymore. What do you think? |
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
- Removed 'Known issues' section
… SPARK-3619-1 * 'SPARK-3619-1' of https://github.com/jongyoul/spark: [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
|
Test build #27833 has started for PR 4361 at commit
|
|
Test build #27833 has finished for PR 4361 at commit
|
|
Test PASSed. |
|
I believe spark-env.sh.template in conf folder has a comment about Mesos native library. Can you change that as well? |
| @@ -15,7 +15,7 @@ | |||
| # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program | |||
| # - SPARK_CLASSPATH, default classpath entries to append | |||
| # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data | |||
| # - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos | |||
There was a problem hiding this comment.
@tnachen I've found only this setting for mesos. Do you think there is another comment?
There was a problem hiding this comment.
I think this is it, thanks !
this PR LGTM.
|
@pwendell @srowen Review this PR, please. This is about SPARK-3619 which is already completed but has some missing documentation about ENV name. From mesos-0.21.0, MESOS_NATIVE_LIBRARY is deprecated and MESOS_NATIVE_JAVA_LIBRARY is introduced newly. we should fix a configuration on this. @tnachen completed reviews on this PR. |
|
OK, so, this change is appropriate for Mesos 0.21.0, but not perhaps earlier versions. So we would only merge into master for 1.4. I don't know Mesos but trust your reviews. I assume that removing the note about MESOS-1688 was intentional because it's no longer a problem with Mesos 0.21.0. |
|
@srowen Master and Branch-1.3 have a dependency of Mesos 0.21.0. Is it ok to merge it into branch-1.3? And MESOS-1688 is resolved by Mesos 0.21.0. So Spark isn't influenced by that issue anymore. So I removed that part. |
|
You're right, SPARK-3619 was merged into 1.3 too. This should follow as well. Will do. |
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY Author: Jongyoul Lee <jongyoul@gmail.com> Closes #4361 from jongyoul/SPARK-3619-1 and squashes the following commits: f1ea91f [Jongyoul Lee] Merge branch 'SPARK-3619-1' of https://github.com/jongyoul/spark into SPARK-3619-1 a6a00c2 [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - Removed 'Known issues' section 2e15a21 [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY 0dace7b [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
