Build Spark from source in Docker image #530

Merged
merged 1 commit into from Jun 29, 2016

Conversation

Projects
None yet
5 participants
@mariussoutier
Contributor

mariussoutier commented Jun 28, 2016

This allows to compile for Scala 2.11. To build for Scala 2.11, run SCALA_VERSION="2.11.8 sbt docker.

I've already built 0.6.2 images for both Scala versions using this build file:
https://hub.docker.com/r/mariussoutier/spark-jobserver/tags/

The 0.7.0 snapshot has an Akka issue inside Docker which I'm still investigating, but it doesn't seem related to building from source.

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Jun 28, 2016

Current coverage is 62.06%

Merging #530 into master will increase coverage by 0.05%

@@             master       #530   diff @@
==========================================
  Files            64         64          
  Lines          1740       1740          
  Methods        1627       1627          
  Messages          0          0          
  Branches        102        102          
==========================================
+ Hits           1079       1080     +1   
+ Misses          661        660     -1   
  Partials          0          0          

Powered by Codecov. Last updated by a3f6293...0ccb870

codecov-io commented Jun 28, 2016

Current coverage is 62.06%

Merging #530 into master will increase coverage by 0.05%

@@             master       #530   diff @@
==========================================
  Files            64         64          
  Lines          1740       1740          
  Methods        1627       1627          
  Messages          0          0          
  Branches        102        102          
==========================================
+ Hits           1079       1080     +1   
+ Misses          661        660     -1   
  Partials          0          0          

Powered by Codecov. Last updated by a3f6293...0ccb870

@noorul

This comment has been minimized.

Show comment
Hide comment
@noorul

noorul Jun 28, 2016

Contributor

Can you squash them to single commit?

Contributor

noorul commented Jun 28, 2016

Can you squash them to single commit?

@mariussoutier

This comment has been minimized.

Show comment
Hide comment
@mariussoutier

mariussoutier Jun 28, 2016

Contributor

Done.

Contributor

mariussoutier commented Jun 28, 2016

Done.

@noorul

This comment has been minimized.

Show comment
Hide comment
@noorul

noorul Jun 28, 2016

Contributor
  1. I think it will be great to take hadoop version also as input.
  2. I am not sure whether we should be compiling spark inside docker. Is there a benefit of doing this instead of using binary?
Contributor

noorul commented Jun 28, 2016

  1. I think it will be great to take hadoop version also as input.
  2. I am not sure whether we should be compiling spark inside docker. Is there a benefit of doing this instead of using binary?
@mariussoutier

This comment has been minimized.

Show comment
Hide comment
@mariussoutier

mariussoutier Jun 28, 2016

Contributor

This allows to compile for Scala 2.11.

Contributor

mariussoutier commented Jun 28, 2016

This allows to compile for Scala 2.11.

@mariussoutier mariussoutier referenced this pull request Jun 28, 2016

Closed

Publish Docker with Scala 2.11 #517

4 of 5 tasks complete
@hntd187

This comment has been minimized.

Show comment
Hide comment
@hntd187

hntd187 Jun 28, 2016

Member

This is probably not super far off from the docker file I made for this awhile back. https://github.com/hntd187/spark-jobserver-docker/blob/master/Dockerfile

Member

hntd187 commented Jun 28, 2016

This is probably not super far off from the docker file I made for this awhile back. https://github.com/hntd187/spark-jobserver-docker/blob/master/Dockerfile

@@ -123,21 +136,28 @@ object JobServerBuild extends Build {
copy(artifact, artifactTargetPath)
copy(baseDirectory(_ / "bin" / "server_start.sh").value, file("app/server_start.sh"))
copy(baseDirectory(_ / "bin" / "server_stop.sh").value, file("app/server_stop.sh"))
+ copy(baseDirectory(_ / "bin" / "manager_start.sh").value, file("app/manager_start.sh"))

This comment has been minimized.

@velvia

velvia Jun 28, 2016

Contributor

Thanks. I somehow thought we had added this fix before, but must have somehow slipped.... :-p

@velvia

velvia Jun 28, 2016

Contributor

Thanks. I somehow thought we had added this fix before, but must have somehow slipped.... :-p

This comment has been minimized.

@mariussoutier

mariussoutier Jun 28, 2016

Contributor

Actually in 0.6.2 it was present...

@mariussoutier

mariussoutier Jun 28, 2016

Contributor

Actually in 0.6.2 it was present...

@velvia

This comment has been minimized.

Show comment
Hide comment
@velvia

velvia Jun 28, 2016

Contributor

Looks fine to me. If it is possible to download the binary with Scala 2.11 support also, that would be better, would significantly reduce Docker build times.

Contributor

velvia commented Jun 28, 2016

Looks fine to me. If it is possible to download the binary with Scala 2.11 support also, that would be better, would significantly reduce Docker build times.

@mariussoutier

This comment has been minimized.

Show comment
Hide comment
@mariussoutier

mariussoutier Jun 28, 2016

Contributor

If that were possible I wouldn't bother to build from source :) Scala 2.11 is not available as a pre-built binary. We could download the 2.10 version, but would make it harder to test. Locally the build takes around 20 minutes.

Contributor

mariussoutier commented Jun 28, 2016

If that were possible I wouldn't bother to build from source :) Scala 2.11 is not available as a pre-built binary. We could download the 2.10 version, but would make it harder to test. Locally the build takes around 20 minutes.

@hntd187

This comment has been minimized.

Show comment
Hide comment
@hntd187

hntd187 Jun 28, 2016

Member

Also @velvia soon enough with spark 2.0 scala 2.11 will be the default

Member

hntd187 commented Jun 28, 2016

Also @velvia soon enough with spark 2.0 scala 2.11 will be the default

+ val sparkBuild = s"spark-$sparkVersion"
+ val sparkBuildCmd = scalaBinaryVersion.value match {
+ case "2.10" =>
+ "./make-distribution.sh -Phadoop-2.4 -Phive"

This comment has been minimized.

@noorul

noorul Jun 29, 2016

Contributor

What if someone wants to compile it with hadoop 2.6?

@noorul

noorul Jun 29, 2016

Contributor

What if someone wants to compile it with hadoop 2.6?

This comment has been minimized.

@velvia

velvia Jun 29, 2016

Contributor

Yes, this is a fair point. We might as well make this Hadoop version configurable.

On Jun 28, 2016, at 8:21 PM, Noorul Islam K M notifications@github.com wrote:

In project/Build.scala #530 (comment):

@@ -108,6 +108,19 @@ object JobServerBuild extends Build {
dockerfile in docker := {
val artifact = (assemblyOutputPath in assembly in jobServerExtras).value
val artifactTargetPath = s"/app/${artifact.name}"
+

  •  val sparkBuild = s"spark-$sparkVersion"
    
  •  val sparkBuildCmd = scalaBinaryVersion.value match {
    
  •    case "2.10" =>
    
  •      "./make-distribution.sh -Phadoop-2.4 -Phive"
    
    What if someone wants to compile it with hadoop 2.6?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/spark-jobserver/spark-jobserver/pull/530/files/0ccb8703f12d5e70a37a0cc6732c464a6c2ea7bc#r68879916, or mute the thread https://github.com/notifications/unsubscribe/ABA32_ATqVfiLvLoI1Dyu_GeSz3ZApcOks5qQeSpgaJpZM4JAAR7.

@velvia

velvia Jun 29, 2016

Contributor

Yes, this is a fair point. We might as well make this Hadoop version configurable.

On Jun 28, 2016, at 8:21 PM, Noorul Islam K M notifications@github.com wrote:

In project/Build.scala #530 (comment):

@@ -108,6 +108,19 @@ object JobServerBuild extends Build {
dockerfile in docker := {
val artifact = (assemblyOutputPath in assembly in jobServerExtras).value
val artifactTargetPath = s"/app/${artifact.name}"
+

  •  val sparkBuild = s"spark-$sparkVersion"
    
  •  val sparkBuildCmd = scalaBinaryVersion.value match {
    
  •    case "2.10" =>
    
  •      "./make-distribution.sh -Phadoop-2.4 -Phive"
    
    What if someone wants to compile it with hadoop 2.6?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/spark-jobserver/spark-jobserver/pull/530/files/0ccb8703f12d5e70a37a0cc6732c464a6c2ea7bc#r68879916, or mute the thread https://github.com/notifications/unsubscribe/ABA32_ATqVfiLvLoI1Dyu_GeSz3ZApcOks5qQeSpgaJpZM4JAAR7.

This comment has been minimized.

@mariussoutier

mariussoutier Jun 29, 2016

Contributor

The current docker image is also fixed to Hadoop 2.4 and I didn't want to change everything in one PR. I suggest trying this PR out and see if it fits the build process, then expand on it afterwards (like configuring Hadoop and Java, building SparkR, ...).

@mariussoutier

mariussoutier Jun 29, 2016

Contributor

The current docker image is also fixed to Hadoop 2.4 and I didn't want to change everything in one PR. I suggest trying this PR out and see if it fits the build process, then expand on it afterwards (like configuring Hadoop and Java, building SparkR, ...).

@noorul noorul merged commit afb081b into spark-jobserver:master Jun 29, 2016

3 checks passed

codecov/patch Coverage not affected when comparing a3f6293...0ccb870
Details
codecov/project 62.06% (+0.05%) compared to a3f6293
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@noorul

This comment has been minimized.

Show comment
Hide comment
@noorul

noorul Jun 29, 2016

Contributor

I will add to notes once #517 is complete.

Contributor

noorul commented Jun 29, 2016

I will add to notes once #517 is complete.

@mariussoutier

This comment has been minimized.

Show comment
Hide comment
@mariussoutier

mariussoutier Jun 30, 2016

Contributor

What needs to be done for your release process?

Contributor

mariussoutier commented Jun 30, 2016

What needs to be done for your release process?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment