spark on docker
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Dockerfile
README.md

README.md

Spark docker image

Docker Pulls

Info

  • Spark 1.6.0
  • Mesos 0.28.1
  • Jre 1.7

Usage

  • SPARK_HOME :/usr/local/spark-1.6.0-bin-hadoop2.6

  • Try on docker image

    sudo docker run -it --name spark-try --net host adolphlwq/docker-spark bash
    

Use it on Mesos Cluster as Mesos Spark executor

Refer Spark on Mesos And Spark configuration for more details.

  • Instance SparkConf

    conf = SparkConf()
    conf.setMaster(mesos_endpoint)
    conf.set("spark.mesos.executor.docker.image", "adolphlwq/docker-spark")
    conf.set("spark.mesos.executor.home", "/usr/local/spark-1.6.0-bin-hadoop2.6")
  • Instance SparkContext

    sc = SparkContext(conf=conf)
    below is your code
    ...

TODOs

No todos. Any issues is welcome.