Apache Spark is a high-performance engine for large-scale computing tasks, such as data processing, machine learning and real-time data streaming. It includes APIs for Java, Python, Scala and R.
$ curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-spark/master/docker-compose.yml
$ docker-compose upYou can find the available configuration options in the Environment Variables section.
- Bitnami closely tracks upstream source changes and promptly publishes new versions of this image using our automated systems.
- With Bitnami images the latest bug fixes and features are available as soon as possible.
- Bitnami containers, virtual machines and cloud images use the same components and configuration approach - making it easy to switch between formats based on your project needs.
- All our images are based on minideb a minimalist Debian based container image which gives you a small base container image and the familiarity of a leading Linux distribution.
- All Bitnami images available in Docker Hub are signed with Docker Content Trust (DCT). You can use
DOCKER_CONTENT_TRUST=1to verify the integrity of the images. - Bitnami container images are released daily with the latest distribution packages available.
This CVE scan report contains a security report with all open CVEs. To get the list of actionable security issues, find the "latest" tag, click the vulnerability report link under the corresponding "Security scan" field and then select the "Only show fixable" filter on the next page.
Deploying Bitnami applications as Helm Charts is the easiest way to get started with our applications on Kubernetes. Read more about the installation in the Bitnami Apache Spark Chart GitHub repository.
Bitnami containers can be used with Kubeapps for deployment and management of Helm Charts in clusters.
Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers in our docs.
Learn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags in our documentation page.
Subscribe to project updates by watching the bitnami/spark GitHub repo.
The recommended way to get the Bitnami Spark Docker Image is to pull the prebuilt image from the Docker Hub Registry.
$ docker pull bitnami/spark:latestTo use a specific version, you can pull a versioned tag. You can view the list of available versions in the Docker Hub Registry.
$ docker pull bitnami/spark:[TAG]If you wish, you can also build the image yourself.
docker build -t bitnami/spark:latest 'https://github.com/bitnami/bitnami-docker-spark.git#master:3/debian-10'When you start the spark image, you can adjust the configuration of the instance by passing one or more environment variables either on the docker-compose file or on the docker run command line. If you want to add a new environment variable:
- For docker-compose add the variable name and value under the application section in the
docker-compose.ymlfile present in this repository:
spark:
...
environment:
- SPARK_MODE=master
...- For manual execution add a -e option with each variable and value:
$ docker run -d --name spark \
--network=spark_network \
-e SPARK_MODE=master \
bitnami/sparkAvailable variables:
- SPARK_MODE: Cluster mode starting Spark. Valid values: master, worker. Default: master
- SPARK_MASTER_URL: Url where the worker can find the master. Only needed when spark mode is worker. Default: spark://spark-master:7077
- SPARK_RPC_AUTHENTICATION_ENABLED: Enable RPC authentication. Default: no
- SPARK_RPC_AUTHENTICATION_SECRET: The secret key used for RPC authentication. No defaults.
- SPARK_RPC_ENCRYPTION_ENABLED: Enable RPC encryption. Default: no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED: Enable local storage encryption: Default no
- SPARK_SSL_ENABLED: Enable SSL configuration. Default: no
- SPARK_SSL_KEY_PASSWORD: The password to the private key in the key store. No defaults.
- SPARK_SSL_KEYSTORE_FILE: Location of the key store. Default: /opt/bitnami/spark/conf/certs/spark-keystore.jks.
- SPARK_SSL_KEYSTORE_PASSWORD: The password for the key store. No defaults.
- SPARK_SSL_TRUSTSTORE_PASSWORD: The password for the trust store. No defaults.
- SPARK_SSL_TRUSTSTORE_FILE: Location of the key store. Default: /opt/bitnami/spark/conf/certs/spark-truststore.jks.
- SPARK_SSL_NEED_CLIENT_AUTH: Whether to require client authentication. Default: yes
- SPARK_SSL_PROTOCOL: TLS protocol to use. Default: TLSv1.2
- SPARK_DAEMON_USER: Spark system user when the container is started as root. Default: spark
- SPARK_DAEMON_GROUP: Spark system group when the container is started as root. Default: spark
More environment variables natively supported by Spark can be found at the official documentation.
For example, you could still use SPARK_WORKER_CORES or SPARK_WORKER_MEMORY to configure the number of cores and the amount of memory to be used by a worker machine.
The Bitnani Spark docker image supports enabling RPC authentication, RPC encryption and local storage encryption easily using the following env vars in all the nodes of the cluster.
+ SPARK_RPC_AUTHENTICATION_ENABLED=yes
+ SPARK_RPC_AUTHENTICATION_SECRET=RPC_AUTHENTICATION_SECRET
+ SPARK_RPC_ENCRYPTION=yes
+ SPARK_LOCAL_STORAGE_ENCRYPTION=yesPlease note that
RPC_AUTHENTICATION_SECRETis a placeholder that needs to be updated with a correct value.
Be also aware that currently is not possible to submit an application to a standalone cluster if RPC authentication is configured. More info about the issue here.
Additionally, SSL configuration can be easily activated following the next steps:
- Enable SSL configuration by setting the following env vars:
+ SPARK_SSL_ENABLED=yes
+ SPARK_SSL_KEY_PASSWORD=KEY_PASSWORD
+ SPARK_SSL_KEYSTORE_PASSWORD=KEYSTORE_PASSWORD
+ SPARK_SSL_TRUSTSTORE_PASSWORD=TRUSTSTORE_PASSWORD
+ SPARK_SSL_NEED_CLIENT_AUTH=yes
+ SPARK_SSL_PROTOCOL=TLSv1.2Please note that
KEY_PASSWORD,KEYSTORE_PASSWORD, andTRUSTSTORE_PASSWORDare placeholders that needs to be updated with a correct value.
- You need to mount your spark keystore and truststore files to
/opt/bitnami/spark/conf/certs. Please note they should be calledspark-keystore.jksandspark-truststore.jksand they should be in JKS format.
A Spark cluster can easily be setup with the default docker-compose.yml file from the root of this repo. The docker-compose includes two different services, spark-master and spark-worker.
By default, when you deploy the docker-compose file you will get a Spark cluster with 1 master and 1 worker.
If you want N workers, all you need to do is start the docker-compose deployment with the following command:
$ docker-compose up --scale spark-worker=3The image looks for configuration in the conf/ directory of /opt/bitnami/spark.
...
volumes:
- /path/to/spark-defaults.conf:/opt/bitnami/spark/conf/spark-defaults.conf
...$ docker run --name spark -v /path/to/spark-defaults.conf:/opt/bitnami/spark/conf/spark-defaults.conf bitnami/spark:latestAfter that, your changes will be taken into account in the server's behaviour.
By default, this container bundles a generic set of jar files but the default image can be extended to add as many jars as needed for your specific use case. For instance, the following Dockerfile adds aws-java-sdk-bundle-1.11.704.jar:
FROM bitnami/spark
RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.704/aws-java-sdk-bundle-1.11.704.jar --output /opt/bitnami/spark/jars/aws-java-sdk-bundle-1.11.704.jarIn a similar way that in the previous section, you may want to use a different version of Hadoop jars.
Go to https://spark.apache.org/downloads.html and copy the download url bundling the Hadoop version you want and matching the Spark version of the container. Extend the Bitnami container image as below:
FROM bitnami/spark:3.0.0
USER root
RUN rm -r /opt/bitnami/spark/jars && \
curl --location http://mirror.cc.columbia.edu/pub/software/apache/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz | \
tar --extract --gzip --strip=1 --directory /opt/bitnami/spark/ spark-3.0.0-bin-hadoop2.7/jars/
USER 1001You can check the Hadoop version by running the following commands in the new container image:
$ pyspark
>>> sc._gateway.jvm.org.apache.hadoop.util.VersionInfo.getVersion()
'2.7.4'The Bitnami Spark Docker image sends the container logs to the stdout. To view the logs:
$ docker logs sparkor using Docker Compose:
$ docker-compose logs sparkYou can configure the containers logging driver using the --log-driver option if you wish to consume the container logs differently. In the default configuration docker uses the json-file driver.
To backup your data, configuration and logs, follow these simple steps:
$ docker stop sparkor using Docker Compose:
$ docker-compose stop sparkWe need to mount two volumes in a container we will use to create the backup: a directory on your host to store the backup in, and the volumes from the container we just stopped so we can access the data.
$ docker run --rm -v /path/to/spark-backups:/backups --volumes-from spark busybox \
cp -a /bitnami/spark:latest /backups/latestor using Docker Compose:
$ docker run --rm -v /path/to/spark-backups:/backups --volumes-from `docker-compose ps -q spark` busybox \
cp -a /bitnami/spark:latest /backups/latestRestoring a backup is as simple as mounting the backup as volumes in the container.
$ docker run -v /path/to/spark-backups/latest:/bitnami/spark bitnami/spark:latestor by modifying the docker-compose.yml file present in this repository:
services:
spark:
...
volumes:
- /path/to/spark-backups/latest:/bitnami/spark
...Bitnami provides up-to-date versions of spark, including security patches, soon after they are made upstream. We recommend that you follow these steps to upgrade your container.
$ docker pull bitnami/spark:latestor if you're using Docker Compose, update the value of the image property to
bitnami/spark:latest.
Before continuing, you should backup your container's data, configuration and logs.
Follow the steps on creating a backup.
$ docker rm -v sparkor using Docker Compose:
$ docker-compose rm -v sparkRe-create your container from the new image, restoring your backup if necessary.
$ docker run --name spark bitnami/spark:latestor using Docker Compose:
$ docker-compose up spark- The container image was updated to use Hadoop
3.2.x. If you want to use a different version, please read Using a different version of Hadoop jars.
- This image now has an aws-cli and two jars: hadoop-aws and aws-java-sdk for provide an easier way to use AWS.
We'd love for you to contribute to this container. You can request new features by creating an issue, or submit a pull request with your contribution.
If you encountered a problem running this container, you can file an issue. For us to provide better support, be sure to include the following information in your issue:
- Host OS and version
- Docker version (
docker version) - Output of
docker info - Version of this container (
echo $BITNAMI_IMAGE_VERSIONinside the container) - The command you used to run the container, and any relevant output you saw (masking any sensitive information)
Copyright (c) 2021 Bitnami
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.