1. About

What's the goal of the Spark Docker Implementation

This implementation aims to provide a Python Spark Development environment with a tiny footprint that can be easily extended.

Many Spark Images are bloated and are running many services at once. This implementation aims to isolate workloads to other docker containers and starts small by only using the following services:

• Python Development Container with Spark Client
• Spark Proxy API Server (Lighter)
• Kubernetes backend for running Spark on Demand
• JupyterLab Server for running notebooks

What's part of this Spark Docker Implementation

Running Spark in Client/Server mode using Kubernetes Running Python/Pyspark applications with Spark running on demand in Kubernetes Providing Jupyter Notebooks for interactive Spark programming Tiny implementation to avoid bloated images Kubernetes backend with a Spark Proxy Server (Lighter) Kubernetes backend with a JupyterLab Server

What's not part of this Spark Docker Implementation

By default the following services and libraries have not been added: Support for Scala in Jupyter Notebooks SparkR Hive Metastore DB Kyuubi Delta Lake