Spark 4.2.0 · Hadoop 3.5.0 · Scala 2.13.18 · Python 3.12

Runtime

The software stack is intentionally version-pinned to provide a stable and reproducible reference environment.

Component	Version
Apache Spark	4.2.0, official bin-hadoop3 distribution
Apache Hadoop	3.5.0 — HDFS, YARN and MapReduce
Java	OpenJDK 17
Python	3.12, with PySpark bundled with Spark
Scala	2.13.18
Almond	0.14.5
Container OS	Ubuntu 24.04 LTS

Spark 3.5.9 · Hadoop 3.3.6 · Scala 2.12.18 · Python 3.10

The software stack is intentionally version-pinned to provide a stable and reproducible reference environment.

Component	Version
Apache Spark	3.5.9, official bin-hadoop3 distribution
Apache Hadoop	3.3.6 — HDFS, YARN and MapReduce
Java	OpenJDK 11
Python	3.10, with PySpark bundled with Spark
Scala	2.12.18
Almond	0.14.1
Container OS	Ubuntu 22.04 LTS

Spark & Hadoop Research Lab

A reproducible reference environment for Apache Spark experimentation and benchmarking, running a multi-node Spark and Hadoop stack in Docker.

The lab provides Spark Standalone, YARN, HDFS, JupyterLab, Spark History Server and MapReduce Job History Server in a version-pinned environment, without requiring Spark, Hadoop or Java to be installed directly on the host.

It is designed for research, learning, controlled experimentation and benchmark validation. The default topology runs one master and three workers as separate containers, allowing you to execute distributed jobs, inspect scheduling and executor behavior, store datasets in HDFS, and revisit completed applications through execution-history services.

Spark 3.3.2 · Hadoop 3.3.2 · Scala 2.12.15 · Python 3.8

This release preserves the existing Hadoop and Spark cluster setup before upgrading to Spark 3.5.x.

Included versions

• Apache Spark 3.3.2
• Apache Hadoop 3.3.2

Features

• Docker Compose configuration for a multi-node Hadoop and Spark cluster.
• HDFS storage and YARN resource management.
• Spark History Server for inspecting application execution.
• Shared directories for configuration, scripts, and benchmark files.

HiBench integration

The cluster can be used with the separately maintained HiBench fork adapted for Spark 3.3.x, Hadoop 3.2.x, and Scala 2.12.x. HiBench must be configured and built separately.

This release remains available as a reference for users who need the existing Spark 3.3.x environment. Further development will target Spark 3.5.x.