Skip to content

1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@dongjoon-hyun dongjoon-hyun released this 26 Jul 22:06
1.0.0
d8b1943

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 1.0.0 (App Version) with Helm chart 1.8.0 (Chart Version).

Notable Features

  • Support K8s v1.34, v1.35, and v1.36
  • Support Apache Spark 4.0, 4.1, and 4.2 and drop Spark 3.5
  • Upgrade Spark to 4.2.0 to support heterogeneous K8s executor management
  • Promote native acceleration for Spark 4 on Kubernetes
  • Support Java 26
  • Improve Security features and document Security Model
  • Audit transitive dependencies and licenses
  • Add more examples like one-zone Spark Connect Server SparkApp and JWSFilter-enabled SparkCluster

Docker Image

docker pull apache/spark-kubernetes-operator:1.0.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list

Full Changelog:

Resolved Issues

  • [SPARK-56824] Bump version to 1.0.0-SNAPSHOT in main branch
  • [SPARK-56848] Upgrade Apache DataFusion Comet to 0.16.0
  • [SPARK-56849] Use Spark 4.1.1 in Apache DataFusion Comet example
  • [SPARK-56850] Add Spark 4.2.0-preview5 K8s integration tests
  • [SPARK-56851] Remove v1alpha1 version of SparkApp and SparkCluster CRDs
  • [SPARK-56852] Upgrade Gradle to 9.5.1
  • [SPARK-56855] Upgrade kubernetes-client to 7.6.0 and update CRDs for K8s 1.36
  • [SPARK-56930] Upgrade spotless-plugin-gradle to 8.5.1
  • [SPARK-56985] Upgrade Iceberg example to use Iceberg 1.11.0
  • [SPARK-56986] Upgrade java-operator-sdk to 5.3.4
  • [SPARK-56987] Upgrade JUnit to 6.1.0
  • [SPARK-56988] Add dev/spark_jira_utils.py
  • [SPARK-57006] Use Spark 4.1.2 in docs, examples, tests, and default values
  • [SPARK-57012] Drop Spark 3.5 Support
  • [SPARK-57015] Upgrade slf4j to 2.0.18
  • [SPARK-57016] Upgrade spotbugs-gradle-plugin to 6.5.5
  • [SPARK-57017] Upgrade Netty to 4.2.14.Final
  • [SPARK-57078] Upgrade Dropwizard metrics to 4.2.39
  • [SPARK-57079] Add *.pyc and *.pyo to .gitignore
  • [SPARK-57107] Pin apache/skywalking-eyes to v0.8.0 commit SHA
  • [SPARK-57114] Upgrade spotless-plugin-gradle to 8.6.0
  • [SPARK-57147] Add AGENTS.md and symlink CLAUDE.md to it
  • [SPARK-57153] Run CI in forked repositories and report build status to PRs
  • [SPARK-57157] Harden fork CI status workflows against run/check-run race and pagination
  • [SPARK-57272] Upgrade Netty to 4.2.15.Final
  • [SPARK-57280] Add SECURITY.md for scan-agent discoverability
  • [SPARK-57350] Enable spark.kubernetes.executor.useDriverPodIP by default
  • [SPARK-57391] Use Apache Spark 4.0.3 in examples
  • [SPARK-57432] Upgrade spotbugs-gradle-plugin to 6.5.6
  • [SPARK-57433] Upgrade spotbugs-tool to 4.10.2
  • [SPARK-57450] Add JWSFilter-enabled Spark Cluster example
  • [SPARK-57466] Support s.k.o.dynamicConfig.source to use mounted-ConfigMap
  • [SPARK-57492] Use K8s Garbage Collection to delete executors only when restart is disabled
  • [SPARK-57494] Add Pod Security Admission section to SECURITY.md
  • [SPARK-57495] Default driver/master/worker container allowPrivilegeEscalation to false if unset
  • [SPARK-57497] Fix NPE in AppInitStep restart backoff check when stateTransitionHistory is null
  • [SPARK-57508] Upgrade java-operator-sdk to 5.3.5
  • [SPARK-57543] Make ConfigOption dynamic override opt-in and filter dynamic config refresh
  • [SPARK-57612] Upgrade Apache DataFusion Comet to 0.17.0
  • [SPARK-57628] Upgrade Gradle to 9.6.0
  • [SPARK-57629] Upgrade spotless-plugin-gradle to 8.7.0
  • [SPARK-57630] Fix publish_snapshot_chart GitHub Actions job
  • [SPARK-57706] Add one-zone Spark Connect Server example
  • [SPARK-57757] Upgrade Gradle to 9.6.1
  • [SPARK-57788] Prevent Build CI cancellation between different commits in main branch
  • [SPARK-57789] Update and bundle LICENSE-binary and licenses-binary to describe all bundled libraries
  • [SPARK-57794] Exclude 3rd party Maven metadata from shadow jar
  • [SPARK-57796] Exclude org.checkerframework dependency
  • [SPARK-57798] Exclude net.sf.py4j dependency
  • [SPARK-57799] Exclude net.razorvine dependency
  • [SPARK-57801] Upgrade fabric8 to 7.8.0
  • [SPARK-57844] Exclude Hadoop YARN classes from shadow jar
  • [SPARK-57845] Exclude org.roaringbitmap dependency
  • [SPARK-57856] Authenticate minikube GitHub API requests to avoid CI rate limit failures
  • [SPARK-57857] Exclude com.ning dependency
  • [SPARK-57859] Exclude com.esotericsoftware dependency
  • [SPARK-57868] Requeue after driver not found because of the stale cache
  • [SPARK-57871] Exclude Hadoop MapReduce classes from shadow jar
  • [SPARK-57874] Exclude Hadoop MapReduce resources from shadow jar
  • [SPARK-57875] Exclude Hadoop filecache classes from shadow jar
  • [SPARK-57886] Upgrade spotless-plugin-gradle to 8.8.0
  • [SPARK-57887] Upgrade spotbugs-gradle-plugin to 6.5.8
  • [SPARK-57888] Upgrade JUnit to 6.1.1
  • [SPARK-57906] Upgrade java-operator-sdk to 5.4.0
  • [SPARK-57917] Fix workload RoleBinding name in operator namespace
  • [SPARK-57918] Use Files.createTempFile instead of File.createTempFile to ensure owner-only temp file
  • [SPARK-57919] Add gradle/verification-metadata.xml for dependency verification
  • [SPARK-57934] Support opt-in NetworkPolicy for operator pod in Helm chart
  • [SPARK-57940] Enable readOnlyRootFilesystem for operator container in Helm chart
  • [SPARK-57942] Replace wildcard RBAC verbs with explicit verb list in Helm chart
  • [SPARK-57943] MetricsSystem should not expose sink property values
  • [SPARK-57950] Support spark.redaction.regex and redact sensitive values in operator logs
  • [SPARK-57968] Use virtual threads for SentinelManager scheduled executor
  • [SPARK-57969] Use virtual thread for periodic GC scheduler
  • [SPARK-58039] Upgrade Netty to 4.2.16.Final
  • [SPARK-58090] Add doc.crds.dev link to Spark custom resources documentation
  • [SPARK-58101] Remove misleading spark.submit.deployMode=cluster from spark-connect-server-with-spark-cluster.yaml
  • [SPARK-58140] Upgrade Spark to 4.2.0
  • [SPARK-58142] Upgrade JUnit to 6.1.2
  • [SPARK-58143] Upgrade spotbugs-gradle-plugin to 6.5.9
  • [SPARK-58144] Add verification-metadata.xml regeneration guide to AGENTS.md and fix README.md
  • [SPARK-58145] Upgrade Apache Celeborn to 0.6.3
  • [SPARK-58146] Upgrade log4j to 2.26.1
  • [SPARK-58147] Upgrade SpotBugs to 4.10.3
  • [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints
  • [SPARK-58194] Upgrade docker images to Apache Spark 4.2.0
  • [SPARK-58197] Upgrade Java Operator SDK to 5.5.0
  • [SPARK-58284] Remove duplicated NetworkPolicy creation in favor of NetworkPolicyFeatureStep of Spark 4.2
  • [SPARK-58287] Add dev/bump_maintenance_version.py to bump maintenance version with dev postfix
  • [SPARK-58289] Add dev/prepare_release.py to automate release version preparation
Assets 4
Loading