Skip to content

1.1.0

Latest
Latest

Choose a tag to compare

View all tags
@dongjoon-hyun dongjoon-hyun released this 12 Nov 23:59
· 3 commits to main since this release
1.1.0
1faa654

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 1.1.0 (App Version) with Helm chart 1.9.0 (Chart Version).

Notable Features

  • Support K8s v1.35, v1.36, and v1.37
  • Built with Spark 4.3.0
  • Support Apache Spark 4.0, 4.1, 4.2 and 4.3
  • Support Java 27
  • Support Kueue (K8s-native job queueing system)
  • Deprecated v1beta1 CRDs
  • Drop K8s v1.34

Docker Image

docker pull apache/spark-kubernetes-operator:1.1.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list

Full Changelog:

Resolved Issues

  • [SPARK-58288] Bump version to 1.1.0-SNAPSHOT in main branch
  • [SPARK-58299] Upgrade Iceberg example to use Spark 4.1.2
  • [SPARK-58301] Upgrade Apache DataFusion Comet to 0.17.1
  • [SPARK-58303] Declare least-privilege permissions in GitHub Actions workflows
  • [SPARK-58304] Simplify examples to show only feature-specific configurations
  • [SPARK-58306] Use ubuntu-26.04(-arm)? in GitHub Action jobs
  • [SPARK-58331] Use Java-friendly sparkVersion API instead of labels override in SparkAppDriverConf
  • [SPARK-58380] Update Spark 4.1/4.0 examples to use 4.1.3 and 4.0.4 docker images
  • [SPARK-58426] Render each JVM option as its own command element
  • [SPARK-58494] Upgrade spotless-plugin-gradle to 8.9.0
  • [SPARK-58495] Use 7077 for spark named container port in master StatefulSet
  • [SPARK-58527] Update cyclonedx to 3.3.0
  • [SPARK-58565] Run every assertion in the Helm RBAC test hooks
  • [SPARK-58596] Remove spark.dynamicAllocation.shuffleTracking.enabled from examples and docs
  • [SPARK-58597] Add ExecutorPVCResizePlugin example
  • [SPARK-58599] Add Recovery-mode Executor example
  • [SPARK-58649] Allow SparkApplication drivers ingress to SparkCluster workers
  • [SPARK-58654] Upgrade Apache DataFusion Comet to 1.0.0
  • [SPARK-58655] Upgrade Gradle to 9.7.0
  • [SPARK-58668] Upgrade operator-framework to 5.5.1
  • [SPARK-58669] Upgrade spotbugs-gradle-plugin to 6.5.10
  • [SPARK-58670] Upgrade JUnit to 6.1.3
  • [SPARK-58676] Upgrade cyclonedx-gradle-plugin to 3.4.0
  • [SPARK-58835] Enable spark.shuffle.service.enabled by default in SparkCluster
  • [SPARK-58839] Update README.md with YuniKorn 1.9.0
  • [SPARK-58840] Update word-count-celeborn example to use Celeborn 0.7.0 and Spark 4.2.0
  • [SPARK-58952] Upgrade Gradle to 9.7.1
  • [SPARK-58953] Upgrade spotbugs-gradle-plugin to 6.5.11
  • [SPARK-58958] Remove stale entries from verification-metadata.xml
  • [SPARK-59002] Upgrade Netty to 4.2.17.Final
  • [SPARK-59003] Upgrade spotless-plugin-gradle to 8.10.0
  • [SPARK-59064] Update CI to test K8s 1.37
  • [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors
  • [SPARK-59220] Pin Minikube container-runtime to docker to fix CI failures with Minikube 1.39.0
  • [SPARK-59230] Upgrade setup-gradle GitHub Action to v6.3.0
  • [SPARK-59231] Use the latest docker/* GitHub Actions
  • [SPARK-59246] Remove pi-v1beta1.yaml example
  • [SPARK-59262] Add Java 27 to GitHub Actions
  • [SPARK-59265] Upgrade spotless-plugin-gradle to 8.10.1
  • [SPARK-59266] Upgrade Lombok to 1.18.48
  • [SPARK-59267] Upgrade SpotBugs to 4.10.4
  • [SPARK-59344] Update operator-sdk version to 5.6.0
  • [SPARK-59381] Upgrade setup-java to v6
  • [SPARK-59388] Upgrade Vert.x to 4.5.33
  • [SPARK-59392] Upgrade Jackson to 2.22.2
  • [SPARK-59423] Upgrade Netty to 4.2.18.Final
  • [SPARK-59475] Add suspend field to BaseSpec for SparkApplication and SparkCluster
  • [SPARK-59486] Add Kueue v1beta1 Workload CRD model and constants
  • [SPARK-59490] Add KueueWorkloadFactory for SparkApplication and SparkCluster
  • [SPARK-59502] Support K8s Warning events for SparkApplication and SparkCluster failures
  • [SPARK-59503] Upgrade Kueue Workload CRD model to v1beta2
  • [SPARK-59504] Use enabled key convention in Helm values.yaml while honoring legacy enable
  • [SPARK-59519] Add opt-in Kueue RBAC rules to Helm chart
  • [SPARK-59537] Honor operatorRbac.configManagement.create in Helm chart
  • [SPARK-59540] Warn about deprecated Helm enable keys via NOTES.txt
  • [SPARK-59542] Support spec.suspend to hold driver and master/worker creation in AppInitStep and ClusterInitStep
  • [SPARK-59559] Add kueue E2E test for resources without kueue.x-k8s.io/queue-name
  • [SPARK-59576] Support K8s Normal events for all SparkApplication and SparkCluster state transitions
  • [SPARK-59578] Clarify that Kueue integrations.externalFrameworks is not needed in operations.md
  • [SPARK-59584] Upgrade actions/checkout to v7
  • [SPARK-59586] Support spark.kubernetes.operator.events.excludedReasons to skip K8s events by reason
  • [SPARK-59587] Narrow Helm RBAC verbs for events to create|get|patch
  • [SPARK-59588] Add KueueWorkloadUtils to request admission and release Kueue Workloads
  • [SPARK-59593] Add opt-in Kueue Workload informer
  • [SPARK-59594] Release Kueue Workload when AppCleanUpStep releases SparkApplication resources
  • [SPARK-59596] Fix Helm chart configuration table in operations.md to match values.yaml
  • [SPARK-59606] Map SparkCluster pods by spark.operator/spark-cluster-name in SparkClusterReconciler
  • [SPARK-59606][FOLLOWUP] Use spark.operator/spark-cluster-name label selector for SparkCluster pod informer
  • [SPARK-59610] Fill missing master and worker requests of Kueue Workload for SparkCluster
  • [SPARK-59612] Wait for Kueue Workload admission in AppInitStep and ClusterInitStep
  • [SPARK-59635] Requeue immediately when AppRunningStep changes the running state
  • [SPARK-59637] Remove duplicate driver termination check from AppDriverStartObserver
  • [SPARK-59638] Support K8s events for Kueue admission of SparkApplication and SparkCluster
  • [SPARK-59644] Add Kueue ResourceFlavor v1beta2 model
  • [SPARK-59649] Upgrade slf4j to 2.0.19
  • [SPARK-59654] Resolve Kueue ResourceFlavor node labels and tolerations of admitted Workload
  • [SPARK-59655] Fill priorityClassRef and priority of Kueue Workload
  • [SPARK-59665] Upgrade operator-sdk to 5.6.1
  • [SPARK-59667] Do not treat a failed read as a missing secondary resource
  • [SPARK-59676] Use the current branch version as the default JIRA fix version in merge_spark_pr.py
  • [SPARK-59677] Support K8s events for suspended SparkApplication and SparkCluster
  • [SPARK-59679] Upgrade PMD to 7.27.0
  • [SPARK-59680] Do not release the Kueue quota of a running driver or master on a failed read
  • [SPARK-59690] Exclude PMD GodClass rule
  • [SPARK-59692] Upgrade the minimum K8s version to v1.35
  • [SPARK-59694] Do not treat an unanswered request as a permanent failure
  • [SPARK-59700] Add a minimum interval between unchanged K8s event repeats
  • [SPARK-59712] Apply Kueue ResourceFlavor node labels and tolerations to Spark pods
  • [SPARK-59715] Release the Kueue quota of a terminated application with retained resources
  • [SPARK-59720] Stop the infinite status update loop of a failed SparkCluster
  • [SPARK-59729] Detect a terminated driver pod without container statuses
  • [SPARK-59730] Update architecture.md to reflect the current code
  • [SPARK-59732] Terminate as ResourceReleased when resources are force released for restart
  • [SPARK-59734] Deprecate the unused resourceRetainPolicy parameter of ApplicationStatus.terminateOrRestart
  • [SPARK-59738] Install Chainsaw from a pinned release binary in CI
  • [SPARK-59739] Add since Javadoc tag for all public spark-operator-api methods
  • [SPARK-59742] Build the operator image once and share it via artifact in CI
  • [SPARK-59743] Consolidate Helm Tests matrix jobs into a single job
  • [SPARK-59750] Support spec.suspend for a running SparkCluster
  • [SPARK-59753] Upgrade Java Operator SDK to 5.6.2
  • [SPARK-59754] Release the Kueue Workload of suspended Spark(Application|Cluster) regardless of the queue label
  • [SPARK-59758] Release the Kueue Workload of a stopped SparkApplication regardless of the queue label
  • [SPARK-59760] Add spark.kubernetes.operator.kueue.enabled to disable the Kueue integration by default
  • [SPARK-59766] Make a pending Kueue Workload follow the changed or removed queue label
  • [SPARK-59767] Consolidate Kueue integration tests into the kueue E2E test
  • [SPARK-59769] Apply the flavors of an admitted Kueue Workload to a resource whose queue label was removed
  • [SPARK-59770] Add ValidatingAdmissionPolicy to keep the Kueue queue of started Spark(Application|Cluster)
  • [SPARK-59771] Release the Kueue Workload of SparkApplication only after its pods are gone
  • [SPARK-59775] Upgrade Gradle to 9.8.0
  • [SPARK-59780] Use Java 27 toolchain while supporting Java 21+
  • [SPARK-59790] Bound the wait for the terminating pods of a suspended SparkCluster
  • [SPARK-59795] Respect user-specified terminationGracePeriodSeconds of SparkCluster pods
  • [SPARK-59797] Preserve user-specified dnsConfig of SparkCluster worker pods
  • [SPARK-59800] Handle an evicted or deactivated Kueue Workload before requesting the driver or master
  • [SPARK-59803] Handle an evicted or deactivated Kueue Workload of a running SparkCluster
  • [SPARK-59809] Remove the AZUL vendor requirement from Java 27 toolchain
  • [SPARK-59810] Use Ubuntu 26.04 LTS-based gradle image for builder stage
  • [SPARK-59811] Check the live driver of a suspended SparkApplication before releasing its admitted Kueue Workload
  • [SPARK-59814] Clarify when the Submitted K8s event is published in configuration.md
  • [SPARK-59815] Add events E2E test for state transition events of Spark(Application|Cluster)
  • [SPARK-59826] Do not serialize is(Admitted|QuotaReserved|Finished) of Kueue WorkloadStatus
  • [SPARK-59830] Unify K8s event sources into spark-kubernetes-operator
  • [SPARK-59833] Upgrade Jackson to 2.22.3
  • [SPARK-59834] Upgrade the base image to alpine:3.24
  • [SPARK-59836] Remove unused libstdc++ package from Docker image
  • [SPARK-59838] Upgrade Mockito to 5.24.0
  • [SPARK-59846] Record the Kueue PodsReady condition of started Spark(Application|Cluster)
  • [SPARK-59848] Use a sequential stream in PodUtils.isPodReady
  • [SPARK-59866] Support K8s events for Kueue eviction of running SparkApplication
  • [SPARK-59868] Update GitHub Actions to the latest approved versions
  • [SPARK-59870] Clarify the Failed message of SparkCluster and the K8s events docs
  • [SPARK-59876] Suspend and requeue a running SparkCluster on Kueue PodsReadyTimeout
  • [SPARK-59878] Exclude (UWF|NP)_UNWRITTEN_FIELD for test classes in spotbugs_exclude.xml
  • [SPARK-59879] Close PrintWriter with try-with-resources in ConfOptionDocGenerator
  • [SPARK-59892] Honor non-positive spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds as unlimited
  • [SPARK-59895] Add migration_guide.md
  • [SPARK-59903] Upgrade Iceberg example to use Iceberg 1.12.0
  • [SPARK-59926] Do not retry failed K8s event writes
  • [SPARK-59930] Upgrade Kueue to v0.20.0 in CI
  • [SPARK-59931] Simplify the K8s event messages of errors and SchedulingFailure
  • [SPARK-59932] Document Kueue v0.20.0+ as the recommended version
  • [SPARK-59935] Fix Prometheus _sum and quantiles of histograms and timers
  • [SPARK-59937] Recommend spark.driver.timeout instead of Kueue max-exec-time-seconds label
  • [SPARK-59939] Deprecate v1beta1 version of SparkApp and SparkCluster CRDs
  • [SPARK-59953] Add Kueue examples and use them as the Kueue E2E fixtures
  • [SPARK-59957] Use spark for meta.helm.sh/release-name of the dynamic config E2E fixtures
  • [SPARK-59958] Deduplicate E2E fixtures and state transition assertions
  • [SPARK-59960] Fix Helm command examples to use spark release name consistently
  • [SPARK-59963] Fix RBAC to allow Spark driver to create Deployment and scale executors
  • [SPARK-59964] Fix workload RBAC to grant deletecollection to Spark drivers
  • [SPARK-59976] Use spark release name for helm uninstall in watched-namespaces E2E test
  • [SPARK-59979] Fix podLogs selectors and collect all log lines in E2E tests
  • [SPARK-59980] Upgrade spotbugs-gradle-plugin to 6.5.12
  • [SPARK-59984] Parse Boolean config values like Apache Spark and use the default value for null
  • [SPARK-59990] Fix the documented precedence of system properties in configuration.md
  • [SPARK-59991] Remove unused operatorConfiguration.append and fix broken conf/ links
  • [SPARK-60004] Upgrade Apache DataFusion Comet to 1.1.0
  • [SPARK-60006] Upgrade Netty to 4.2.19.Final
  • [SPARK-60007] Upgrade Apache Gluten to 1.7.0
  • [SPARK-60008] Enable K8s events by default
  • [SPARK-60014] Upgrade PMD to 7.28.0
  • [SPARK-60015] Remove JIRA IDs from migration_guide.md
  • [SPARK-60017] Fix Workload access revocation and Suspended downgrade notes
  • [SPARK-60018] Add ApplicationStateSummary.Suspended and ApplicationStatus#resume
  • [SPARK-60020] Fix KubernetesMetricsInterceptor exceptions for namespaces paths and out-of-range response codes
  • [SPARK-60025] Support spec.suspend for a running SparkApplication
  • [SPARK-60039] Upgrade checkstyle to 13.11.0
  • [SPARK-60042] Support Kueue admission check podSetUpdates
  • [SPARK-60052] Suspend and requeue a running SparkApplication on Kueue eviction
  • [SPARK-60056] Split kueue E2E test group into kueue and kueue-pods-ready
  • [SPARK-60057] Add suspendReason to Suspended states
  • [SPARK-60065] Build the operator image without Gradle in the build-image CI job
  • [SPARK-60068] Run unit tests on each matrix JDK and skip linters in build-test CI
  • [SPARK-60080] Add Kueue Admission and Workload Lifecycle section to architecture.md
  • [SPARK-60082] Deduplicate the remaining E2E SparkApplication fixtures
  • [SPARK-60086] Document Kueue v0.20.0+ as the required version
  • [SPARK-60087] Add Kueue feature support table to spark_custom_resources.md
  • [SPARK-60116] Document Kueue default LocalQueue as not supported
  • [SPARK-60184] Fix workload RBAC to allow ExecutorResizePlugin to resize executor pods
  • [SPARK-60191] Add ExecutorResizePlugin E2E test to pi-java25 group
  • [SPARK-60224] Upgrade Gradle to 9.8.1
  • [SPARK-60225] Support dynamic allocation with Kueue by reserving maxExecutors on opt-in
  • [SPARK-60236] Document Kueue v0.20.0+ as the required version in spark_custom_resources.md
  • [SPARK-60284] Use Eclipse Temurin instead of Azul Zulu in CIs and Docker image
  • [SPARK-60287] Support Java 27
  • [SPARK-60291] Drop Java 26 Support
  • [SPARK-60302] Change the default value of spark.redaction.regex like Spark 4.3.0
  • [SPARK-60307] Exclude org.fusesource.leveldbjni additionally
  • [SPARK-60401] Stop the infinite status update loop of a ClientMode SparkApplication
  • [SPARK-60402] Ignore sidecar containers when observing the driver pod
  • [SPARK-60408] Migrate notify_test_workflow.yml from pull_request_target to workflow_run
  • [SPARK-60427] Update GitHub Actions to the latest approved versions and pin setup-minikube to a commit SHA
  • [SPARK-60429] Fix Helm chart to create sentinel resources in all sentinel namespaces
  • [SPARK-60438] Add the name getters of the Services and NetworkPolicy of SparkCluster
  • [SPARK-60463] Release the resources of a SparkCluster on SchedulingFailure and end it as ResourceReleased
  • [SPARK-60475] Delete the HPA and PDB of a SparkCluster after releasing its Kueue Workload
  • [SPARK-60512] Fix kueue E2E test flakiness by making QuotaReserved assertions nil-safe
  • [SPARK-60513] Fix LICENSE-binary, NOTICE-binary and licenses-binary to match the bundled libraries
  • [SPARK-60626] Upgrade Spark to 4.2.1
  • [SPARK-60630] Upgrade Java Operator SDK to 5.6.3
  • [SPARK-60638] Upgrade spotless-plugin-gradle to 8.10.4
  • [SPARK-60660] Upgrade SpotBugs to 4.10.5 and spotbugs-gradle-plugin to 6.5.13
  • [SPARK-60793] Upgrade Spark to 4.3.0
  • [SPARK-60795] Upgrade fabric8 to 7.9.0 and update CRDs for K8s 1.37
  • [SPARK-60798] Upgrade Jackson to 2.22.4
  • [SPARK-60802] Upgrade Gradle to 9.8.2
  • [SPARK-60815] Upgrade JaCoCo to 0.8.16 for Java 27
  • [SPARK-61105] Upgrade docker images to Apache Spark 4.3.0, 4.1.4, and 4.0.5
Assets 4
Loading