issue #7486 : A dedicated Spark pipeline execution engine#7539
Merged
Conversation
…ution information location)
…a integration tests)
… streams via side loading)
…et streams, multi-input, docs) Add target-stream fan-out for Filter Rows and Switch/Case, union of multiple previous transforms with the same row layout, no-header CSV positional mapping, metrics fix for leaf mapPartitions transforms, integration tests, and user-manual getting-started documentation for the native Spark engine.
…sic I/O, ITs, docs) Support classic file I/O on mapPartitions (accept-filenames, partition-scoped Internal.Transform.*), active-graph collection for disabled hops, integration tests 0007/0008, and a full native Spark getting started guide with screenshots.
Contributor
Author
Update since PR openedFollow-up work on the native Spark engine after the original PR (target streams / multi-input / early docs): Engine / runtime
Integration tests (
|
…house Delta/Iceberg) Add optional open table format support on the native Spark engine: Lake Table Input/Output/Merge/Maintenance, Spark Catalog metadata, connector probe and session plan, Maven -Plakehouse tests, user docs, and sample walkthroughs. Connectors remain operator-installed (not in default assembly).
Contributor
Author
Changelog: Open table formats (lakehouse) series on native SparkThis commit adds optional Delta Lake / Apache Iceberg support on the native Spark pipeline engine (not Beam). Summary of the planned PR stack landed together as Pins
Distribution: connectors are not in the default Hop assembly (docs-only + operator install). Install under PR 1 — Packaging spike + connector probe
PR 2 — Lake Table Input/Output PATH (Delta)
PR 3 — Iceberg PATH Input/Output
PR 4 — Time travel on Input
PR 5 — Spark Catalog + LakeSessionPlan + TABLE mode
PR 6 — MERGE
PR 7 — Maintenance
PR 8 — Docs + samples
How to try# Default tests (no connector download)
./mvnw -pl plugins/engines/spark -am test
# Lakehouse classpath + PATH/MERGE/maintenance ITs
./mvnw -pl plugins/engines/spark -am test -PlakehouseManual overlay: copy Out of scope (optional later / PR 9)
|
… lakehouse connectors) Embed Delta Lake 4.3.1 and Apache Iceberg 1.11.0 in the default engines-spark assembly under lib/delta and lib/iceberg. native-provided fat jars keep these connectors; docs and probe messages updated for shipped packaging.
Contributor
Author
Changelog: PR 9 — ship Delta/Iceberg in default assemblyFollow-up to the lakehouse series (PR 1–8): connectors are now packaged with the native Spark engine by default. What changed
How to verify./mvnw -pl plugins/engines/spark -am package -DskipTests
unzip -l plugins/engines/spark/target/hop-engines-spark-*.zip | grep 'lib/delta\|lib/iceberg'
./mvnw -pl plugins/engines/spark test # 103 tests, includes lakehouse ITsStill optional / out of scope
|
… table MDI) Enable metadata injection on Spark Lake Table Input/Output/Merge/Maintenance so multi-table landings, CDC merges, and fleet maintenance can use template pipelines. Add injection unit tests and document inject keys.
Contributor
Author
Lake Table metadata injection (MDI)Follow-up for complex multi-table / CDC-style work: all four Lake Table transforms are now metadata-injection capable. What
Shared inject keys
Plus transform-specific keys (save mode, merge condition/actions, OPTIMIZE/VACUUM params, …) listed on the transform docs. PatternTemplate lake pipeline → ETL Metadata Injection maps rows from a control/source pipeline → run under Native Spark. |
…house/iceberg ITs) Add integration-tests/lakehouse (Delta) and integration-tests/iceberg projects with PATH Input/Output, overwrite + time travel (Delta VERSION 0), and MERGE upsert workflows. Wire Docker compose and docs cross-links.
Contributor
Author
Integration tests: lakehouse (Delta) + icebergNew Hop IT projects for end-to-end Lake Table transforms on native Spark:
Docker: Local verify (both green, exit 0): export HOP_LOCATION=…/assemblies/client/target/hop
./integration-tests/scripts/run-tests.sh lakehouse
./integration-tests/scripts/run-tests.sh iceberg |
…ssions
Use /tmp/hop-it-{lakehouse,iceberg} for tables and extracts, and replace
DELETE_FILES with a Shell cleanup that always exits 0. Avoids failures
deleting host-owned Spark .crc files on the mounted project volume.
… config fields Introduce MULTI_LINE_TEXT with multiLineTextHeight (lines, default 1) on @GuiWidgetElement, wire creation/layout/bind in GuiCompositeWidgets, and use it for native Spark sparkConfigs (5 lines).
…config Add a Load configuration template button with Local, capped local, standalone, spark-submit, Databricks, YARN, and lakehouse presets. Wire PipelineRunConfigurationEditor button listener and refresh widgets after button actions so form fields update.
…UTTON widgets Add Load catalog template presets (Iceberg Hadoop/REST, Hive, Glue, Nessie, Unity, Delta) with # docs: links in conf extra. Wire SparkCatalogEditor button flush/rebind, multi-line conf extra, and GuiCompositeWidgets afterButtonPressed refresh so form fields update after modal template apply. Document templates in spark-catalog and cross-link from lakehouse/native Spark.
Add workspace connection (PAT) for the Jobs REST API, a RestDatabricksJobsClient with WireMock tests, and a Big Data workflow action to run-now or one-time submit with wait/fire-and-forget and result variables. Document connection and action; nav and metadata index links.
…nSpark JAR) Add Deploy & run: upload fat jar, pipeline, and metadata JSON to DBFS, create or update a JAR job (MainSpark) on an existing cluster, then run-now. JobSpecFactory, HopSparkDeployHelper, client uploadToDbfs, dialog fields, tests, and docs.
Extract shared DatabricksRunWaiter used by Job Run and new Job Wait.
Job Wait polls by run id (default ${DatabricksRunId}) with timeout,
poll interval, cancel-on-stop, and the same result variables. Docs and
nav cross-links for fire-and-forget + wait orchestration.
Document that Spark File/Lake paths use Hadoop URIs (s3a://, hdfs://), not Hop VFS (s3://, named MinIO). Relabel UI fields, fix cluster examples, and append error hints via SparkPathDialect when known Hop-only schemes appear in Dataset I/O failures.
Add an optional from=to path scheme map on the Native Spark run configuration so Dataset and Lake PATH I/O can rewrite schemes (e.g. s3=s3a, minio=s3a) after variable resolution. Classic Hop VFS is unchanged; FS credentials still come from Spark/Hadoop config.
…ster
Add a Native Spark project package (export via Tools menu and hop-conf
--export-spark-project) so Simple Mapping and Pipeline Executor can resolve
${PROJECT_HOME} children on workers. MainSpark accepts --HopProjectPackage;
the engine distributes the zip with SparkContext.addFile and mini-pipelines
materialize PROJECT_HOME from SparkFiles on each executor.
Add spark-mapping-demo sample project, Spark 4.1 compose cluster with shared data volume, prepare-dist and spark-submit-demo scripts, and a user-manual walk-through so Simple Mapping under PROJECT_HOME can be verified on multi-node spark-submit with a project package.
…HOME --project/-p only names a managed project and does not set PROJECT_HOME. Export now accepts --export-spark-project-home and loads metadata from that folder; prepare-dist.sh no longer requires registering the sample.
…aults Open SparkFiles-local package zips via java.io (HopVfs misses bare absolute paths). Default DIST_DIR to /tmp/spark-mapping-demo-dist. Spark 4 master uses --host instead of --ip.
Stop stashing SparkFiles userFiles paths in HOP_SPARK_PROJECT_PACKAGE (not valid on other JVMs). Resolve package from SparkFiles, then $HOP_DATA/packages/<zip>, then the portable URI. Demo submit copies the zip onto the shared volume and passes --files as backup.
Contributor
Author
Follow-up: VFS vs Spark paths, project packages, and cluster mapping demoPushed additional work on this branch after the initial Native Spark engine landings. Summary of improvements: Hop VFS vs Spark/Hadoop path dialects
Native Spark project package (nested pipelines / mappings)
Docker walk-through (end-to-end proof)
VerifiedLocal multi-node Docker submit of Happy path (from repo root): ./plugins/engines/spark/src/samples/spark-mapping-demo/scripts/prepare-dist.sh
docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
up -d --build --scale spark-worker=2
docker compose -f docker/integration-tests/integration-tests-spark-native-cluster.yaml \
exec spark /opt/hop-samples/spark-mapping-demo/scripts/spark-submit-demo.sh |
…ark-demo Add DISTRIBUTED/DRIVER_ONLY generic transform run mode with per-transform override and canvas badge, nested Native Spark session reuse for Pipeline Executor, Hop VFS-safe execution-info cache writes, and rename/expand the cluster sample to spark-demo (workflows, nested pipelines, host hop-data bind mount).
Contributor
Author
Update: Spark run mode, nested pipelines, and spark-demo expansionPushed follow-up work that has been exercised on the Docker Spark cluster. Engine
Execution info
spark-demo (renamed from
Docs
Happy path on the host after a run: ls -la /tmp/spark-demo-dist/hop-data/out
ls -la /tmp/spark-demo-dist/hop-data/executions |
mattcasters
marked this pull request as ready for review
July 17, 2026 20:06
…ummary paths Write temp Excel/ODS outputs under project output/ (world-writable), align container UID defaults with the ASF Jenkins agent identity, and fix relative path bugs when printing passed/failed test summaries from the repo root.
# Conflicts: # integration-tests/scripts/run-tests-docker.sh
…rent folders, hop_server rebuild Clear empty field mappings on PipelineExecutor so child parameter defaults apply (IT main-0003). Harden JsonOutput parent-folder creation and messages for Beam 0005. Force docker compose --build for hop_server so remote export ITs use a server image matching current assemblies.
…ithout GCP key Wipe residual Spark output under project output/ (including other-UID files) before IT runs so delete-output steps succeed. Skip spreadsheet Google Sheets workflows when GCP_KEY_FILE is missing or not a service-account JSON, and surface a clear credentials error from GoogleSheetsCredentials.
…unner The Google Sheets skip only ran on the classic per-workflow path. Spreadsheet ITs default to run-project-tests.hpl, so main-0009/0010 still executed with the dummy GCP key. Force the classic runner for the spreadsheet project when SKIP_GOOGLE_SHEETS=true so those workflows are skipped.
Do not delete tracked .gitkeep/.gitignore under project output/ when wiping residual Spark/UID-mismatched files before integration tests.
LDAP stores setup pipelines in ldap/output/*.hpl. The pre-test output cleanup was deleting everything except .gitkeep/.gitignore, which broke main-ldap-tests. Only remove untracked residuals (git clean) and Spark part-*/.crc artifacts.
… fat jar, library-cache lesson Stage UC Volume project packages to local disk for SparkFiles addFile so executors rematerialize PROJECT_HOME (mappings and nested pipelines). Deploy fat jars as hop-native-<sha12>.jar so Dedicated clusters pick up new library URIs after rebuilds; document overwrite-vs-classpath caching and cluster restart. MainSpark prints pkg-dist fingerprint for verification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a native Apache Spark 4.1.x pipeline engine for Hop batch pipelines (without Apache Beam), so the community can try Dataset-based execution alongside the existing Beam Spark runner.
Highlights
local[*]and cluster /spark-submitviaMainSpark)integration-tests/spark-native/(0001–0006, 0099 complex)Notes for reviewers / try-out
plugins/engines/sparklocal[*]uses Spark libs underplugins/engines/spark/lib--spark-client-version=native-providedandorg.apache.hop.spark.run.MainSparkTest plan
./mvnw -pl plugins/engines/spark testintegration-tests/spark-nativeworkflows (main-0001…main-0006, complex pipeline)local[*]) and run a small pipeline from Hop GUIspark-submitwith native-provided fat jar against a Spark 4.1 clusterDraft PR for early community feedback and try-out before full review/merge.
Related: #7486