Skip to content

[feat][EC][spark] add support for setting driver parameters to driver side in spark engine#5398

Merged
casionone merged 1 commit intoapache:masterfrom
v-kkhuang:master-sup-spark-driver-conf
Apr 2, 2026
Merged

[feat][EC][spark] add support for setting driver parameters to driver side in spark engine#5398
casionone merged 1 commit intoapache:masterfrom
v-kkhuang:master-sup-spark-driver-conf

Conversation

@v-kkhuang
Copy link
Copy Markdown
Contributor

What is the purpose of the change

Background/Problem:
Currently, the Spark engine in Linkis does not fully support passing driver-specific parameters to the driver side. Users need to configure driver parameters through complex workarounds or manual configuration files, which reduces usability.

Purpose of Change:
To address this problem, this PR adds support for setting driver parameters directly to the driver side in the Spark engine, allowing users to configure driver-specific settings more conveniently through Linkis configuration.

Value/Impact:
After the change, users can pass driver parameters such as memory, cores, or Java options directly through Linkis configuration, improving flexibility and ease of use for Spark task submissions.

Related issues/PRs

Related issues: close #5397
Related pr:none

Brief change log

  • Add driver parameter configuration support to SparkConfiguration
  • Update SparkEngineConnExecutor to pass driver parameters correctly to driver side
  • Add configuration tests for spark executor parameters
  • Add documentation including design, requirements, and test cases

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible
  • If this is a code change: I have written unit tests to fully verify the new behavior.

Copy link
Copy Markdown
Contributor

@casionone casionone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@casionone casionone merged commit cb5d91a into apache:master Apr 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature][EC][spark] add support for setting driver parameters to driver side in spark engine

2 participants