Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](spark-load)support dynamic set env #12276

Merged
merged 2 commits into from Sep 7, 2022

Conversation

chenlinzhong
Copy link
Contributor

@chenlinzhong chenlinzhong commented Sep 1, 2022

Proposed changes

Issue Number: close #xxx

Problem summary

Fe submits tasks to spark through the spark submit client, which needs two environment variables Hadoop_ user_ name hadoop_ user_ The two variables password do not support dynamic settings now. I want these two variables to support dynamic settings so that different jobs can be submitted with different accounts

fe通过spark-submit客户提交任务到spark,这个客户端需要两个环境变量hadoop_user_name hadoop_user_password 这两个变量现在不支持动态设置,我想要这两个变量支持动态设置,这样不同的job可以使用不同的账号来提交

前缀为env. 开头的变量为环境变量,hadoop认证方式不是simple的不用填

CREATE EXTERNAL RESOURCE "test_env"  PROPERTIES(
 "type" = "spark",
 "spark.master" = "yarn",
"spark.executor.memory"="8g",
"spark.master"="yarn",
"spark.driver.extraLibraryPath"="/usr/lib/hadoop/lzo/lib:/usr/lib/hadoop/lib/native",
"env.HADOOP_USER_NAME"="user",
"env.HADOOP_USER_PASSWORD"="password"
);

Describe your changes.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@chenlinzhong chenlinzhong changed the title [enhancement](spark-load)support dynamic set env and display spark appid [enhancement](spark-load)support dynamic set env Sep 1, 2022
@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Sep 1, 2022
@spaces-X spaces-X added the area/spark-load Issues or PRs related to the spark load label Sep 1, 2022
@@ -103,19 +104,22 @@ public static DeployMode fromString(String deployMode) {
// broker username and password
@SerializedName(value = "brokerProperties")
private Map<String, String> brokerProperties;
@SerializedName(value = "envConfigs")
private Map<String, String> envConfigs;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether the envConfigs can be overwrited in users' load sqls.
If so, please add the corresponding variable and update logic in ResourceDesc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be overwrite,the code is in updateProperties function

spaces-X
spaces-X previously approved these changes Sep 2, 2022
Copy link
Contributor

@spaces-X spaces-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 2, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2022

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2022

PR approved by anyone and no changes requested.

@hf200012
Copy link
Contributor

hf200012 commented Sep 2, 2022

@chenlinzhong rebase

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 5, 2022
@chenlinzhong
Copy link
Contributor Author

@chenlinzhong rebase

done

@yangzhg yangzhg reopened this Sep 6, 2022
Copy link
Member

@yangzhg yangzhg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yangzhg yangzhg merged commit 941bda5 into apache:master Sep 7, 2022
chenlinzhong added a commit to chenlinzhong/incubator-doris that referenced this pull request Oct 26, 2022
* [enhancement](spark-load)support dynamic set env and display spark appid

* [enhancement](spark-load)support dynamic set env
@morningman morningman added dev/merged-1.1.4 usercase Important user case type label and removed dev/1.1.4-deprecated labels Oct 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/load Issues or PRs related to all kinds of load area/spark-load Issues or PRs related to the spark load dev/merged-1.1.4 reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants