Skip to content

[Improvement] [tez]:Optimize user switch to shuffle mode local/remote. #1024

Description

@baitian77

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

Currently, users need to set the parameter tez.am.launch.launch.cmd-opts to switch between local shuffle and remote shuffle, but this parameter is difficult for users to understand.
We need a simpler and easier to understand parameter,such as tez.shuffle.mode = local/remote

How should we improve?

  1. set tez.am.launch.launch.cmd-opts=-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster。 【 note: this config will not change 】
  2. add new conf tez.shuffle.mode which will used in RssDAGAppMaster. when this conf equal local, RssDAGAppMaster will use all DAGAppMaster functions

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions