Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#986] improvement(tez): Optimize the method of obtain the vertex id. #990

Merged
merged 1 commit into from
Jul 6, 2023

Conversation

zhengchenyu
Copy link
Collaborator

What changes were proposed in this pull request?

Optimize the method of obtain the vertex id.

Why are the changes needed?

For now, vertex id is extract from vertex name. This way only support the vertex name like "Map 0", "Reduce 1", generally generated from hive.
For tez examples, the vertex name is arbitrary, so we can't get the vertex id. So we need a new way to get vertex id.

Fix: #986

How was this patch tested?

integration test, unit test, test in yarn cluster, test in tez local mode.

@jerqi jerqi changed the title [#986] [Improvement][tez] Optimize the method of obtain the vertex id. [#986] improvement(tez): Optimize the method of obtain the vertex id. Jul 4, 2023
@jerqi jerqi requested a review from lifeSo July 4, 2023 03:34
@codecov-commenter
Copy link

codecov-commenter commented Jul 4, 2023

Codecov Report

Merging #990 (d0c37b4) into master (3bd2315) will increase coverage by 1.10%.
The diff coverage is 38.00%.

@@             Coverage Diff              @@
##             master     #990      +/-   ##
============================================
+ Coverage     53.39%   54.50%   +1.10%     
+ Complexity     2513     2509       -4     
============================================
  Files           376      356      -20     
  Lines         20521    18192    -2329     
  Branches       1764     1774      +10     
============================================
- Hits          10958     9915    -1043     
+ Misses         8875     7656    -1219     
+ Partials        688      621      -67     
Impacted Files Coverage Δ
.../java/org/apache/tez/common/InputContextUtils.java 100.00% <ø> (ø)
.../main/java/org/apache/tez/common/RssTezConfig.java 92.85% <ø> (ø)
...library/common/shuffle/impl/RssTezFetcherTask.java 0.00% <0.00%> (ø)
...tez/runtime/library/input/RssUnorderedKVInput.java 0.00% <0.00%> (ø)
.../library/output/RssOrderedPartitionedKVOutput.java 46.15% <0.00%> (-2.07%) ⬇️
...z/runtime/library/output/RssUnorderedKVOutput.java 46.55% <0.00%> (-2.10%) ⬇️
...ibrary/output/RssUnorderedPartitionedKVOutput.java 46.55% <0.00%> (-2.10%) ⬇️
...library/common/shuffle/impl/RssShuffleManager.java 27.12% <50.00%> (ø)
...untime/library/input/RssOrderedGroupedKVInput.java 47.31% <80.00%> (+1.28%) ⬆️
...c/main/java/org/apache/tez/common/RssTezUtils.java 55.67% <100.00%> (-2.92%) ⬇️
... and 3 more

... and 20 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Collaborator

@lifeSo lifeSo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is great !

Copy link
Contributor

@jerqi jerqi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged to master. Thanks @lifeSo @zhengchenyu

@jerqi jerqi merged commit bdade5c into apache:master Jul 6, 2023
27 checks passed
@zhengchenyu zhengchenyu deleted the tez.vertex.id branch July 12, 2023 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement][tez] Optimize the method of obtain the vertex id.
4 participants