Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEMO-321] Fix the data skew pass metric mismatch #188

Merged
merged 2 commits into from
Jan 21, 2019

Conversation

sanha
Copy link
Contributor

@sanha sanha commented Jan 20, 2019

JIRA: NEMO-321: Fix the data skew pass metric mismatch

Major changes:

  • Makes DataSkewRuntimePass access to the partitioning logic by moving Partitioner interface and the implementations from runtime executor to runtime common.
    • The mismatch between the data metric produced by MetricCollectionVertex and the data metric understood by DataSkewRuntimePass is caused by the fact that the partitioning logic is hidden from both components.

Minor changes to note:

  • Adds an empty edge from every metric aggregation vertex to the next stage (as a control dependency) to delay the next stage's scheduling. At now, the order of these stages' scheduling depends on luck.
  • Fix the location of MetricCollection property.
  • Remove the DynamicOptimization property (not needed).
  • Re-enable the skewness aware scheduling.

Tests for the changes:

  • Existing integration test and unit tests.

Other comments:

  • N/A.

Closes #188

@sanha sanha self-assigned this Jan 21, 2019
@sanha sanha added the bug Something isn't working label Jan 21, 2019
@johnyangk johnyangk self-requested a review January 21, 2019 05:17
@johnyangk
Copy link
Contributor

Thanks @sanha I'll take a look.

Copy link
Contributor

@johnyangk johnyangk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@johnyangk johnyangk merged commit 2d0f781 into apache:master Jan 21, 2019
@sanha sanha deleted the 321-FixSkewMisMatch branch January 21, 2019 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants