Skip to content

Conversation

@zhijiangW
Copy link
Contributor

What is the purpose of the change

In ResultPartitionDeploymentDescriptor (RPDD) the boolean invariant sendScheduleOrUpdateConsumersMessage indicates whether the schedule mode is LAZY_FROM_SOURCES or not. The schedule mode is a global property, that means all the RPDDs of a task should have the same value. So it is better to migrate this invariant from RPDD to TaskDeploymentDescriptor (TDD).

Furthermore, we try to refactor the RPDD via only concerning with basic graph info and shuffle info later, then this schedule related info should also be removed from RPDD and migrated into task level.

Brief change log

  • Migrate the invariant sendScheduleOrUpdateConsumersMessage from RPDD to TDD
  • Modify the related process during creating ResultPartition in task stack

Verifying this change

This change is already covered by existing tests, such as TaskDeploymentDescriptorTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@zhijiangW
Copy link
Contributor Author

Considering future extending schedule module which might result in different sendScheduleOrUpdateConsumers for different edges in graph, we keep the current logic to close this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants