Skip to content

Conversation

@zhuzhurk
Copy link
Contributor

@zhuzhurk zhuzhurk commented Dec 6, 2018

…locks for finite stream jobs when resources are limited

What is the purpose of the change

This PR add a job config InputDependencyConstraint, which helps to avoid resource deadlocks in scheduling when resources are limited, as described in FLINK-10945.

Brief change log

  • Add InputDependencyConstraint to ExecutionConfig
  • Adjust isConsumable interface in IntermediateResultPartition to fit for the data actual consumable definition
  • Change current execution lazy scheduling logic(in Execution.scheduleOrUpdateConsumers(edges)) to schedule tasks only if the InputDependencyConstraint is satisfied(an interface ExecutionVertex.checkInputDependencyConstraints is added to serve this purpose).

Verifying this change

This change added tests and can be verified as follows:

  • Added IntermediateResultPartitionTest to validate IntermediateResultPartition changes
  • Added ExecutionVertexInputConstraintTest to validate the constraint check logic in ExecutionVertex

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (JavaDocs)

…locks for finite stream jobs when resources are limited
@zentol
Copy link
Contributor

zentol commented Dec 6, 2018

Please open the PR against the master branch.

@zhuzhurk
Copy link
Contributor Author

zhuzhurk commented Dec 6, 2018

Please open the PR against the master branch.

Thanks Zentol for pointing out the fault. The new PR is #7255

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants