Skip to content

Conversation

StefanRRichter
Copy link
Contributor

What is the purpose of the change

This PR combines [FLINK-7719] and [FLINK-7720], which are both related to our efforts to introduce task recovery from local state. [FLINK-7719] makes the checkpoint id part of the task deployment descriptior, which is required so that tasks can figure our the right state for local recovery. [FLINK-7720] introduces state manager components on task manager, task, and stream task level that build upon each other. The purpose of them is unifying the creation of state related elements such as backends. State reporting now goes through those state managers.

Brief change log

[FLINK-7719]

  • JobManagerTaskRestore carries the checkpoint state handles and checkpoint id that the JM wants the task to restore

[FLINK-7720]

  • Introduced TaskExecutorLocalStateStoresManager (mostly as placeholder/mock), TaskLocalStateStore (currently only a placeholder/ mock), TaskStateManager, and StreamTaskStateManager. Those build a hierarchy for state reporting and recovery.
  • All state related elements are now created by the StreamTaskStateManager and owned by the stream operator.

Verifying this change

This change is already covered by existing tests and is currently mostly a refactoring and preparation for future extension. New test cases have been added for implementations of new components. There should currently no observable changes for users.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)

Documentation

  • Does this pull request introduce a new feature? (no)

@StefanRRichter
Copy link
Contributor Author

CC @aljoscha @StephanEwen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants