Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat(split): child replica learn parent prepare list and checkpoint #309

Merged
merged 7 commits into from
Sep 20, 2019

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Sep 10, 2019

Simple partition split process

  1. meta receives client partition split request, and change partition count split: meta start partition split #286
  2. replica notices partition count changed during on_config_sync
  3. parent partition create child partition split: parent replica create child replica #291
  4. parent prepare states for child to learn feat(split): parent replica prepare states #299
  5. child partition async learn states from parent
    • copy prepare list
    • apply checkpoint
    • replay private logs
    • learn in-memory mutations
    • catch up parent states while async learn
  6. all child partitions in group finish learn process, parent will stop read and write
  7. meta server register child partitions
  8. child partition active, and parent recover read and write

More partition split discussion in issue #69
This pr solves the part of fifth step of partition split, which is bold in process description.

What this pr solved

  • child_copy_prepare_list
  • child_learn_states
    • async learn is a time-consuming task, so it should execute in replication_long thread pool
    • child replica should apply checkpoint, replay private log and learn in-memory mutations, this pr display the process, implement apply checkpoint and add function definition of other steps
    • when child finish async learn, we should generate a checkpoint to make child states persistent
    • durable current ballot and update last_durable_decree
    • child catch up parents states during execute async learn task
  • partition_split_context
    • is_prepare_list_copied
    • async_learn_tesk
    • is_cleaned function

Tests

  • add unit test for two new functions child_copy_prepare_list and child_learn_states
  • refactor unit test

@hycdong hycdong changed the title split: child replica learn parent prepare list and checkpoint feat(split): child replica learn parent prepare list and checkpoint Sep 10, 2019
src/dist/replication/lib/replica.cpp Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
}

// generate a checkpoint synchronously
err = _app->sync_checkpoint();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里 sync_checkpoint 是什么原因?必须要 checkpoint 吗?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里child已经异步learn完成,包括checkpoint, prepare_list, private_log和in-memory mutation,这时打一个checkpoint相当于标识异步learn完成,而且方便接下来的catch up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

”方便接下来的catch up“ 我没有理解主要是哪些方面方便了?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在异步learn完成之后,child还需要catch up在它进行异步learn过程中parent的states,这时打一个checkpoint,能把之前异步learn的states durable,更新一下last_durable_decree,主要是这个作用

src/dist/replication/lib/replica_split.cpp Outdated Show resolved Hide resolved
src/dist/replication/lib/replica_split.cpp Show resolved Hide resolved
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants