Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement CHECKPOINT for cluster snapshot #1199

Merged
merged 13 commits into from
Dec 4, 2019
Merged

Implement CHECKPOINT for cluster snapshot #1199

merged 13 commits into from
Dec 4, 2019

Conversation

bright-starry-sky
Copy link
Contributor

No description provided.

@nebula-community-bot
Copy link
Member

Can one of the admins verify this patch?

critical27
critical27 reviewed Nov 4, 2019
View reviewed changes
src/kvstore/RocksEngine.cpp Show resolved Hide resolved
src/kvstore/RocksEngine.cpp Show resolved Hide resolved
src/meta/processors/admin/CreateSnapshotProcessor.cpp Outdated Show resolved Hide resolved
dangleptr
dangleptr reviewed Nov 4, 2019
View reviewed changes
src/daemons/MetaDaemon.cpp Outdated Show resolved Hide resolved
src/daemons/StorageDaemon.cpp Outdated Show resolved Hide resolved
src/meta/processors/admin/CreateSnapshotProcessor.cpp Outdated Show resolved Hide resolved
src/storage/StorageServiceHandler.h Outdated Show resolved Hide resolved
dangleptr
dangleptr reviewed Nov 4, 2019
View reviewed changes
src/kvstore/NebulaStore.cpp Outdated Show resolved Hide resolved
src/kvstore/RocksEngine.cpp Show resolved Hide resolved
src/kvstore/RocksEngine.cpp Outdated Show resolved Hide resolved
@bright-starry-sky
Copy link
Contributor Author

Improved checkpoint directory structure as below :
|--FLAGS_data_path
|----nebula
|------space1
|--------data
|--------wal
|--------checkpoints
|----------snapshot1
|----------snapshot2
|----------snapshot3

@bright-starry-sky
Copy link
Contributor Author

Later, I will record the metadata of checkpoint to the metadata server.

wadeliuyi
wadeliuyi reviewed Nov 5, 2019
View reviewed changes
src/kvstore/NebulaStore.h Outdated Show resolved Hide resolved
@bright-starry-sky
Copy link
Contributor Author

1, Recode snapshot info to meta server.
2, Move blocking writes logic to raftex.
3, Addressed comments.

@bright-starry-sky bright-starry-sky added the ready-for-testing PR: ready for the CI test label Nov 11, 2019
@bright-starry-sky
Copy link
Contributor Author

Add cluster status check thread, Used for async rollback after checkpoint creat failed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@bright-starry-sky
Copy link
Contributor Author

Ready to review. come on please .

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

1 similar comment
@nebula-community-bot
Copy link
Member

Unit testing passed.

dangleptr
dangleptr reviewed Nov 18, 2019
View reviewed changes
Copy link
Contributor

@dangleptr dangleptr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After skim the pr. Totally, the code structure looks good to me.
I will check this pr in detail later. Please see my inline comments firstly.

src/kvstore/NebulaStore.cpp Outdated Show resolved Hide resolved
src/kvstore/NebulaStore.cpp Outdated Show resolved Hide resolved
src/meta/processors/admin/SnapShot.cpp Outdated Show resolved Hide resolved
src/meta/processors/admin/SnapShot.cpp Outdated Show resolved Hide resolved
@bright-starry-sky
Copy link
Contributor Author

Addressed @dangleptr's comment:
1:Rename buildLeaderPlan;2:Using part(spaceId, partId) instead of find part;3:remove method asyncBlockingLeader

@nebula-community-bot
Copy link
Member

Unit testing passed.

1 similar comment
@nebula-community-bot
Copy link
Member

Unit testing passed.

@bright-starry-sky
Copy link
Contributor Author

Addressed @Amber1990Zhang and @darionyaphet's comment.

@nebula-community-bot
Copy link
Member

Unit testing passed.

@amber-moe amber-moe mentioned this pull request Dec 4, 2019
Merged
@nebula-community-bot
Copy link
Member

Unit testing passed.

@dangleptr dangleptr merged commit 7b2dfa5 into vesoft-inc:master Dec 4, 2019
@bright-starry-sky bright-starry-sky deleted the snapshotlink branch December 5, 2019 03:00
yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Feb 16, 2020
@bright-starry-sky @dangleptr
Implement CHECKPOINT for cluster snapshot (vesoft-inc#1199)
c22f784 
Support to create checkpoint for the whole cluster. 
We could use command in console to create/drop/list checkpoints.

To restore from some checkpoint,  users should stop the whole cluster and override data dir with related snapshot dir.  Of course, we will give a tool for that later.
tong-hao pushed a commit to tong-hao/nebula that referenced this pull request Jun 1, 2021
@bright-starry-sky @dangleptr
Implement CHECKPOINT for cluster snapshot (vesoft-inc#1199)
b0c7e4f 
Support to create checkpoint for the whole cluster. 
We could use command in console to create/drop/list checkpoints.

To restore from some checkpoint,  users should stop the whole cluster and override data dir with related snapshot dir.  Of course, we will give a tool for that later.
yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Jan 31, 2023
@Shylock-Hg
Fix the compile error. (vesoft-inc#1199)
944e389
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amber-moe amber-moe amber-moe left review comments

@dangleptr dangleptr dangleptr approved these changes

@darionyaphet darionyaphet darionyaphet approved these changes

@liuyu85cn liuyu85cn Awaiting requested review from liuyu85cn

@critical27 critical27 Awaiting requested review from critical27 critical27 is a code owner

@wadeliuyi wadeliuyi Awaiting requested review from wadeliuyi

Assignees
No one assigned
Labels
ready-for-testing PR: ready for the CI test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bright-starry-sky @nebula-community-bot @darionyaphet @critical27 @wadeliuyi @dangleptr @amber-moe