feat: maintain the logidx-seqno in SST files #246
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
longfar-ncy
commented
Mar 30, 2024
•
longfar-ncy
commented
- Storage 中的 Redis 类封装了一个 RocksDB, 在 Redis 对象内增加一个队列，该队列保存了 raft applied log index 和 RocksDB 内所有 CF sequence number 的映射关系关系
- 每次 apply 写入成功后向该队列追加 log index 和 sequence number 的映射
- 增加一个 prop, 产生新 SST 文件的时候，通过该队列找到该 SST 文件中最大 sequence number 对应的 log index，并写入该 SST 文件的 prop meta block 进行持久化.
- 监听 Flush 事件, 每次 Flush 完成后，清理队列中不再有用的映射关系.
longfar-ncy
force-pushed
the
feat/logidx-seqno
branch
4 times, most recently
from
834534d
to
56d8abd
Compare
dingxiaoshuai123
approved these changes
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
| @@ -42,6 +42,7 @@ enum ColumnFamilyIndex { | |||
| kZsetsMetaCF = 7, | |||
| kZsetsDataCF = 8, | |||
| kZsetsScoreCF = 9, | |||
| kColumnFamilyNum, | |||
AlexStocks
reviewed
Apr 10, 2024
src/storage/src/log_index.cc
Outdated
| auto res = sscanf(s.c_str(), "%" PRIi64 "/%" PRIu64 "", &applied_log_index, &largest_seqno); | ||
| assert(res == 2); | ||
|
|
||
| LogIndexAndSequencePair p(applied_log_index, largest_seqno); |
There was a problem hiding this comment.
return LogIndexAndSequencePair(applied_log_index, largest_seqno);
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
AlexStocks
reviewed
Apr 10, 2024
hotjump
reviewed
Apr 11, 2024
src/storage/src/log_index.h
Outdated
| // cf_[cf_id].applied_log_index = std::max(cf_[cf_id].applied_log_index.load(), cur_log_index); | ||
| // return cur_log_index == cf_[cf_id].applied_log_index.load(); | ||
| // } | ||
| bool IsAppliedOrUpdate(size_t cf_id, LogIndex cur_log_index) { |
There was a problem hiding this comment.
IsApplied可能就够了，就是命名上还是有点问题，会误以为只是判断实际上还有隐式的置值。
hotjump
reviewed
Apr 11, 2024
src/storage/src/storage.cc
Outdated
| for (const auto& entry : log.entries()) { | ||
| // If the log has been applied, skip it. Otherwise, update the logidx of the column family. | ||
| if (inst->IsRestarting() && inst->IsAppliedOrUpdate(entry.cf_idx(), log_idx)) { |
There was a problem hiding this comment.
这里可以加个hint unlikely()，表示正常情况不会走进来。
longfar-ncy
force-pushed
the
feat/logidx-seqno
branch
2 times, most recently
from
4e7ade7
to
59d5a2b
Compare
This reverts commit 59d5a2b.
longfar-ncy
force-pushed
the
feat/logidx-seqno
branch
from
20123ef
to
6b3d2fa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.