Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(remote_wal): add skeleton for remote wal related to datanode #2941

Merged
merged 26 commits into from Dec 20, 2023

Conversation

niebayes
Copy link
Contributor

@niebayes niebayes commented Dec 15, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

  • Allocated wal options, including kafka topic, will be persisted in RegionInfo of DatanodeTableValue, and will be passed to datanodes through RegionCreateRequest. See: feat(remote_wal): add skeleton for remote wal related to meta srv #2933
  • Upon initializing a region server, each region's wal options are decomposed from the RegionInfo in DatanodeTableValue. Those wal options are then merged into region options and passed to region workers.
  • When a region worker receives a region create/open request, it extracts wal options from region options therein. The wal options are used to construct wal entry namespace when replaying memtable. On the other hand, the wal options are stored in MitoRegion to be used during writing to a region.
  • When a region worker receives a RegionWriteRequest, it constructs wal entry namespaces with wal options. Entries are then written into the log store.

image

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@niebayes niebayes self-assigned this Dec 15, 2023
@niebayes niebayes marked this pull request as draft December 15, 2023 06:25
config/datanode.example.toml Show resolved Hide resolved
config/standalone.example.toml Show resolved Hide resolved
src/cmd/src/datanode.rs Outdated Show resolved Hide resolved
src/common/config/src/wal/kafka.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/mito2/src/wal.rs Outdated Show resolved Hide resolved
src/store-api/src/logstore.rs Outdated Show resolved Hide resolved
config/datanode.example.toml Outdated Show resolved Hide resolved
src/cmd/src/datanode.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/mito2/src/region/options.rs Outdated Show resolved Hide resolved
src/store-api/src/logstore/entry.rs Outdated Show resolved Hide resolved
Copy link

codecov bot commented Dec 19, 2023

Codecov Report

Attention: 91 lines in your changes are missing coverage. Please review.

Comparison is base (6ac47e9) 85.85% compared to head (c80369c) 85.70%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2941      +/-   ##
===========================================
- Coverage    85.85%   85.70%   -0.16%     
===========================================
  Files          770      775       +5     
  Lines       123951   124230     +279     
===========================================
+ Hits        106424   106471      +47     
- Misses       17527    17759     +232     

config/datanode.example.toml Show resolved Hide resolved
config/standalone.example.toml Show resolved Hide resolved
src/common/config/src/wal/kafka.rs Outdated Show resolved Hide resolved
src/common/config/src/wal/raft_engine.rs Show resolved Hide resolved
src/common/config/src/wal/raft_engine.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka.rs Show resolved Hide resolved
src/mito2/src/region/options.rs Outdated Show resolved Hide resolved
src/mito2/src/wal.rs Show resolved Hide resolved
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@v0y4g3r v0y4g3r added this pull request to the merge queue Dec 20, 2023
Merged via the queue into develop with commit 9da1f23 Dec 20, 2023
20 checks passed
@v0y4g3r v0y4g3r deleted the feat/remote_wal_skeleton_datanode branch December 20, 2023 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants