Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Recovering WAL records will not add schema to tskv; stores drop_table & remove_tsfamily in WAL #1340

Merged
merged 4 commits into from Jul 14, 2023

Conversation

zipper-meng
Copy link
Member

@zipper-meng zipper-meng commented Jul 10, 2023

Required checklist

  • Sample config files updated (config,meta/config and default config)
  • If there are user-facing changes, the documentation needs to be updated prior to approving the PR( Link )
  • If there are any breaking changes to public APIs, please add the api change label.
  • Signed CLA (if not already signed)

Which issue does this PR close?

Related #1178, but has some differences.

Rationale for this change

WAL changes

  • Simplify WAL write api in kvcore.rs
  • Store drop_table and remove_tsfamily in WAL files.
    • Recovering tskv from WAL write will not create database schemas, and if database&table&tsfamily of a WAL record does not exist in tskv, this WAL record will be ignored.
    • Recovering tskv from WAL drop_table or remove_tsfamily will not do anything, because WAL write that schema not existed in tskv was ignored.

Here is the newest schema of the record data in WAL file:

# type = Write
+------------+------------+------------+------------+--------------+-----------------+-----------+
| 0: 1 byte  | 1: 8 bytes | 9: 4 bytes | 13: 1 byte | 14: 8 bytes  | 22: tenant_size |  n bytes  |
+------------+------------+------------+------------+--------------+-----------------+-----------+
|    type    |  sequence  |  vnode_id  |  precision | tenant_size  |  tenant         |   data    |
+------------+------------+------------+------------+--------------+-----------------+-----------+

# type = DeleteVnode
+------------+------------+------------+-------------+-------------+----------+
| 0: 1 byte  | 1: 8 bytes | 9: 4 bytes | 13: 8 bytes | 21: n bytes | n bytes  |
+------------+------------+------------+-------------+-------------+----------+
|    type    |  sequence  |  vnode_id  | tenant_size |  tenant     | database |
+------------+------------+------------+-------------+-------------+----------+

# type = DeleteTable
+------------+------------+-------------+---------------+-----------------+---------------+---------+
| 0: 1 byte  | 1: 8 bytes | 9: 8 bytes  | 17: 4 bytes   | 21: tenant_size | database_size | n bytes |
+------------+------------+-------------+---------------+-----------------+---------------+---------+
|    type    |  sequence  | tenant_size | database_size |  tenant         |  database     | table   |
+------------+------------+-------------+---------------+-----------------+---------------+---------+

Other improves

  • Fix unit tests in test_kvcore_interface.rs sometime report unexpected error: update table conflict.
  • Fix some errors that holds some strings instanced too earily, for those errors that stores data in heap:
    • Use Option::ok_or_else() instead of Option::ok_or()
    • Use Result::with_context() insead of Result::context()
    • Some others.

Are there any user-facing changes?

@zipper-meng zipper-meng changed the title Feature/wal [Feature] Recovering WAL records will not create schema; stores drop_table & remove_tsfamily in WAL Jul 10, 2023
@zipper-meng zipper-meng changed the title [Feature] Recovering WAL records will not create schema; stores drop_table & remove_tsfamily in WAL [Feature] Recovering WAL records will not add schema to tskv; stores drop_table & remove_tsfamily in WAL Jul 10, 2023
Copy link
Contributor

@roseboy-liu roseboy-liu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • refactor the WAL mod
  • add drop vnode/table entry for WAL
  • support drop vnode/table recover from WAL

tskv/src/database.rs Show resolved Hide resolved
tskv/src/kvcore.rs Show resolved Hide resolved
tskv/src/lib.rs Outdated Show resolved Hide resolved
@zipper-meng zipper-meng merged commit 825dd4e into cnosdb:main Jul 14, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants