Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: make ingest and split_region interface compatible with API v2 #12389

Merged
merged 129 commits into from May 16, 2022
Merged
Show file tree
Hide file tree
Changes from 119 commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
94b5a14
wip
pingyu Feb 24, 2022
3c324ed
cdc: implement causal timestamp provider
pingyu Feb 24, 2022
80c9b2c
wip
pingyu Feb 26, 2022
0c8319f
wip
pingyu Feb 26, 2022
6a40169
api_version: add codec for raw key.
pingyu Feb 27, 2022
8aed19f
Merge branch 'master' into feat-rawkv-cdc-api-ver
pingyu Feb 28, 2022
b4109cc
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ca…
pingyu Feb 28, 2022
de61907
Merge branch 'feat-rawkv-cdc-api-ver' into feat-rawkv-cdc-causal-ts
pingyu Feb 28, 2022
de1d226
api_version: improve prefix verification
pingyu Feb 28, 2022
c1818b3
Merge branch 'master' into feat-rawkv-cdc-api-ver
pingyu Feb 28, 2022
e152861
Merge branch 'feat-rawkv-cdc-api-ver' into feat-rawkv-cdc-causal-ts
pingyu Feb 28, 2022
8613440
api_version: refactor to using txn_types::Key
pingyu Feb 28, 2022
a02891a
api_version: address comments.
pingyu Mar 1, 2022
5b7d954
Merge branch 'master' into feat-rawkv-cdc-api-ver
andylokandy Mar 1, 2022
0c68160
Merge remote-tracking branch 'origin/feat-rawkv-cdc-api-ver' into fea…
pingyu Mar 2, 2022
93165c5
wip
pingyu Mar 2, 2022
ba5fbae
Merge branch 'master' into feat-rawkv-cdc-causal-ts
pingyu Mar 3, 2022
648b64e
Merge branch 'master' into feat-rawkv-cdc-api-ver
pingyu Mar 3, 2022
c663c22
Merge branch 'feat-rawkv-cdc-api-ver' into feat-rawkv-cdc-causal-ts
pingyu Mar 3, 2022
d8d6a42
causal_ts: Add causal timestamp component for RawKV CDC
pingyu Mar 3, 2022
e7e6434
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ca…
pingyu Mar 4, 2022
6de9adb
causal_ts: bug fix.
pingyu Mar 4, 2022
f71e1ce
wip
pingyu Mar 4, 2022
d2bf71d
wip
pingyu Mar 4, 2022
fa511ed
rawkv: Add pre-propose callback to append causal timestamp. Enable ca…
pingyu Mar 5, 2022
77aba80
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Mar 7, 2022
c66fc0f
wip
pingyu Mar 7, 2022
68b4268
wip
pingyu Mar 7, 2022
2065e93
api_v2: raw_delete
haojinming Mar 7, 2022
b057a26
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Mar 8, 2022
80f7a30
wip
pingyu Mar 8, 2022
589c846
wip
pingyu Mar 8, 2022
734b0e6
address review comment
haojinming Mar 8, 2022
3190b34
wip
pingyu Mar 8, 2022
95ac455
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Mar 8, 2022
546f955
wip
pingyu Mar 8, 2022
ceb9051
storage: Modify rawkv interfaces for API V2 new encoding.
pingyu Mar 8, 2022
64364c5
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 9, 2022
1de467b
wip
pingyu Mar 9, 2022
74ae821
rawkv: address comments
pingyu Mar 9, 2022
c2cd559
donnot trancate mce padding and ts in RawMvccSnapshot
haojinming Mar 8, 2022
d2c7491
simpily option usage
haojinming Mar 8, 2022
c2965c8
wip
pingyu Mar 9, 2022
73a1f24
wip
pingyu Mar 9, 2022
d4fd0c4
refactor build_pre_propose_cbs
andylokandy Mar 9, 2022
eb5dde9
revert some unexpected changes
andylokandy Mar 9, 2022
2cb7716
wip
pingyu Mar 10, 2022
3b9c53d
storage: Modify raw atomic interfaces for API V2
pingyu Mar 10, 2022
1ba5572
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 10, 2022
01622cd
causal_ts: address comments
pingyu Mar 11, 2022
dda8adf
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 14, 2022
de369f2
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Mar 14, 2022
552dbd4
wip
pingyu Mar 14, 2022
95bae6d
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 16, 2022
7344949
wip
pingyu Mar 16, 2022
679232c
wip
pingyu Mar 17, 2022
31d54ad
causal_ts: Refactor according to new proposal.
pingyu Mar 17, 2022
a6c25d6
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 17, 2022
ed45aa0
causal_ts: fix Cargo.lock.
pingyu Mar 17, 2022
5b4654e
Merge branch 'feat-rawkv-cdc-server' into feat-rawkv-cdc-raw-get-put
pingyu Mar 17, 2022
ab44b41
wip
pingyu Mar 17, 2022
5653d6d
wip
pingyu Mar 18, 2022
edd4054
fix unit test
pingyu Mar 19, 2022
e922d6e
rawkv: fix tls_collect_query
pingyu Mar 19, 2022
12a748e
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 20, 2022
1c9a2cd
fix "fail" version
pingyu Mar 20, 2022
48a345c
address comments
pingyu Mar 21, 2022
32a0934
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 21, 2022
1aa512c
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-se…
pingyu Mar 22, 2022
4ee5ede
causal_ts: Renew on used up
pingyu Mar 22, 2022
8081444
causal_ts: pick "modifies to requests" to this PR
pingyu Mar 22, 2022
d549fcb
causal_ts: fix CI
pingyu Mar 22, 2022
2451b0d
fix CI
pingyu Mar 23, 2022
1b3b1d6
fix CI
pingyu Mar 23, 2022
37d7a7f
fix CI
pingyu Mar 23, 2022
29cfead
Merge branch 'master' into feat-rawkv-cdc-server
pingyu Mar 23, 2022
b27939b
enable causal observer
pingyu Mar 24, 2022
4328824
causal_ts: address comments.
pingyu Mar 25, 2022
1fabdb4
Merge branch 'master' into feat-rawkv-cdc-server
pingyu Mar 29, 2022
7750113
wip
pingyu Mar 29, 2022
c983160
tiny fix
pingyu Mar 29, 2022
948833e
bugfix of encode_raw_key validation
pingyu Mar 29, 2022
ac412ea
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Mar 30, 2022
04d1492
wip
pingyu Mar 30, 2022
8732505
fix merge error
pingyu Mar 30, 2022
49c5f2b
wip
pingyu Mar 31, 2022
90f09ad
wip
pingyu Mar 31, 2022
0843787
fix ut
pingyu Apr 1, 2022
da6d672
wip
pingyu Apr 1, 2022
21430fd
remove match_api_version_template
pingyu Apr 2, 2022
5743ea5
Merge remote-tracking branch 'upstream/master' into feat-rawkv-cdc-ra…
pingyu Apr 2, 2022
673c5dc
fix ut
pingyu Apr 2, 2022
b95982c
add ts for RawWriteBatch to be compatiable with API v2
iosmanthus Apr 7, 2022
fa4805c
Merge branch 'master' into feat-rawkv-cdc-raw-get-put
pingyu Apr 8, 2022
3230099
wip
pingyu Apr 8, 2022
300bba4
migrate to `Api` generic parameter
pingyu Apr 9, 2022
69f1b01
small improve
pingyu Apr 10, 2022
00f8ad2
rebase pingyu's feat-rawkv-cdc-raw-get-put
iosmanthus Apr 19, 2022
0c80079
fix delete miss key encoding
iosmanthus Apr 19, 2022
b2e5773
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 20, 2022
7c42593
remove dead code
iosmanthus Apr 20, 2022
efd2812
make format
iosmanthus Apr 20, 2022
1fde4d5
Merge branch 'master' into add-ts-for-raw-write-batch
iosmanthus Apr 20, 2022
1d930fd
address jinming's comments
iosmanthus Apr 20, 2022
336ae80
address pingyu's comments
iosmanthus Apr 21, 2022
eef045b
make format
iosmanthus Apr 21, 2022
118b72b
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 22, 2022
b3f2219
catch kvproto update
iosmanthus Apr 25, 2022
6df43b7
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 25, 2022
9fecb39
fix cargo test
iosmanthus Apr 25, 2022
e0f7588
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 26, 2022
8a24f69
fix wrong error msg
iosmanthus Apr 28, 2022
2b1c9c9
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 28, 2022
7999bea
fix wrong generic parameter
iosmanthus Apr 29, 2022
30b5490
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus Apr 29, 2022
9154521
Merge branch 'master' of github.com:tikv/tikv into add-ts-for-raw-wri…
iosmanthus May 9, 2022
77a5c69
fix test module imports
iosmanthus May 9, 2022
7438403
make clippy happy
iosmanthus May 9, 2022
682667d
make format
iosmanthus May 9, 2022
d229523
Merge branch 'master' into add-ts-for-raw-write-batch
iosmanthus May 9, 2022
a1b1dbc
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 10, 2022
6e1e7dc
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 11, 2022
07c55e0
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 11, 2022
25503d1
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 12, 2022
5bd5b82
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 12, 2022
4cb63d9
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 12, 2022
bb534f9
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 13, 2022
ad3a062
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 13, 2022
3441d41
Merge branch 'master' into add-ts-for-raw-write-batch
ti-chi-bot May 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion components/error_code/src/sst_importer.rs
Expand Up @@ -20,5 +20,6 @@ define_error_codes!(
FILE_CONFLICT => ("FileConflict", "", ""),
TTL_NOT_ENABLED => ("TtlNotEnabled", "", ""),
TTL_LEN_NOT_EQUALS_TO_PAIRS => ("TtlLenNotEqualsToPairs", "", ""),
INCOMPATIBLE_API_VERSION => ("IncompatibleApiVersion", "", "")
INCOMPATIBLE_API_VERSION => ("IncompatibleApiVersion", "", ""),
INVALID_KEY_MODE => ("InvalidKeyMode", "", "")
);
26 changes: 24 additions & 2 deletions components/sst_importer/src/errors.rs
Expand Up @@ -8,11 +8,11 @@ use encryption::Error as EncryptionError;
use error_code::{self, ErrorCode, ErrorCodeExt};
use futures::channel::oneshot::Canceled;
use grpcio::Error as GrpcError;
use kvproto::import_sstpb;
use kvproto::{import_sstpb, kvrpcpb::ApiVersion};
use tikv_util::codec::Error as CodecError;
use uuid::Error as UuidError;

use crate::metrics::*;
use crate::{metrics::*, sst_writer::SstWriterType};

pub fn error_inc(type_: &str, err: &Error) {
let label = match err {
Expand Down Expand Up @@ -115,6 +115,27 @@ pub enum Error {

#[error("Importing a SST file with imcompatible api version")]
IncompatibleApiVersion,

#[error("Key mode mismatched with the request mode, writer: {:?}, storage: {:?}, key: {}", .writer, .storage_api_version, .key)]
InvalidKeyMode {
writer: SstWriterType,
storage_api_version: ApiVersion,
key: String,
},
}

impl Error {
pub fn invalid_key_mode(
writer: SstWriterType,
storage_api_version: ApiVersion,
key: &[u8],
) -> Self {
Error::InvalidKeyMode {
writer,
storage_api_version,
key: log_wrappers::hex_encode_upper(key),
}
}
}

impl From<String> for Error {
Expand Down Expand Up @@ -159,6 +180,7 @@ impl ErrorCodeExt for Error {
Error::TtlNotEnabled => error_code::sst_importer::TTL_NOT_ENABLED,
Error::TtlLenNotEqualsToPairs => error_code::sst_importer::TTL_LEN_NOT_EQUALS_TO_PAIRS,
Error::IncompatibleApiVersion => error_code::sst_importer::INCOMPATIBLE_API_VERSION,
Error::InvalidKeyMode { .. } => error_code::sst_importer::INVALID_KEY_MODE,
}
}
}
2 changes: 1 addition & 1 deletion components/sst_importer/src/import_file.rs
Expand Up @@ -355,7 +355,7 @@ impl ImportDir {
.check_api_version(&meta_vec, key_manager.clone(), api_version)
.unwrap()
{
panic!("cannot ingest because of imcompatible api version");
panic!("cannot ingest because of incompatible api version");
}

let mut paths = HashMap::new();
Expand Down
2 changes: 2 additions & 0 deletions components/sst_importer/src/sst_importer.rs
Expand Up @@ -44,6 +44,7 @@ pub struct SstImporter {
dir: ImportDir,
key_manager: Option<Arc<DataKeyManager>>,
switcher: ImportModeSwitcher,
// TODO: lift api_version as a type parameter.
api_version: ApiVersion,
compression_types: HashMap<CfName, SstCompressionType>,
file_locks: Arc<DashMap<String, ()>>,
Expand Down Expand Up @@ -700,6 +701,7 @@ impl SstImporter {
default_meta,
write_meta,
self.key_manager.clone(),
self.api_version,
))
}

Expand Down