adding retryable to scan #456
Conversation
Signed-off-by: limbooverlambda <schakra1@gmail.com>
There was a problem hiding this comment.
Thanks for your contribution !
The PR overall looks good. And I left some minor suggestions.
Besides, If you would like to verify the correctness when region error happens, consider to use failpoint. Please refer to failpoint_test.rs. It's not a must before the PR is accepted.
src/raw/client.rs
Outdated
| plan::handle_region_error(self.rpc.clone(), err.clone(), store.clone()) | ||
| .await?; | ||
| return if status { | ||
| self.retryable_scan(scan_args.clone()).await |
There was a problem hiding this comment.
Suggest to eliminate the recursion and let caller do the retry, to reduce overhead.
There was a problem hiding this comment.
Removed the recursion, in the initial implementation, I followed the pattern as outlined in
client-rust/src/request/plan.rs
Line 109 in 54fd720
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Signed-off-by: limbooverlambda <schakra1@gmail.com>
|
Hi @pingyu, are there any more changes you need me to make before this change can be merged? Thanks for taking the time to look at this. |
src/raw/client.rs
Outdated
| let scan_args = ScanInnerArgs { | ||
| start_key: current_key.clone(), | ||
| range: range.clone(), | ||
| limit, |
There was a problem hiding this comment.
It would be better to use current_limit here, otherwise it would return kv pairs more than limit. Moreover, the following current_limiit -= kvs.en() would overflow.
There was a problem hiding this comment.
Good catch. Changed.
src/raw/client.rs
Outdated
| current_limit -= kvs.len() as u32; | ||
| result.append(&mut kvs); | ||
| } | ||
| if end_key |
There was a problem hiding this comment.
Nit:
| if end_key | |
| if end_key.is_some_and(|ek| ek <= next_key.as_ref()) |
There was a problem hiding this comment.
Thanks. Changed. end_key had to be cloned however.
| while current_limit > 0 { | ||
| let scan_args = ScanInnerArgs { | ||
| start_key: current_key.clone(), | ||
| range: range.clone(), |
There was a problem hiding this comment.
I think we should scan from start_key. Otherwise if there is region merge during scan, we would get duplicated kv paires, and lose some others if limit is reached.
There was a problem hiding this comment.
I was trying to trace through the logic and from what I understand, we will only be looping if the limit of the scan has not been exhausted. So regardless of a split or merge, won't we be resuming the next scan from the end_key returned by the previous scan call? So if the first scan result returns an end_key "foo", doesn't the system guarantee that if we start the next scan starting from "foo", we are guaranteed to return all results that are lexicographically larger than "foo" and smaller than whatever end_key has been provided. This is regardless of whether the underlying regions are undergoing any churn(due to any splits or merges). There may be gaps in my understanding so will be more than happy to get some more feedback here.
src/raw/client.rs
Outdated
| @@ -953,4 +1016,63 @@ mod tests { | |||
| ); | |||
| Ok(()) | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn test_raw_scan_retryer() -> Result<()> { | |||
There was a problem hiding this comment.
This test case looks a little not necessary to me, as it just check that we call error handler on region error, but introduce more complexity.
The core changes in this PR is the retry on region error, so if we want to verify the correctness, I think we need to simulate such scene.
For example, we first put some little entries, and perform the scan to fill the region cache. Then we put much more (or large) entries to trigger the region split (see tests::common::init()), and scan again. The later scan should meet region errors.
The test case would be a little complex to implement, so it's OK to me if we don't have it in this PR.
There was a problem hiding this comment.
Good point. The current setup, to your point, is a bit cumbersome. I have removed this test case. I will add the test as you outlined in a subsequent PR.
Signed-off-by: limbooverlambda <schakra1@gmail.com>
|
@pingyu Thanks for your feedback. It will be great if you could take another look. |
solve: #455
We were running into an issue with scans where periodically we noticed scans returning empty results for datasets that were present in a cluster. The hypothesis was that the scans were returning empties when the regions are undergoing splits. While trying to reproduce the issue (by issuing splits from pd-ctl), we found out that when there's a region error (epoch_version_mismatch et al), the scan_inner is not triggering any cache invalidations and subsequent retries. The scan simply returns an empty. This PR is fixing the issue by triggering the invalidations and retry for such issues.
@pingyu @ekexium @andylokandy.