storage: support reverse raw_scan and raw_batch_scan

[kamijin-fanta]

What have you changed? (mandatory)

• support to reverse scan
• to follow the update of kvproto. add reverse option in RawBatchScanRequest pingcap/kvproto#304

What are the type of the changes? (mandatory)

• New feature (non-breaking change which adds functionality)

How has this PR been tested? (mandatory)

yes. I added unit test for raw_batch_scan.

Does this PR affect documentation (docs/docs-cn) update? (mandatory)

no.

Does this PR affect tidb-ansible update? (mandatory)

no.

Refer to a related PR or issue link (optional)

• Backward raw scan #3705
• add reverse option in RawBatchScanRequest pingcap/kvproto#304

Benchmark result if necessary (optional)

Add a few positive/negative examples (optional)

[sre-bot]

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

[MyonKeminta]

Great job!
Please sign-off your commits and run cargo fmt

[kamijin-fanta]

thanks!
I used "sign-off" for the first time. is this ok?

[MyonKeminta]

Yes! And, please also fix the error messages from CircleCI

[MyonKeminta]

In fact, I think it's better if the range to scan is always [start_key, end_key) whenever we are scanning forward or backward, which is more close to our semantics of the term "start_key" and "end_key".
@breeswish @zhangjinpeng1987 How do you think about it?

[kamijin-fanta]

lowerBound / upperBound and startKey / endKey are distinguished in the code.
so, I wrote such a code.

[breezewish]

Actually currently our semantic is that there is a range [lower_bound, upper_bound). when we do forward scan, we scan from lower_bound to upper_bound in the range. When we do reverse scan, we scan from upper_bound to lower_bound in the range. Maybe we can rename start_key to lower_bound and end_key to upper_bound to be less ambiguous?

[MyonKeminta]

@breeswish Seems it's not easy, since you may need to change the fields' name in kvproto

[MyonKeminta]

@breeswish oh, sorry, it seems renaming fields in kvproto won't break compatibility between tikv and rpc client..?

[MyonKeminta]

At least I think the fields of KeyRange should keep their meanings.
Neither should we rename the fields in it.

In coprocessor.proto:

// [start, end)
message KeyRange {
    bytes start = 1;
    bytes end = 2;
}

In kvrpcpb.proto

message KeyRange {
  bytes start_key = 1;
  bytes end_key = 2;
}

If we change them, we may introduce too much changes to TiKV and TiDB.
I think it's better to let [start_key, end_key) equivalent to [lower_bound, upper_bound) now.

[kamijin-fanta]

the consistency between startKey of ScanRequest and startKey of inside is lost.

message ScanRequest {
    Context context = 1;
    bytes start_key = 2;
    uint32 limit = 3;
    uint64 version = 4;
    bool key_only = 5;
    bool reverse = 6;
}

https://github.com/pingcap/kvproto/blob/855d2192cdc79c7353cf36e6b60e049bf436dc8a/proto/kvrpcpb.proto#L94-L101

I think it is easy to understand clearly distinguish between startKey and lowerBound.

[MyonKeminta]

OK. Let's accept this. But please add some comments to explain why start_key is the greater one.

[breezewish]

For the following two ranges, it looks wired in the reverse mode.. The first one is a, b, c and the second one is c, b, a. I prefer to keep the same global total order for all possible ranges in reverse mode, i.e. for every range (a_i, b_i) at index i, we have a_i > a_i+1.

        let ranges = make_ranges(vec![
            (b"a3".to_vec(), b"a".to_vec()),
            (b"b3".to_vec(), b"b".to_vec()),
            (b"c3".to_vec(), b"c".to_vec()),
        ]);

         let ranges = make_ranges(vec![
            (b"c3".to_vec(), vec![]),
            (b"b3".to_vec(), vec![]),
            (b"a3".to_vec(), vec![]),
        ]);

[breezewish]

How do you think @MyonKeminta

[MyonKeminta]

@breeswish IMO, I think both are ok..

[MyonKeminta]

LGTM

[siddontang]

PTAL @breeswish

[breezewish]

Is it possible to specify ranges in reverse mode like below?

        let ranges = make_ranges(vec![
            (b"c3".to_vec(), b"c".to_vec()),
            (b"b3".to_vec(), b"b".to_vec()),
            (b"a3".to_vec(), b"a".to_vec()),
        ]);

[kamijin-fanta]

@breeswish yes

[breezewish]

Okay

[kamijin-fanta]

what is the status of this PR?

[MyonKeminta]

Oh, sorry
/run-integration-tests

[MyonKeminta]

/run-integration-common-test

[MyonKeminta]

/run-integration-common-test

[kamijin-fanta]

Thank you for the merge! 🎉