*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen #8081

MyonKeminta · 2018-10-28T13:21:20Z

What problem does this PR solve?

When TiDB invokes kv_scan interface of TiKV, it provides an start_key and a limit (which means at most how many keys are retrieved in one RPC call). However, it may scan to a range that is already deleted.
The data in there ranges is undefined, which may cause some trouble. This PR tries to add an end_key limit to it.

What is changed and how it works?

This PR added end_key limit to the kv scan interface, and added the upper_bound parameter to the scanner of tikv client.

Meanwhile, the Seek method of Retriever interface returns an Iterator. The end_key limit is perfectly suitable to be passed when an Iterator is creating, however it weird that the end_key parameter occurs in an function named Seek. Since it returns an Iterator, I think it's better to be renamed to Iter ratherthan Seek. In fact, the changes are just similar to #7581 .

This PR depends on pingcap/kvproto#306 , tikv/tikv#3720 and #8178 . Also, this PR will be separated into multiple after merging the PR of kvproto.

Check List

Unit test (TODO)

Code changes

Has interface methods change
Seek and SeekReverse of Retriever are renamed to Iter and IterReverse

Side effects

None

Related changes

Need to be included in the release note (maybe)

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

kv/memdb_buffer.go

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta · 2018-10-29T11:53:12Z

/run-integration-tests

MyonKeminta · 2018-10-29T12:18:48Z

/run-integration-common-test tikv=pr/3720

disksing · 2018-11-05T06:19:00Z

ddl/delete_range.go

@@ -154,7 +154,7 @@ func (dr *delRange) doTask(ctx sessionctx.Context, r util.DelRangeTask) error {
 		finish := true
 		dr.keys = dr.keys[:0]
 		err := kv.RunInNewTxn(dr.store, false, func(txn kv.Transaction) error {
-			iter, err := txn.Seek(oldStartKey)
+			iter, err := txn.Iter(oldStartKey, nil)


r.EndKey?

disksing · 2018-11-05T06:38:19Z

store/tikv/scan.go

@@ -199,7 +202,7 @@ func (s *Scanner) getData(bo *Backoffer) error {
 			// No more data in current Region. Next getData() starts
 			// from current Region's endKey.
 			s.nextStartKey = loc.EndKey
-			if len(loc.EndKey) == 0 {
+			if len(loc.EndKey) == 0 || (len(s.endKey) > 0 && kv.Key(s.nextStartKey).Cmp(kv.Key(s.endKey)) >= 0) {


I think you want to check endKey in Scanner.Next() too.

I think the underlying TiKV or mock TiKV here should promise that it doesn't return any keys that >= endKey here, so I didn't do any more check. How do you think?

It makes sense.

Then TiDB can't work with older TiKV normally. Do you think that's ok?

Good point. I'd like to say it is acceptable -- it won't break anything because current usages do not rely on the iterator to stop at upperBound. But it could be more safe to check bound in Next.

disksing · 2018-11-05T06:41:36Z

table/tables/index.go

@@ -240,7 +240,7 @@ func (c *index) Delete(sc *stmtctx.StatementContext, m kv.Mutator, indexedValues

 // Drop removes the KV index from store.
 func (c *index) Drop(rm kv.RetrieverMutator) error {
-	it, err := rm.Seek(c.prefix)
+	it, err := rm.Iter(c.prefix, nil)


Do wee need to set endKey to TableIndexPrefix(index.ID+1)?

disksing · 2018-11-05T06:43:04Z

structure/hash.go

@@ -238,7 +238,7 @@ func (t *TxStructure) HClear(key []byte) error {

 func (t *TxStructure) iterateHash(key []byte, fn func(k []byte, v []byte) error) error {
 	dataPrefix := t.hashDataKeyPrefix(key)
-	it, err := t.reader.Seek(dataPrefix)
+	it, err := t.reader.Iter(dataPrefix, nil)


Do we need to set endKey to dataPrefix.PrefixNext()?

disksing · 2018-11-05T06:44:02Z

table/tables/tables.go

@@ -782,7 +782,7 @@ func (t *tableCommon) buildIndexForRow(ctx sessionctx.Context, rm kv.RetrieverMu
 // IterRecords implements table.Table IterRecords interface.
 func (t *tableCommon) IterRecords(ctx sessionctx.Context, startKey kv.Key, cols []*table.Column,
 	fn table.RecordIterFunc) error {
-	it, err := ctx.Txn().Seek(startKey)
+	it, err := ctx.Txn().Iter(startKey, nil)


Do we want to set endKey to table's index prefix?

disksing · 2018-11-05T06:44:29Z

table/tables/tables.go

@@ -912,7 +912,7 @@ func (t *tableCommon) RebaseAutoID(ctx sessionctx.Context, newBase int64, isSetS
 // Seek implements table.Table Seek interface.
 func (t *tableCommon) Seek(ctx sessionctx.Context, h int64) (int64, bool, error) {
 	seekKey := tablecodec.EncodeRowKeyWithHandle(t.physicalTableID, h)
-	iter, err := ctx.Txn().Seek(seekKey)
+	iter, err := ctx.Txn().Iter(seekKey, nil)


Do we want to set endKey to table's index prefix?

disksing · 2018-11-05T06:45:25Z

util/prefix_helper.go

@@ -26,7 +26,7 @@ import (

 // ScanMetaWithPrefix scans metadata with the prefix.
 func ScanMetaWithPrefix(retriever kv.Retriever, prefix kv.Key, filter func(kv.Key, []byte) bool) error {
-	iter, err := retriever.Seek(prefix)
+	iter, err := retriever.Iter(prefix, nil)


Set endKey to prefix.PrefixNext()?

disksing · 2018-11-05T06:45:39Z

util/prefix_helper.go

@@ -56,7 +56,7 @@ func ScanMetaWithPrefix(retriever kv.Retriever, prefix kv.Key, filter func(kv.Ke
 // DelKeyWithPrefix deletes keys with prefix.
 func DelKeyWithPrefix(rm kv.RetrieverMutator, prefix kv.Key) error {
 	var keys []kv.Key
-	iter, err := rm.Seek(prefix)
+	iter, err := rm.Iter(prefix, nil)


Set endKey to prefix.PrefixNext()?

ddl/index.go

zhangjinpeng87 · 2018-11-05T08:03:30Z

kv/memdb_buffer.go

-	}
+// Iter creates an Iterator.
+func (m *memDbBuffer) Iter(k Key, upperBound Key) (Iterator, error) {
+	i := &memDbIter{iter: m.db.NewIterator(&util.Range{Start: []byte(k), Limit: []byte(upperBound)}), reverse: false}


How about k or upperBound is nil?

It's ok. If k is nil then []byte(k) is still nil.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta · 2018-11-06T07:11:46Z

@disksing It seems those usages of Seek can be updated in more PRs in the future, or this PR may change too much logic, which causes more difficulty in reviewing.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…kv-scan-end-key

MyonKeminta · 2018-11-08T11:58:40Z

/run-all-tests

MyonKeminta · 2018-11-08T12:00:31Z

@disksing @tiancaiamao PTAL

MyonKeminta · 2018-11-08T12:06:48Z

@winkyao PTAL

MyonKeminta · 2018-11-08T12:18:08Z

Is there any one who can help me remove the DNM tag? 😭

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

shenli · 2018-11-08T21:10:50Z

/run-all-tests

disksing · 2018-11-09T03:10:47Z

LGTM.

MyonKeminta · 2018-11-09T03:30:38Z

/run-unit-test

tiancaiamao · 2018-11-09T04:35:28Z

LGTM @winkyao

winkyao

LGTM

MyonKeminta · 2018-11-09T09:17:37Z

if there's no more problems , is there any one who can help me merge it?

…re no overbound scan will happen (pingcap#8081)

…re no overbound scan will happen (#8081) (#8257)

…re no overbound scan will happen (pingcap#8081) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…re no overbound scan will happen (#8081) (#8310)

MyonKeminta and others added 5 commits October 28, 2018 17:36

Update vendor

8f24d6f

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

seek bound initial commit

aa23f25

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

fix potential log error

3995026

Use Key directly as the arg type

5c12af2

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Rename Seek to Iter

3d118e3

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta mentioned this pull request Oct 28, 2018

service/kv: Add end_key limit to kv_scan interface tikv/tikv#3720

Merged

morgo added the status/DNM label Oct 28, 2018

MyonKeminta commented Oct 29, 2018

View reviewed changes

kv/memdb_buffer.go Outdated Show resolved Hide resolved

MyonKeminta added 2 commits October 29, 2018 16:56

Fix tests

4a20ceb

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

use upperbound in membuffer

6f92d47

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

disksing reviewed Nov 5, 2018

View reviewed changes

zhangjinpeng87 reviewed Nov 5, 2018

View reviewed changes

ddl/index.go Outdated Show resolved Hide resolved

zhangjinpeng87 reviewed Nov 5, 2018

View reviewed changes

ddl/index.go Outdated Show resolved Hide resolved

zhangjinpeng87 reviewed Nov 5, 2018

View reviewed changes

MyonKeminta added 2 commits November 5, 2018 16:44

Update kvproto

e64e7c9

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Add end_key param to ticlient scanner

d6ee090

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta mentioned this pull request Nov 5, 2018

tikvclient: Add endKey param to Scanner #8178

Merged

MyonKeminta added 3 commits November 5, 2018 17:46

Use endKey in mock tikv

a059b8f

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge branch 'master' into misono/kv-scan-end-key

9c20c8c

Merge branch 'misono/ticlient-scan-end-key' into misono/kv-scan-end-key

972ff01

MyonKeminta added 4 commits November 6, 2018 15:45

Remove the -1

63fdd1c

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Do not change logic

f8426e5

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Test how scan end_key of mvcc_leveldb works

6e893fc

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Add upperbound-limited scan test to the ScanTest

aae45bc

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta force-pushed the misono/kv-scan-end-key branch from def5e25 to aae45bc Compare November 6, 2018 10:02

MyonKeminta added 2 commits November 6, 2018 18:02

Recover changes

cf6c377

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Make use of the upperBound limit every where

b8326f3

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta changed the title ~~store/tikv: Add end_key limit to kv_scan interface~~ *: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen Nov 8, 2018

Merge branch 'master' of https://github.com/pingcap/tidb into misono/…

d9ed06f

…kv-scan-end-key

XuHuaiyu removed the status/DNM label Nov 8, 2018

MyonKeminta and others added 2 commits November 8, 2018 20:46

Check key in Next

11f4b36

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge branch 'master' into misono/kv-scan-end-key

fc5bea3

disksing added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 9, 2018

tiancaiamao added the priority/release-blocker This PR blocks a release. Please review it ASAP. label Nov 9, 2018

MyonKeminta mentioned this pull request Nov 9, 2018

tikvclient: Add endKey param to Scanner (#8178) #8247

Merged

2 tasks

winkyao approved these changes Nov 9, 2018

View reviewed changes

Merge branch 'master' into misono/kv-scan-end-key

02ee433

zz-jason added status/LGT2 Indicates that a PR has LGTM 2. status/all tests passed and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 9, 2018

zz-jason merged commit 09c6bff into pingcap:master Nov 9, 2018

MyonKeminta deleted the misono/kv-scan-end-key branch November 9, 2018 11:34

MyonKeminta added a commit to MyonKeminta/tidb that referenced this pull request Nov 9, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensu…

3159f34

…re no overbound scan will happen (pingcap#8081)

MyonKeminta mentioned this pull request Nov 9, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen (#8081) #8257

Merged

zz-jason pushed a commit that referenced this pull request Nov 12, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensu…

21330da

…re no overbound scan will happen (#8081) (#8257)

MyonKeminta added a commit to MyonKeminta/tidb that referenced this pull request Nov 14, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensu…

4a3706c

…re no overbound scan will happen (pingcap#8081) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta mentioned this pull request Nov 14, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen (#8081) #8310

Merged

winkyao pushed a commit that referenced this pull request Nov 19, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensu…

51b4882

…re no overbound scan will happen (#8081) (#8310)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen #8081

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen #8081

MyonKeminta commented Oct 28, 2018 •

edited

MyonKeminta commented Oct 29, 2018

MyonKeminta commented Oct 29, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

MyonKeminta Nov 6, 2018

disksing Nov 6, 2018

MyonKeminta Nov 7, 2018

disksing Nov 7, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

disksing Nov 5, 2018

zhangjinpeng87 Nov 5, 2018

MyonKeminta Nov 6, 2018

MyonKeminta commented Nov 6, 2018 •

edited

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

shenli commented Nov 8, 2018

disksing commented Nov 9, 2018

MyonKeminta commented Nov 9, 2018

tiancaiamao commented Nov 9, 2018

winkyao left a comment

MyonKeminta commented Nov 9, 2018

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen #8081

*: Make use of the upperBound of ticlient's kv_scan interface to ensure no overbound scan will happen #8081

Conversation

MyonKeminta commented Oct 28, 2018 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

MyonKeminta commented Oct 29, 2018

MyonKeminta commented Oct 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MyonKeminta commented Nov 6, 2018 • edited

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

MyonKeminta commented Nov 8, 2018

shenli commented Nov 8, 2018

disksing commented Nov 9, 2018

MyonKeminta commented Nov 9, 2018

tiancaiamao commented Nov 9, 2018

winkyao left a comment

Choose a reason for hiding this comment

MyonKeminta commented Nov 9, 2018

MyonKeminta commented Oct 28, 2018 •

edited

MyonKeminta commented Nov 6, 2018 •

edited