*: ignore tombstone stores #4223

overvenus · 2019-02-15T09:52:26Z

What have you changed? (mandatory)

Ignore tombstone stores.

What are the type of the changes? (mandatory)

Bug fix (non-breaking change which fixes an issue)

How has this PR been tested? (mandatory)

Unit tests.

Does this PR affect documentation (docs) update? (mandatory)

No

Does this PR affect tidb-ansible update? (mandatory)

No.

Refer to a related PR or issue link (optional)

Fixes #4221
Cc pingcap/tidb#9269

Signed-off-by: Neil Shen <overvenus@gmail.com>

zhangjinpeng87 · 2019-02-15T12:06:25Z

src/pd/client.rs

+        if store.get_state() != metapb::StoreState::Tombstone {
+            Ok(store)
+        } else {
+            Err(Error::StoreTombstone(format!("{:?}", store)))


Why return error when the store is Tombstone?

Tombstone stores are no longer belongs to the cluster, other TiKVs should ignore such stores. See more pingcap/tidb#9269

seem we don't need to use a string for the StoreTombstone. maybe integer is enough.

PD may return StoreTombston error too, but it does not contain a store id, so I choose string for simplicity.

zhangjinpeng87 · 2019-02-15T12:08:47Z

src/pd/util.rs

    let err = header.get_error();
    match err.get_field_type() {
        ErrorType::ALREADY_BOOTSTRAPPED => Err(Error::ClusterBootstrapped(header.get_cluster_id())),
        ErrorType::NOT_BOOTSTRAPPED => Err(Error::ClusterNotBootstrapped(header.get_cluster_id())),
        ErrorType::INCOMPATIBLE_VERSION => Err(Error::Incompatible),
-        _ => Err(box_err!(err.get_message())),
+        ErrorType::STORE_TOMBSTONE => Err(Error::StoreTombstone(err.get_message().to_owned())),
+        ErrorType::UNKNOWN => Err(box_err!(err.get_message())),


When the UNKNOWN error type will be returned?

It depends on PD.

src/pd/client.rs

cwen0 · 2019-02-16T04:00:18Z

/test

zhangjinpeng87 · 2019-02-16T09:27:26Z

src/pd/client.rs

    }

-    fn get_all_stores(&self) -> Result<Vec<metapb::Store>> {
+    fn get_all_stores(&self, include_tombstone: bool) -> Result<Vec<metapb::Store>> {


I think PD support such API is more elegant. BTW, how lightning interact with this API?

Lightning doesn't interact with TiKV stores directly, Importer does (src/import/import.rs in this PR).

Importer uses this API to get all TiKV nodes to forward the "switch to import mode" or "compact cluster" command. It doesn't affect SST ingestion.

Cc @nolouch What do you think? Is it ok to support this feature in PD side?

I think it's ok to do that.

Signed-off-by: Neil Shen <overvenus@gmail.com>

…ore-tombstone

zhangjinpeng87

LGTM

src/import/client.rs

Signed-off-by: Neil Shen <overvenus@gmail.com>

…ore-tombstone

overvenus · 2019-02-21T05:18:17Z

/run-all-tests

overvenus · 2019-02-22T02:01:31Z

PTAL
/run-all-tests

overvenus · 2019-02-22T03:58:54Z

@huachaohuang PTAL, thanks!

huachaohuang · 2019-02-22T09:18:51Z

When I first added this get_all_stores API, I didn't expect PD to return tombstone stores at all (yes, I implemented that API in PD too, sorry 😂). IMO, it's more elegant to fix this in PD, and we can add an option like include_tombstones in the request if we really need that in the future.

overvenus · 2019-02-22T09:51:29Z

IMO, it's more elegant to fix this in PD, and we can add an option like include_tombstones in the request if we really need that in the future.

Yes, but it does not with to this PR, they all have the same API. Some users already met this bug, so I think we should merge this PR ASAP.

huachaohuang · 2019-02-22T13:20:45Z

@overvenus how about just fixing this in PD?
tikv/pd#1440

kennytm · 2019-02-22T14:25:39Z

Could we have a solution that can be easily cherry-picked to 2.1?

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus · 2019-02-26T05:24:20Z

PTAL, thanks!

zhangjinpeng87

LGTM

overvenus · 2019-02-26T08:56:10Z

/run-all-tests

overvenus · 2019-02-26T11:43:25Z

/run-all-tests

overvenus · 2019-02-26T14:34:25Z

/rebuild

Signed-off-by: Neil Shen <overvenus@gmail.com>

*: ignore tombstone stores

b682c27

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus force-pushed the ignore-tombstone branch from 8bfbdb7 to b682c27 Compare February 15, 2019 09:55

overvenus added the S: WIP label Feb 15, 2019

pd: return an error when get a tombstone store

1243ade

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus removed the S: WIP label Feb 15, 2019

zhangjinpeng87 reviewed Feb 15, 2019

View reviewed changes

overvenus added the type/bugfix Type: PR - Fix a bug label Feb 15, 2019

kennytm reviewed Feb 15, 2019

View reviewed changes

src/pd/client.rs Outdated Show resolved Hide resolved

src/pd/client.rs Outdated Show resolved Hide resolved

Merge branch 'master' into ignore-tombstone

c4c216e

zhangjinpeng87 reviewed Feb 16, 2019

View reviewed changes

overvenus added 2 commits February 18, 2019 11:18

Address comments

fabd9b5

Signed-off-by: Neil Shen <overvenus@gmail.com>

Merge branch 'ignore-tombstone' of github.com:overvenus/tikv into ign…

6474790

…ore-tombstone

zhangjinpeng87 previously approved these changes Feb 18, 2019

View reviewed changes

Merge branch 'master' into ignore-tombstone

a613d6c

Connor1996 reviewed Feb 19, 2019

View reviewed changes

src/import/client.rs Outdated Show resolved Hide resolved

overvenus added 3 commits February 21, 2019 13:15

Merge branch 'master' into ignore-tombstone

e553932

Address comments

fd4910c

Signed-off-by: Neil Shen <overvenus@gmail.com>

Merge branch 'ignore-tombstone' of github.com:overvenus/tikv into ign…

7a271a6

…ore-tombstone

overvenus dismissed zhangjinpeng87’s stale review via 7a271a6 February 21, 2019 05:16

Merge branch 'master' into ignore-tombstone

8128314

zhangjinpeng87 previously approved these changes Feb 22, 2019

View reviewed changes

overvenus requested a review from huachaohuang February 22, 2019 09:52

huachaohuang mentioned this pull request Feb 26, 2019

pd: execlude tombstone stores when getting all stores #4280

Closed

use exclude tombstone stores

0693e95

Signed-off-by: Neil Shen <overvenus@gmail.com>

overvenus dismissed zhangjinpeng87’s stale review via 0693e95 February 26, 2019 02:47

Merge branch 'master' into ignore-tombstone

9aa3a90

overvenus mentioned this pull request Feb 26, 2019

Rollup of 3 cherry-picks #4284

Merged

zhangjinpeng87 approved these changes Feb 26, 2019

View reviewed changes

nolouch approved these changes Feb 26, 2019

View reviewed changes

Merge branch 'master' into ignore-tombstone

7f2f6ba

huachaohuang added the R: release-2.1 label Feb 26, 2019

Merge branch 'master' into ignore-tombstone

cc2cc91

overvenus merged commit 9a66560 into tikv:master Feb 26, 2019

overvenus deleted the ignore-tombstone branch February 26, 2019 15:50

overvenus added a commit to overvenus/tikv that referenced this pull request Feb 26, 2019

*: ignore tombstone stores (tikv#4223)

458f5ad

Signed-off-by: Neil Shen <overvenus@gmail.com>

sticnarf pushed a commit to sticnarf/tikv that referenced this pull request Oct 27, 2019

*: ignore tombstone stores (tikv#4223)

76a2e4d

Signed-off-by: Neil Shen <overvenus@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: ignore tombstone stores #4223

*: ignore tombstone stores #4223

overvenus commented Feb 15, 2019 •

edited

zhangjinpeng87 Feb 15, 2019

overvenus Feb 15, 2019

siddontang Feb 18, 2019

overvenus Feb 18, 2019

zhangjinpeng87 Feb 15, 2019

overvenus Feb 15, 2019

cwen0 commented Feb 16, 2019

zhangjinpeng87 Feb 16, 2019

kennytm Feb 16, 2019

zhangjinpeng87 Feb 17, 2019

overvenus Feb 18, 2019

nolouch Feb 19, 2019

zhangjinpeng87 left a comment

overvenus commented Feb 21, 2019

overvenus commented Feb 22, 2019

overvenus commented Feb 22, 2019

huachaohuang commented Feb 22, 2019

overvenus commented Feb 22, 2019

huachaohuang commented Feb 22, 2019

kennytm commented Feb 22, 2019

overvenus commented Feb 26, 2019

zhangjinpeng87 left a comment

overvenus commented Feb 26, 2019

overvenus commented Feb 26, 2019

overvenus commented Feb 26, 2019

*: ignore tombstone stores #4223

*: ignore tombstone stores #4223

Conversation

overvenus commented Feb 15, 2019 • edited

What have you changed? (mandatory)

What are the type of the changes? (mandatory)

How has this PR been tested? (mandatory)

Does this PR affect documentation (docs) update? (mandatory)

Does this PR affect tidb-ansible update? (mandatory)

Refer to a related PR or issue link (optional)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwen0 commented Feb 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

overvenus commented Feb 21, 2019

overvenus commented Feb 22, 2019

overvenus commented Feb 22, 2019

huachaohuang commented Feb 22, 2019

overvenus commented Feb 22, 2019

huachaohuang commented Feb 22, 2019

kennytm commented Feb 22, 2019

overvenus commented Feb 26, 2019

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

overvenus commented Feb 26, 2019

overvenus commented Feb 26, 2019

overvenus commented Feb 26, 2019

overvenus commented Feb 15, 2019 •

edited