In some scenarios, the ‘sendSnapshot’ does not really transfer a Snapshot

[developer-zc]

Describe the bug (required)
as the title

Your Environments (required)

• nebula 3.1 ~ 3.4

How To Reproduce(required)

Steps to reproduce the behavior:
Supposed we have a space with 1 partition with 3 replica factor, partA(leader)/partB/partC.
The initial status is that Storaged which holds partC is stopped, then put some graph data into partA&partB, and some WAL that expired are cleaned .

1. The initial status is that Storaged which holds partC is stopped, then put some graph data into partA&partB;
2. Wait util some WAL which expired are cleaned, then stop the partA && partB, and start the partC;
3. Start partA(or partB), the sendSnapshot operation is then triggered, the process will go smoothly, but the actual transmission is the status machine real-time data, not from a snapshot;

Underlying reason

1. PartA just started, no log committed in the term, so RaftPart::commitInThisTerm_ is false;
2. In NebulaStore::GetSnapshot process, NebulaStore::checkLeader will be invoked, and the result is false, because RaftPart::commitInThisTerm_ == false;
3. The final result of NebulaStore::GetSnapshot is nullptr;

Proposal

1. Existing NebulaStore::checkLeader logic applies for read and write operations;
2. A new checkLeader may be required for sendSnapshot, not required for checking commitInThisTerm_.

[critical27]

Good catch, what you mentioned is correct except one place: if GetSnapshot returns a nullptr, there maybe inconsistency between several reads, but each Seek in rocksdb is a snapshot. So maybe it is not real-time data.

[wey-gu]

Dear @developer-zc ,

We would like to send you a gift for the "good catch" NebulaGraph community award, would you mind sending us a mail to lisa.liu@vesoft.com with your address that can receive gift shipment?

Thanks and welcome to the community!

cc @QingZ11 @lisahui

[flymysql]

2. commitInThisTerm_

I have a problem that may be caused by this, but how can I solve it?

#5347

[liwenhui-soul]

yes, will fix it later