RheaKVStore#bContainsKey(byte[]) inconsistent results returned after Snapshot.

[lfygh]

Describe the bug

RheaKVStore#bContainsKey(byte[]) inconsistent results returned after Snapshot.

1、put a key to RheaKVStore

2、wait background thread to do snapshot

3、call RheaKVStore#bContainsKey(byte[]) , it returns false (error, because I haven’t remove it);

4、but call RheaKVStore#bGet(byte[]), it returns the value.

5、call RheaKVStore#bContainsKey(byte[]) again, it returns true.

Expected behavior

call RheaKVStore#bContainsKey(byte[]) , it always returns true until explicitly remove it.
.

Actual behavior

call RheaKVStore#bContainsKey(byte[]) , it returns false after the snapshot.

Steps to reproduce

it did the test under jraft-example/src/main/java/com/alipay/sofa/jraft/example/rheakv package.

1、add options to server1,server2,server3

       NodeOptions nodeOptions = new NodeOptions(); // 1、new options
        nodeOptions.setSnapshotIntervalSecs(2);  // 2、smaller SnapshotInterval for fast do snapshot

        final RegionEngineOptions rOpts = new RegionEngineOptions();
        rOpts.setRegionId(Constants.DEFAULT_REGION_ID);
        rOpts.setNodeOptions(nodeOptions);
        List<RegionEngineOptions> rOptsList = Lists.newArrayList();
        rOptsList.add(rOpts);
        final PlacementDriverOptions pdOpts = PlacementDriverOptionsConfigured.newConfigured()
                .withFake(true) // use a fake pd
                .config();
        final StoreEngineOptions storeOpts1 = StoreEngineOptionsConfigured.newConfigured() //
                .withStorageType(StorageType.RocksDB)
                .withRocksDBOptions(RocksDBOptionsConfigured.newConfigured().withDbPath(Configs.DB_PATH).config())
                .withRaftDataPath(Configs.RAFT_DATA_PATH)
                .withServerAddress(new Endpoint("127.0.0.1", 8181))
                .withCommonNodeOptions(nodeOptions) //  3、add nodeOptions
                .withRegionEngineOptionsList(rOptsList)
                .config();

2、example class

package com.alipay.sofa.jraft.example.rheakv;

public class Example {
    private static final Logger LOG = LoggerFactory.getLogger(BContainsKeyFalseExample.class);
    
    public static void main(final String[] args) throws Exception {
        final Client client = new Client();
        client.init();
        get(client.getRheaKVStore());
        client.shutdown();
    }

    public static void get(final RheaKVStore rheaKVStore) throws InterruptedException {

        final byte[] key = writeUtf8("hello");
        final byte[] value = writeUtf8("world");
        rheaKVStore.bPut(key, value);

        for (int i = 0; i < 5; i++) {
            TimeUnit.SECONDS.sleep(1);
            final boolean b= rheaKVStore.bContainsKey(key);
            LOG.info("Sync bContainsKey result={}", b);

            final byte[] b3 = rheaKVStore.bGet(key);
            LOG.info("Sync bGet result={}", readUtf8(b3));
        }
    }
}

2022-11-18 17:01:10 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:10 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:11 [main] INFO  Example:31 - Sync bContainsKey result = false
2022-11-18 17:01:11 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:12 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:12 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:13 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:13 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:14 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:14 [main] INFO  Example:34 - Sync bGet result = world

More

I copied the code from com.alipay.sofa.jraft.rhea.storage.RocksRawKVStore#backupDB, and use RocksDB API
to shrink the scope, yes, it is reproduced.

Are there some options I did not set? or is it a bug?

Minimal yet complete reproducer code (or GitHub URL to code)

Environment

• SOFAJRaft version:
  fork from the GitHub master branch

• JVM version (e.g. java -version):

  openjdk version "17.0.4" 2022-07-19 LTS
  OpenJDK Runtime Environment Zulu17.36+13-CA (build 17.0.4+8-LTS)
  OpenJDK 64-Bit Server VM Zulu17.36+13-CA (build 17.0.4+8-LTS, mixed mode, sharing)

• OS version (e.g. uname -a):
  Darwin MacBook-Pro.local 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64

• Maven version:

• IDE version:

[killme2008]

@fengjiachun PTAL

[fengjiachun]

@lfygh

Thank you very much, I reproduced this bug, I will check the exact cause and reply later.

[fengjiachun]

@lfygh

Thanks a lot, this is a bug, during one of the rocksdb upgrades, we misunderstood the new API of rocksdb, resulting in a wrong way to use it: The valueHolder only return the value when the key is found in memory.

See

I will fix it and release a new version ASAP.