Corruption: block checksum mismatch on RocksDb 6.7.3

[patkivikram]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

We are running our application in Kubernetes with PV's mapped to NFS storage using Netapp. We ran into 2 instances of corruption in our production environment in the last 1 week. Both corruptions were in different db's with the following error message b'Corruption: block checksum mismatch: expected 3032677561, got 182668963 in /data/fos/fos-0/v1/tables/kv_table-3.db/001581.sst offset 18018905 size 4204'

Based on the python rocksdb bindings we are using the use_fsync is set to False by default.

Running this on centos 7.5

Expected behavior

RocksDb should not corrupt data.

Actual behavior

Data gets corrupted with no way to recover

Steps to reproduce the behavior

We have about 4G of data in about 15 DB's managed by the kafka streams application

[koldat]

It is most likely HW issue. That is why there is a checksum so it does not need to be RocksDB fault. You can try to fix SST using sst_tool in this PR #6955