block-size not sanitized, set it larger than 4GB by mistake may cause data corruption

[zhangjinpeng87]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

Expected behavior

When set block-size larger than 4GB, it should be forbidden.

Actual behavior

When i set block-size as 8GB, RocksDB started up and encountered "Corruption: bad entry in block" error after some compaction.

Steps to reproduce the behavior

1. Load data
2. Set block-size = 8GB
3. Compact range
4. Read data

@siddontang @yiwu-arbug

[miasantreble]

I wrote a test but was unable to reproduce the failure, do you mind taking a look?

TEST_F(DBTest, LargeBlockSizeTest) {
  Options options = CurrentOptions();
  CreateAndReopenWithCF({"pikachu"}, options);
  ASSERT_OK(Put(0, "foo", "bar"));
  BlockBasedTableOptions table_options;
  table_options.block_size = 8LL*1024*1024*1024LL;
  options.table_factory.reset(NewBlockBasedTableFactory(table_options));
  ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
  ASSERT_OK(Put(0, "foo1", "bar1"));
  db_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
  ASSERT_EQ("bar", Get(0, "foo"));
  ASSERT_OK(TryReopenWithColumnFamilies({"default", "pikachu"}, options));
  ASSERT_EQ("bar1", Get(0, "foo1"));
}

[yiwu-arbug]

@miasantreble try having two >4GB size values in a data block?

[siying]

@miasantreble as what @yiwu-arbug said, our restart offset is encoded as fixed uint32. If there is only one key in the block, the offset of the key is 0. But if there are multiple keys, the offset of the key might excceeds max of uint32_t. There is also a a keys per restart index setting, so that you might want to have many keys, or tune the restart index to be 1 to reproduce it.

But clearly block size over max uint32_t won't work.