RocksDB PlainTable doesn't work as expected with SstFileWriter

[gaojieliu]

Hi RocksDB folks:
I noticed some wired behavior while using PlainTable format with SstFileWriter and the rocksdbjni version being used is 6.11.4.
Steps to reproduce this issue:

1. Initialize an Option with PlainTable format enabled.
2. Generated 100 ordered key/value pairs.
3. Write the ordered key/value pairs via SstFileWriter.
4. Ingest the generated sst file in step 3 into a RocksDB database.
5. Verify whether all the generated key/value pairs exist in the database or not.

Following the above steps, I couldn't query any entries from the database, but if I add the following statements between step 4 and step 5:

    rocksDB.close();
    rocksDB = RocksDB.openReadOnly(options, dbDir);

I could read all the entries.

Expected behavior

I should be able to read all the entries after ingesting sst files without re-opening it in read-only mode.

Actual behavior

Can't read any entry without re-opening the database in read-only mode.

Steps to reproduce the behavior

Sample code:

import java.io.File;
import java.nio.ByteBuffer;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;
import org.apache.commons.lang.RandomStringUtils;
import org.rocksdb.ComparatorOptions;
import org.rocksdb.Env;
import org.rocksdb.EnvOptions;
import org.rocksdb.IngestExternalFileOptions;
import org.rocksdb.Options;
import org.rocksdb.PlainTableConfig;
import org.rocksdb.RocksDB;
import org.rocksdb.RocksDBException;
import org.rocksdb.SstFileWriter;
import org.rocksdb.util.BytewiseComparator;
import org.testng.annotations.Test;


public class RocksDBPlaintableTest {
  private static final String DATA_BASE_DIR = Paths.get(System.getProperty("java.io.tmpdir"), "").toAbsolutePath().toString();;
  private static final String keyPrefix = "key_";
  private static final String valuePrefix = "value_";

  private Map<String, String> generateInput(int recordCnt, boolean sorted, int padLength) {
    Map<String, String> records;
    if (sorted) {
      BytewiseComparator comparator = new BytewiseComparator(new ComparatorOptions());
      records = new TreeMap<>((o1, o2) -> {
        ByteBuffer b1 = ByteBuffer.wrap(o1.getBytes());
        ByteBuffer b2 = ByteBuffer.wrap(o2.getBytes());
        return comparator.compare(b1, b2);
      });
    } else {
      records = new HashMap<>();
    }
    for (int i = 0; i < recordCnt; ++i) {
      String value = valuePrefix + i;
      if (padLength > 0) {
        value += RandomStringUtils.random(padLength, true, true);
      }
      records.put(keyPrefix + i, value);
    }
    return records;
  }

  private String getTempDatabaseDir(String storeName) {
    File storeDir = new File(DATA_BASE_DIR, storeName).getAbsoluteFile();
    if (!storeDir.mkdirs()) {
      throw new RuntimeException("Failed to mkdirs for path: " + storeDir.getPath());
    }
    storeDir.deleteOnExit();
    return storeDir.getPath();
  }

  private void removeDir(String path) {
    File file = new File(path);
    if (file.exists() && !file.delete()) {
      throw new RuntimeException("Failed to remove path: " + path);
    }
  }

  private Options getStoreOptions() {
    Options options = new Options();

    options.setEnv(Env.getDefault());
    options.setCreateIfMissing(true);
    PlainTableConfig tableConfig = new PlainTableConfig();
    tableConfig.setStoreIndexInFile(true);
    tableConfig.setBloomBitsPerKey(10);
    options.setTableFormatConfig(tableConfig);

    options.setAllowMmapReads(true);
    options.useCappedPrefixExtractor(16);
    options.setWriteBufferSize(16 * 1024 * 1024);

    return options;
  }

  @Test
  public void testPlainTableWithSSTFileWriter() throws RocksDBException {
    String storeName = "test_store";
    String dbDir = getTempDatabaseDir(storeName);
    Options options = getStoreOptions();
    RocksDB rocksDB = RocksDB.open(options, dbDir);
    EnvOptions envOptions = new EnvOptions();
    Map<String, String> kvPairs = generateInput(100, true, 0);

    SstFileWriter sstFileWriter = new SstFileWriter(envOptions, options);
    String folderForTempSSTFile = dbDir + "/.sst";
    File folder = new File(folderForTempSSTFile);
    folder.mkdirs();
    String fullPathForTempSSTFile = folderForTempSSTFile + "/0.sst";
    sstFileWriter.open(fullPathForTempSSTFile);
    for (Map.Entry<String, String> entry : kvPairs.entrySet()) {
      sstFileWriter.put(entry.getKey().getBytes(), entry.getValue().getBytes());
    }
    sstFileWriter.finish();
    IngestExternalFileOptions ingestOptions = new IngestExternalFileOptions();
    ingestOptions.setMoveFiles(true);
    rocksDB.ingestExternalFile(Arrays.asList(fullPathForTempSSTFile), ingestOptions);
    System.out.println(fullPathForTempSSTFile);

    // Reopen the database
//    rocksDB.close();
//    rocksDB = RocksDB.openReadOnly(options, dbDir);
    // Verify
    for (Map.Entry<String, String> entry : kvPairs.entrySet()) {
      byte[] key = entry.getKey().getBytes();
      byte[] expectedValue = entry.getValue().getBytes();

      byte[] value = rocksDB.get(key);
      if (! Arrays.equals(expectedValue, value)) {
        System.out.println("Received unexpected value: [" + value + "] for key: [" + entry.getKey() + "], should be [" + entry.getValue() + "]");
      }
    }

    rocksDB.close();
    RocksDB.destroyDB(dbDir, options);
    options.close();
    envOptions.close();
    sstFileWriter.close();
    ingestOptions.close();
    removeDir(folderForTempSSTFile);
    removeDir(dbDir);
  }
}

[gaojieliu]

Any update?
I am wondering whether Plain Table format could work with SstFileWriter at all or not?

cc @adamretter

[adamretter]

@jay-zhuang Do you know if this is a known issue?

[jay-zhuang]

It should work, but I don't know, maybe a bug.

[siying]

@gaojieliu I don't think ingestExternalFile() supports PlainTable, unfortunately... If you are interested in implementing it for PlainTable, it will be easier to only support IngestExternalFileOptions.write_global_seqno = false case. You are welcome to contribute.