ZSTD_TrainDictionary runs even when the compression is set to kNoCompression for a given level

[kwadhwa18]

ZSTD_TrainDictionary [link] runs for SSTFileWriter::Finish even when bottommost_compression option is set to kNoCompression. This reduces throughput for SstFileWriter::Finish

We construct rocksdb options using ZSTD compression for levels including 2 and above. For levels 0 and 1, we set it to kNoCompression. We also set zstd_max_train_bytes to a non-zero positive value (which is applicable for levels with ZSTD compression enabled). These options are used for the database and also passed to SstFileWriter for creating sst files to be later added to that database. Since the BlockBasedTableBuilder::Finish [link] only checks for zstd_max_train_bytes to be non-zero positive value, it runs ZSTD_TrainDictionary even when it shouldn't since SSTFileWriter is operating at bottommost level

Expected behavior

If the bottommost_compression or compression_per_level for a level is set to kNoCompression, then don't run ZSTD_TrainDictionary

Actual behavior

ZSTD_TrainDictionary is also run for level which has kNoCompression set

[ajkr]

Another case is max_dict_bytes > 0 will build a dictionary even when compression type is kNoCompression. These sound like good sanitizations to add. Would you be interested in adding some of them?

[kwadhwa18]

are you referring to

rocksdb/table/block_based/block_based_table_builder.cc

Lines 1874 to 1887 in 210c8df

	for (size_t i = 0;
	i < kNumBlocksBuffered && compression_dict_samples.size() < kSampleBytes;
	++i) {
	size_t copy_len = std::min(kSampleBytes - compression_dict_samples.size(),
	r->data_block_buffers[buffer_idx].size());
	compression_dict_samples.append(r->data_block_buffers[buffer_idx], 0,
	copy_len);
	compression_dict_sample_lens.emplace_back(copy_len);
	buffer_idx += kPrimeGeneratorRemainder;
	if (buffer_idx >= kNumBlocksBuffered) {
	buffer_idx -= kNumBlocksBuffered;
	}
	}

?

I can help with the sanitizations - is the level information available inside BlockBasedTableBuilder?

[ajkr]

are you referring to

rocksdb/table/block_based/block_based_table_builder.cc

Lines 1874 to 1887 in 210c8df

	for (size_t i = 0;
	i < kNumBlocksBuffered && compression_dict_samples.size() < kSampleBytes;
	++i) {
	size_t copy_len = std::min(kSampleBytes - compression_dict_samples.size(),
	r->data_block_buffers[buffer_idx].size());
	compression_dict_samples.append(r->data_block_buffers[buffer_idx], 0,
	copy_len);
	compression_dict_sample_lens.emplace_back(copy_len);
	buffer_idx += kPrimeGeneratorRemainder;
	if (buffer_idx >= kNumBlocksBuffered) {
	buffer_idx -= kNumBlocksBuffered;
	}
	}

?

Yes.

I can help with the sanitizations - is the level information available inside BlockBasedTableBuilder?

Yes , BlockBasedTableBuilder::Rep has compression_type and compression_opts, which are the settings specific to the level for which the table is being built

edit: Technically the answer to your question is no, but my point is BlockBasedTableBuilder::Rep has everything you need without it

[kwadhwa18]

I have attempted a fix #12420. PTAL!