Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Clone in Desktop Download ZIP

Loading…

Needs more space efficient row checksums #93

Closed
yoshinorim opened this Issue · 1 comment

2 participants

@yoshinorim
Owner

Here are sizes of loading LinkBench data with 1.5B max ids.
Without row checksum: 570959236
With row checksum enabled: 705439184

23.6% space increase is too much. This was because current row checksum adds 9 bytes (1B + CRC32 key and CRC32 value) into value for each index entry.

How about doing some optimizations like below?

  • Adding new checksum format using CRC8 instead of CRC32. This makes per row size reduced from 9B to 3B.
  • Adding rocksdb_checksum_row_pct global variable to control how many percentage of rows to be checksummed. For example, by setting to 10, checksum is enabled for only 10% of rows. This reduces checksum space overhead by 90%. This approach may miss some checksum corruptions. But in practice, many more than 10 rows will be affected if there is any corruption bug, so sooner or later it is possible to detect corruption.
@yoshinorim
Owner

Introducing rocksdb_checksums_pct variable -- https://reviews.facebook.net/D43419

I'll keep this issue opened until we figure out if CRC8|16 are needed.

@hermanlee hermanlee referenced this issue in facebook/mysql-5.6
Open

Needs more space efficient row checksums #50

@hermanlee hermanlee closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.