erlang-rocksdb 3.2.0

Upgrades

• update vendored RocksDB to 11.13.2 (from 11.1.2) (#22)

New Features

• two-phase ingestion: prepare_ingestion/3,4 stages SST files without blocking writes, commit_ingestion/2 makes one or several of them visible at once, and abort_ingestion/1 cancels. See the SST files guide. (#22)
• abort_compactions/2 and resume_compactions/2 stop and restart compactions of a single column family, leaving the others running (#22)
• new column family options: read_triggered_compaction_threshold compacts files that are read often, min_tombstones_for_range_conversion speeds up reads over long runs of deleted keys, periodic_compaction_seconds and blob_file_writable_file_max_buffer_size (#22)
• new database options: use_direct_io_for_compaction_reads keeps compaction scans out of the page cache, max_compaction_trigger_wakeup_seconds, and the experimental optimize_manifest_for_recovery and reuse_manifest_on_open for faster reopen (#22)
• two new statistics histograms, ingest_external_file_prepare_micros and ingest_external_file_run_micros, show how long ingestion takes and how long it blocks writes. The statistics guide also documents the database properties that show if compactions fall behind. (#22)
• block based tables accept format_version 8 and optimize_key_common_prefix, which stores the shared key prefix of a block once (#22)

Bug Fixes

• a database opened with an env from new_env/1 crashed the VM once the env term was garbage collected, for example when the process that opened it exited. The database now keeps its env alive. In-memory envs created by the in_memory and {env, memenv} options were never freed, they are now freed when the database closes. (#23)
• using a write batch after release_batch/1 crashed the VM, it now raises badarg. Concurrent calls on the same batch are serialized. (#23)
• one malformed merge operand made flush fail and then every write, so the database stopped accepting writes. Now only reads of that key fail. This applies to the Erlang, bitset and counter merge operators. (#23)
• the bitset merge operator wrote one byte past the bitmap for the first position after its end, and changed the stored value in place, so snapshots and concurrent readers saw bits they should not. Positions must now be plain decimal numbers inside the bitmap. (#23)
• int_add in the Erlang merge operator and the counter merge operator wrapped around on overflow, they now fail the merge (#23)
• closing a database, a column family or another handle while other processes still used it could crash the VM (#23)
• an iterator using auto_refresh_iterator_with_snapshot read a freed snapshot after release_snapshot/1. Iterators now keep their snapshot. Reads given a released snapshot raise badarg, where they used to silently read the latest data. (#23)
• an SST file reader iterator dropped without sst_file_reader_iterator_close/1 kept its reader forever, and closing it could free the reader while the iterator still used it (#23)
• posting_list_contains/2 trusted the sizes inside the binary, a malformed one could ask for gigabytes of memory (#23)
• repeating iterate_upper_bound or iterate_lower_bound in iterator options leaked memory (#23)
• sharing a transaction, an iterator, a WAL iterator or an SST file writer between processes could crash the VM or corrupt the transaction: RocksDB requires these calls to be serialized, they now are (#25)
• an iterator over a transaction crashed the VM once the transaction was released, it now keeps the transaction until it is closed (#25)
• destroy_column_family/2 freed the column family while other calls, or iterators over several column families, still used it. The column family is now freed when the last of them is done. (#25)
• handles created while the database was closing (snapshots, iterators, transactions, column families) were not closed with it, so RocksDB stayed open and locked after close/1. They are now refused with badarg. (#25)
• a parsed posting list with a few very large keys still ran on a normal scheduler, it is now scheduled by size as well as by key count (#25)
• closing a database while another process dropped one of its handles, or created an iterator on it, could crash the VM or leave RocksDB open, and the cleanup of every handle read the object after its destructor had run. Handles now free their RocksDB objects when the last user is done and are destroyed only when Erlang collects them. (#26)
• the build no longer downloads anything from the network: the vendored CRoaring build fetched a CMake helper from GitHub at configure time, so offline builds failed (#22)

Changes

• close/1 now returns once the database is really closed and its files are released, so the same path can be opened again right away. Before, a call still running in another process, or the cleanup of a handle, could keep RocksDB open a moment after close/1 returned, and reopening failed on the lock file. As a consequence close/1 now waits for calls still running on the database, such as a long compact_range/4. (#26)
• the bundled zstd is now really used to build RocksDB: before, RocksDB looked for zstd on its own and could compile against the system headers while the NIF linked the bundled library. A clean parallel build could also fail when RocksDB started before the bundled zstd was ready (#21)
• new build option WITH_STATIC_ZSTD links a static system zstd into the NIF, so it has no libzstd runtime dependency. Off by default, the system archive must be built with -fPIC (#21)
• posting list functions run on a dirty scheduler when the posting list is large (over 64 KB, or over 4096 keys for a parsed one), so they no longer hold a normal scheduler for milliseconds. Small lists still run inline. (#24)
• a compaction filter handler that does not answer in time is not asked again for the rest of that compaction, the remaining keys are kept. Before, every key waited the full timeout, so a stuck handler could stall compaction for hours. The handler is asked again at the next compaction. (#24)
• the batch_size option of the compaction filter is now documented as having no effect: keys are always sent one per message (#24)

Documentation

• the SST files guide covers two-phase ingestion, the statistics guide the ingestion histograms and compaction queue properties (#22)
• the compaction guide covers per column family abort, direct I/O for compaction reads, read-triggered and periodic compaction
• the OTP integration guide explains that close/1 waits for running calls, and that handles shared between processes are safe but serialized
• the bitset merge operator guide documents the operand format
• the compaction filter guide no longer documents batching, which never existed (#24)

Deprecations

• the external_sst_file_global_seqno_offset entry returned by sst_file_reader_get_table_properties/1 is always 0: RocksDB 11.13 no longer records it (#22)