libmdbx v0.14.2 "Буревестник" (Stormy Petrel)

[erthink]

The frontward release with new major features and internal refactoring.

609 commits, 209 files changed, 29521 insertions(+), 15701 deletions(-)

Important:

• Due to numerous user requests, this ChangeLog will be kept in English.
  However, it should be noted that it was originally provided in Russian and then translated by AI during the formation of the release, because of this, there may be flaws in the text.

• Since 2026 libmdbx project has changed its code development and distribution model.
  To get acquainted with important changes and plans, we recommend reading the compact presentation "libmdbx: successes, obstacles, goals and roadmap", which contains important explanations in the form of embedded comments.

• The upstream of libmdbx project has been relocated to the jurisdiction of the Russian Federation.
  We are confident that this will protect the project from any sanctions and ensure its accessibility to all users around the world.
  Please use https://libmdbx.dqdkfa.ru for documentation and https://sourcecraft.dev/dqdkfa/libmdbx for the source code.
  Nonetheless libmdbx is still open source and provided with first-class free support.

Appreciations:

• Erigon for sponsorship.
• The "AntiPublic" project for sponsorship.
• Artyom Vorotnikov for Rust bindings, reporting bugs and testing.
• Stefan de Konink for Python bindings and documentation improvement.
• Cosmin Apreutesei for error reporting and testing.
• Chloe Cano for fuzzing, bug reporing and fixes.
• Weixie Cui for bug fixing through many pull-requests.
• Alexander Kelchin (the "Serebrium" Company) for error messages and prototypes of exploits.
• Anton Maisak for new .NET bindings.

Backward compatibility breaks:

• The typedefs of the various callbacks are now unified and includes an asterisk of indirection in C syntax notation.
  Perhaps this is the most annoying change that breaks the builds and requires changing your code. However, it is necessary to restore order.
  In most cases, the required changes are limited to removing the * chars after a callback type(s).

• The size and composition of the MDBX_envinfo structure has been changed, and the mdbx_env_info_ex() function no longer supports the old versions. This breaks the compatibility of the ABI with older versions of the library, but preserves API compatibility at the source code level.

• The template mdbx::buffer<ALLOCATOR, POLICY> is now inherited from mdbx::slice and mdbx::buffer_tag, which simplified the C++ API and the use of the meta-programming approach.

• When building using GNU Make and CMake now, instead of a single config.h, different config-gnumake.h and config-cmake.h files are generated.

• It is forbidden to open existing tables with different flags, unless the MDBX_DB_ACCEDE option is explicitly set.

• The dumps generated by the mdbx_dump utility no longer output the current size of the database and maxreaders, so that the contents of the dump depend only on the contents of the database.

• The build option MDBX_FORCE_ASSERTIONS has been deprecated, and MDBX_CHECKING (within range of -1..3) should be used instead.

• The NDEBUG macro, which is generally accepted in C, no longer affects assert checks inside the library, but retains its traditional influence on assert checks related to argument control in the inline methods of the C++ API.

New features:

• Implemented "Early GC Cleanup".

  Now the recycled GC records are deleted not when the writing transaction is committed, but as soon as possible. This opens the way to the implementation of explicit defragmentation (without copying the database) and further to non-sequential GC processing (which will eliminate the problem of DB swelling/overflow due to GC processing stopping during long-term reading transactions).

  The amount of overhead is now proportional to the volume of operations performed. Therefore, in most scenarios, the overhead is slightly less, but on the contrary, a little more when canceling nested transactions.

• Database defragmentation/compaction support has been implemented and the mdbx_defrag utility has been added with a set of command-line options that allow you to define key parameters and defragmentation limits.

• API Extension:

  • the MDBX_CP_OVERWRITE option has been added to the database copy function (overwriting the target file), and the mdbx_copy utility has a similar command-line option -f.
  • added the functions mdbx_cursor_bunch_delete() and mdbx_cursor_delete_range(), which perform massive deletion of adjacent elements much faster by excluding pages and branches with deleted items from a B-tree entirely.
  • added data retrieval functions with "caching" the mdbx_cache_get() and mdbx_cache_get_SingleThreaded().
  • added the mdbx_txn_refresh() function to quickly refresh the reading transaction.
  • added the mdbx_txn_checkpoint() function to commit write transaction without releasing locks.
  • added the mdbx_txn_commit_embark_read() function to commit a writing transaction and start a reading one without interfering with other changes.
  • added the mdbx_txn_amend() function to change data starting from a snapshot of the data used in a given read transaction.
  • added the mdbx_txn_rollback() function to abort and restart a transaction with the cancellation of all changes, but without releasing locks.
  • added support for cloning reading transactions using mdbx_txn_clone().
  • added support for nested read-only transactions.
  • added the mdbx_gc_info() function to get information about GC, page usage, and the ability to iterate GC content.
  • added the mdbx_env_defrag() function for explicit DB defragmentation, as well as the mdbx_defrag utility.
  • added the MDBX_opt_split_reserve option to control the fullness of tree pages when splitting them.
  • added the functions mdbx_cursor_distance(), mdbx_cursor_scroll() and mdbx_cursor_distribute() to simplify multithreaded parallel scanning.

• Support for Harmony OS (OHOS) and Haiku OS.

• Floating-point operations are no longer used both inside the library and in utilities, and linking to libm has been removed from build scripts.

• It is possible to set debugging options MDBX_DBG_ASSERT, MDBX_DBG_AUDIT and others through environment variables. However, the corresponding debugging capabilities still need to be activated during the build.

• Expanded and redesigned the composition of information generated by the function mdbx_chk_env() and the output utility mdbx_chk.

• The main libmdbx repository has been migrated from GitFlic to SourceCraft.

  For my part, I am saddened by the need to perform such manipulations, because they create significant inconvenience to users, but (unfortunately) there are sufficient reasons for this:

  • Instead of the promised internationalization, GitFlic has only a Russian-language localization with a lot of technical features that make it difficult to use machine translation systems. This made it impossible for many users to use the service and generated a number of legitimate complaints/reproaches, including from developers from China, Brazil, Korea, Iran, etc.
  • In the three years since migrating to GitFlic, several outrageous errors in the markdown editor have not been fixed, which turned the design of releases into an annoying struggle. In addition, the development roadmap has disappeared from the public space. In total, this forced me to abandon GitFlic.

• The number of page receiving/loading operations has been added to the transaction statistics, which makes it possible to quantify the amount of work with cursors and the effectiveness of various indexing and data retrieval approaches.
  The collection of relevant statistics is controlled by the additional build option MDBX_ENABLE_PGET_STAT.

• The command-line options -b number, -L megabytes, -d percent and -G geometry have been added to the mdbx_load utility, allowing you to set the size of batch inserts, limit the volume of transactions, set the desired page filling density and redefine the geometry of the database when loading data from a dump.

• Search was accelerated by using a branchless algorithm and embedding code of built-in/default comparators.

• Redesigned internal verification statements and related build options.
  At the same time, NDEBUG no longer affects checks in the main engine code, which eliminates the causes of unexpected performance drops due to the lack of a definition of NDEBUG in non-debugging builds of users.

  The checks are divided into three categories (cheap, medium, expensive), controlled by the build option MDBX_CHECKING, which takes values from -1 to 3 inclusive.
  The value of 3 corresponds to the maximum number of checks, and -1 disables both all assert() and ENSURE() checks.
  By default, MDBX_CHECKING is assumed to be equal to the MDBX_DEBUG option, which in turn defaults to 0, which corresponds to a regular (non-debugging) library build.
  This way, compatibility with the previous behavior is maintained and at the same time precise control of debugging checks is ensured.

Behavior change:

• Re-enabled/enabled on older Linux kernels, starting with version 3.16, since now there is no reason to stop working on 3.16 while supporting 4.x kernels, and there are still projects (Isar, Isar-Community, Hive) that require such support.

• The default value of the page merge threshold has been changed from 25% to 33%.

• To reduce the likelihood of unexpected errors due to transients and delayed processing in the OS kernel during competitive closing and opening of databases by different processes, the number of repeated attempts to capture locks has been tripled. Presumably, this will also solve the problem of unexpected EAGAIN (11) errors on Android when restarting applications and opening the database immediately after closing.

• By default, Windows builds are now performed using the Windows 10 SDK, rather than Windows 7.

• The error MDBX_WANNA_RECOVERY when opening the database in read-only mode is now returned if the database size is not a multiple of the system page size, but not a multiple of the size of the virtual memory allocation block is ignored. This eliminates the regression that occurred due to a change in behavior after using the system call fallocate() to prevent SIGBUS after incrementing the database file in a populated file system.

Other improvements:

• The logic of not using OFD locks on POSIX platforms has been finalized.
  Now, in addition to EINVAL, additional error codes are taken into account (ENOSYS, ENOIMPL, ENOTSUP, ENOSUPP, EOPNOTSUPP), which will allow the compiled library to work in some cases when the current kernel/container/emulator does not support the required system calls.

• The tests are supplemented with scenarios to check the added features, identified regressions and errors.

• Support for the --numa# option has been added to the test framework to link a stochastic test to a NUMA node, and explicit distribution across NUMA nodes has been added to the battery/tmux script, which significantly increased efficiency when testing on NUMA machines.

• The stochastic script implements a random order of running individual tests.

• The output of a histogram of filling pages forming the tree structure and participating in split/merge/rebalance operations has been added to the database integrity verification functionality and the mdbx_chk utility.

• A workaround has been added for Android to reduce the likelihood of an EAGAIN system error due to a lack of system resources and transients when closing and quickly re-opening the database.

• For Linux, added error prevention in the fast_commit implementation of the Ext4 file system.

• Support for the "Skip" and "Repeat" options has been added to debugging builds on Windows when assert checks are triggered.

• File locks used on the Windows platform involve waiting with timeouts, which theoretically should reduce the likelihood of ERROR_LOCK_VIOLATION (33) errors when opening a database in competitive scenarios.

• Redesign of the buffer implementation and other improvements in the C++ API.

• The tests fixed several minor memory leaks and the UBSAN warning about the memcmp(, length = 0) call.

Downloads:

• 377K, libmdbx-amalgamated-0.14.2.zpaq, sha256sum: 936cb1699add1fe8918e7f3de88df11456e8fd2a8b839c4ef942c9073db7a857

• 511K, libmdbx-amalgamated-0.14.2.tar.xz, sha256sum: 208044935dd6585d76b372eb0d14ee72fe7171152bc31884843309cec7050b91

---

For more information please see ChangeLog.

---

This discussion was created from the release libmdbx v0.14.2 "Буревестник" (Stormy Petrel).