v0.4.1 -- perf-instrumentation infrastructure

v0.4.1 — perf-instrumentation infrastructure release

Layer-by-layer plugin perf capture surface. No engine optimisations ship in this release — the deliverable is the measurement surface that the v0.5.0 cycle will optimise against. The bundled engine is unchanged from v0.4.0 (still TidesDB v9.3.2, still shipped unpatched).

The default perconalab/tidesdb-mysql:0.4.1 image is a byte-identical re-stamp of the v0.4.0 release image: with the default TIDESDB_PERF=0 build, every TDB_PERF_SCOPE(...) macro expands to ((void)0), so the default user-facing code path is unchanged. Instrumentation lives in a separate, opt-in perconalab/tidesdb-mysql:0.4.1-perf variant.

What's new

• TDB_PERF_SCOPE(MethodId) RAII macro across the full plugin entry surface (32 MethodId values: write_row, update_row, delete_row, index_read_map, index_next, index_prev, rnd_next, rnd_pos, external_lock, start_stmt, store_lock, commit, rollback, four savepoint_*, open, close, info, table_flags_cache_init, create, delete_table, four inplace-ALTER virtuals, serialize_row, deserialize_row, key_copy_to_comparable, pk_from_record, encrypt_row_into, decrypt_row). On capture-off the overhead is two __rdtsc() reads + one atomic load per scope. Compile-time-gated on TIDESDB_PERF.
• Thread-local rdtsc ring buffer (default 2^16 samples per thread, runtime-configurable). 24-byte Sample packed natural-alignment with static_assert on the layout. Rings self-register into a lock-free linked list (g_rings_head) on first use.
• Background flusher thread pinned to CPU 0; walks the ring list every tidesdb_perf_flush_interval_ms, bucket-sorts samples by method, pwritevs each bucket to its method's append-only .bin file. Calibrates TSC via 20 ms std::chrono::steady_clock spin and writes the calibration to meta.json.
• Four perf sysvars (all gated on TIDESDB_PERF=1): tidesdb_perf_capture (BOOL, default OFF — the kill-switch), tidesdb_perf_output_dir, tidesdb_perf_ring_capacity_pow2, tidesdb_perf_flush_interval_ms.
• tools/tidesdb_perf_analyze — Python 3 + numpy offline analyser. Parses .bin files (struct.pack('<BBH4xQQ', ...)), aggregates per-method calls / total_ms / mean_us / p50 / p95 / p99 / max, emits markdown. --compare A B produces a side-by-side diff.
• bench/perf/run-perf-capture.sh integration harness. Wraps bench/hammerdb/run-hammerdb.sh, forces perf sysvars ON, mounts the perf output directory into the container, copies + chowns artifacts back to host, runs the analyser.
• docker/patches/tidesql/0001-perf-instrumentation.patch (880 lines) — ports Sample / TLS_Ring / PerfScope + 30 TDB_PERF_SCOPE call sites into TideSQL's ha_tidesdb.cc for the eventual MySQL-vs-MariaDB side-by-side run.
• 5 MTR perf tests + 8 gtest ring tests + 2 analyser pytests — all green.

Headline numbers (WARE=10 RUNVU=8 1m+3m HammerDB TPROC-C, perf ON)

Captured from the perf variant on the validation host:

method	calls	mean_us	p99_us	max_us
write_row	5,294,113	16.54	17.13	72,783.99
index_read_map	1,900,638	13.19	108.09	18,306.95
deserialize_row	3,950,283	0.90	3.61	1,034.43
serialize_row	6,011,734	0.41	2.32	4,298.43
pk_from_record	6,361,758	0.19	0.72	7,047.85

Throughput with perf ON: 1841 NOPM (vs ~2028 NOPM baseline → ~9% capture overhead).

write_row p99 = 17 μs but max = 72 ms — long compaction-stall tail dominates total. index_read_map p95 (44.7 μs) is 8× p50 (6.1 μs) — read path also has a meaningful long tail. These are the v0.5.0 optimisation entry points.

Validation

Full validation matrix green — see docs/v0.4.1-validation-report.md.

Docker

Default image (no perf, equivalent to v0.4.0):
```
docker pull perconalab/tidesdb-mysql:0.4.1
```

Perf variant (opt-in; instrumented build for the v0.5.0 optimisation cycle):
```
docker pull perconalab/tidesdb-mysql:0.4.1-perf
```

Deferred to v0.5.0

• Engine-side optimisations driven by the captured hotspots above.
• sut-mariadb-tidesdb:9.3.0-perf SUT image build (TideSQL patch is committed; image not yet built).
• TideSQL perf sysvars (the patch wires scopes + ring; sysvar wiring deferred).

v0.4.0 -- atomic-DDL participation (closes A-5)

v0.4.0 — atomic-DDL participation (closes A-5)

Wires the TidesDB-MySQL plugin into MySQL 9.7's atomic-DDL contract end-to-end. Crash-during-DDL no longer leaves the data dictionary and the column-family list divergent, and engine-private schema metadata is now portable via SDI. The bundled TidesDB engine is unchanged from v0.3.1 (still v9.3.2, still unpatched) — all changes are plugin-side.

Added

• HTON_SUPPORTS_ATOMIC_DDL flag activated on the handlerton — MySQL now treats TidesDB as a full participant in the DD transaction.
• 6 SDI tablespace callbacks (sdi_create / sdi_drop / sdi_get_keys / sdi_get / sdi_set / sdi_delete) backed by a dedicated __tidesdb_sdi metadata column family. mysqldump --tab, IMPORT TABLE, and clone can now capture TidesDB engine-private metadata portably.
• 8 DDSE callback stubs (ddse_dict_init / dict_init / dict_recover / dict_cache_reset / dict_cache_reset_tables_and_tablespaces / dict_get_server_version / dict_set_server_version / is_dict_readonly) — wired for a future "TidesDB hosts the data dictionary" capability; inert in v0.4.0.
• Single logical tablespace tidesdb_system registered via Plugin_tablespace.
• prepare_create / validate_open / prepare_drop — persist CF binding (cf_name + SHA-256 schema fingerprint + CRC32 options checksum + atomic_ddl marker + creation epoch) into dd::Table::se_private_data on CREATE; validate on OPEN; clean up SDI on DROP.
• Inplace ALTER state machine now consumes its dd::Table * parameters; emits updated SDI on commit.
• DdSyncReconciler — symmetric-difference reconciliation between the DD and the TidesDB CF list (excluding __-prefixed internal CFs), driven via the tidesdb_orphan_action sysvar (drop / quarantine / log_only).
• 2 new sysvars: tidesdb_atomic_ddl_strict (BOOL, default ON) and tidesdb_orphan_action (ENUM, default quarantine).
• 21 new MTR tests under the tidesdb_ddl_* prefix + 10 new gtest cases for the pure helpers.

Changed

• Inplace ALTER virtuals now consume dd::Table * (previously [[maybe_unused]]).
• ha_tidesdb::rename_table / ha_tidesdb::delete_table flush the engine transaction before MySQL's CF mutations begin (pre-Task-13 ordering preserved at the engine layer).

Fixed

• COPY-ALTER 2PC SIGSEGV (commit 0d7fe2c). Latent heap-use-after-free inside the TidesDB C library (tidesdb_iter_release_sst_source_block) — exposed only after the HTON flag activation changed MySQL's commit ordering. Tactical fix flushes the engine txn at the top of rename_table and delete_table; long-term fix (SE-private DDL journal) deferred to v0.5.0.

Upgrade notes

• Pre-v0.4.0 tables don't auto-emit SDI on open. To upgrade: run ALTER TABLE t ENGINE=TIDESDB (no-op ALTER) to populate se_private_data + emit SDI.
• Default tidesdb_atomic_ddl_strict=ON rejects tables whose se_private_data is missing the atomic-DDL keys. Set OFF temporarily during the upgrade window.

Validation

Full validation matrix green — see docs/v0.4.0-validation-report.md:

• MTR: 68 PASS / 0 FAIL / 15 SKIPPED
• gtest: 10/10 PASS (SdiPackKey 3/3 + ReconcilerDelta 7/7)
• mwbench 1 GiB integrity: PASS (290 MiB/s, 0 corruption, 0 misses)
• HammerDB SMOKE + WARE=10: PASS, 0 handler errors
• SIGKILL crash-recovery: PASS (WAL durably recovered)
• v0.3.1 vs v0.4.0 head-to-head HammerDB: within single-iteration noise

Artifact

• perconalab/tidesdb-mysql:0.4.0 — digest sha256:6fcfc8ebf142c4a078185ef1b3cdf8ba7d28b3132d3a0e554926a88c457ce04c
• Plugin .so SHA-256: af149b16afebe45c96c3a5d6b4451a44fcfec596c4a9d8434f03fc0607e618e4 (9,483,640 bytes)

docker pull perconalab/tidesdb-mysql:0.4.0

Note: an initial release artifact pushed at 2026-06-04 17:50 contained a stale .so built before commits 374a789 and 0d7fe2c (size 6,903,416, SHA cf9eb516…) — that build lacked the HTON_SUPPORTS_ATOMIC_DDL activation and the COPY-ALTER fix. Corrected artifact re-pushed at 20:57. If you pulled the image during the broken window, re-pull. See the rebuild note in the validation report.

Full commit history

v0.3.1...v0.4.0

v0.3.1 — TidesDB engine v9.3.2 (concurrency hardening, fd-starvation fix, parallel compaction)

v0.3.1 — TidesDB engine v9.3.2 (concurrency hardening, fd-starvation fix, parallel compaction)

Patch-level bump of the bundled storage engine from TidesDB v9.3.0 to v9.3.2 (two upstream patch releases). The engine continues to ship unpatched, and no plugin code changes were required — v9.3.1 and v9.3.2 are patch-level and introduce no new public enums or error codes for the handler's tdb_rc_to_ha to map.

What's new (inherited from upstream)

From v9.3.1 — concurrency and durability hardening:

• Five memory-safety / race fixes: clock-cache reader-pin wraparound at 128 concurrent readers, flush-cleanup use-after-free over the 16-immutable threshold, transaction-reset dangling pointer (repeatable-read / snapshot), duplicate column-family registration race, 32-bit MSVC atomics.
• Reader FD starvation fixed — engine-side counterpart to the fd-pressure behaviour the v0.3.0 100 GiB stress run documented. A flush-path descriptor-accounting leak is fixed, and reader/reaper budgets are unified so the reserve always stays available.
• Backpressure simplified — L1 file-count hard stop removed; write admission is now governed by the L0 queue stall and the active-memtable ceiling.
• Parallel compaction within a round — per-CF compaction borrows ephemeral helper threads with work-stealing and shards merge output across key-range subcompactions; ~25 % higher ingest throughput in upstream's tests with clean mid-round-kill recovery.

From v9.3.2:

• Backwards-compatible chunking of bloom filters and block indexes that exceed the 4 GB block-manager size.
• _tidesdb_cancel_background_work_ helper for quick shutdown under large flush / compaction queues.

Validation

Full suite re-run against the v9.3.2 images (see docs/v9.3.2-validation-report.md):

Gate	Result
MTR --suite=tidesdb	61/61 pass (2 skipped)
mwbench engine-integrity (8 GiB, deletes on)	0 misses / 0 mismatches
HammerDB SIGKILL crash-recovery	PASS — WAL recovered all committed work
HammerDB WARE=100 throughput	PASS — 23,500 NOPM, no OOM (v0.3.0: 22,583 — within run-to-run noise)

Migration

Drop-in for v0.3.0 users. Data directory layout unchanged (<datadir>/.tidesdb). No on-disk format change; v9.3.2's chunked bloom/block-index format is backwards-compatible.

Docker

docker pull perconalab/tidesdb-mysql:0.3.1

v0.3.0 — TidesDB engine v9.3.0 (unpatched); TDB_ERR_BUSY mapping; active-memtable ceiling

v0.3.0 — TidesDB engine v9.3.0 (unpatched)

Bumps the bundled storage engine from TidesDB v9.2.5 to v9.3.0, now shipped with zero patches, and adapts the plugin to v9.3.0's new backpressure semantics.

What changed

• Engine → TidesDB v9.3.0, unpatched. The 0001-bloomfix.patch we carried (the bloom_filter_new use-after-free, TidesDB PR #626) landed upstream verbatim in v9.3.0 — both the *bf = NULL guards and the return-checked tidesdb_partitioned_merge caller. With 0001-walfix.patch already retired at v9.2.5, the docker/patches/ engine step is gone entirely.
• TDB_ERR_BUSY (-14) mapping. v9.3.0 returns this from backpressure-stall timeouts (L0 queue, active-memtable ceiling, memory-pressure critical) that previously returned TDB_ERR_IO or TDB_ERR_MEMORY_LIMIT. The handler now maps it to HA_ERR_LOCK_WAIT_TIMEOUT (transient, statement-only rollback, retriable) instead of letting it surface as a false HA_ERR_CRASHED.
• tidesdb_default_l0_queue_stall_threshold default lowered 20 → 10, matching upstream now that v9.3.0's active-memtable backpressure ceiling (2× write_buffer_size) bounds the other growth surface.
• Inherited from v9.3.0 (no plugin change): the active-memtable ceiling, four additional use-after-free/race fixes, compaction-trigger correctness, and max_concurrent_flushes pinned to num_flush_threads.

Validation

Full suite against the v9.3.0 images (see docs/v9.3.0-validation-report.md):

Gate	Result
MTR --suite=tidesdb	61/61 pass (2 skipped)
mwbench engine-integrity (8 GiB, deletes on)	0 misses / 0 mismatches
HammerDB SIGKILL crash-recovery	PASS — WAL recovered all committed work
HammerDB WARE=100 throughput	PASS — 22,583 NOPM, no OOM

The WARE=100 profile that OOM-killed mysqld during v0.2.5 validation now runs clean to completion under v9.3.0's active-memtable ceiling.

A mwbench harness false-positive (the delete phase published its deleted-set predicate after issuing deletes, mis-scoring reads of just-deleted keys as misses) was root-caused and fixed in test tooling, then confirmed clean against the engine. To be reported upstream to tidesdb/mwbench.

Migration

Drop-in for v0.2.5 users. Data directory layout is unchanged (<datadir>/.tidesdb). No on-disk format change.

Docker

docker pull perconalab/tidesdb-mysql:0.3.0

v0.2.5 — TidesDB engine v9.2.5 (durability bugs fixed upstream; .tidesdb data dir)

TidesDB MySQL plugin — v0.2.5

Bumps the bundled engine from TidesDB v9.2.0 → v9.2.5. The four durability bugs we carried as 0001-walfix.patch are fixed upstream in v9.2.5; the only patch we still ship is the bloom_filter_new use-after-free fix (TidesDB PR #626). Also moves the default data directory inside the MySQL datadir, and adds two new automated test gates.

Docker image:

docker pull perconalab/tidesdb-mysql:0.2.5

Validation (all gates pass)

Gate	Result
SIGKILL crash-recovery (HammerDB TPC-C, docker kill -9 mid-write)	PASS — no committed-row loss, no order-row gaps
mwbench engine integrity (8.26M keys, 88 peak SSTables, byte-verified)	PASS — 0 lost writes, 0 corrupted values
TPC-C throughput (tuned, durability OFF)	25,624 NOPM @ 40 WH · 23,053 NOPM @ 100 WH, server stable

Full report: docs/v9.2.5-validation-report.md.

What changed

Engine bumped to TidesDB v9.2.5 — 0001-walfix.patch retired

All four durability bugs the walfix patch fixed against v9.2.0 were verified fixed upstream in v9.2.5, one by one:

• convert_sync_mode inversion — the engine sync enum was reordered (NONE=0, FULL=1, INTERVAL=2), making the existing switch correct. ⚠️ Applying the old walfix rewrite to v9.2.5 would now map FULL → NONE and silently re-break durability — which is why the patch is retired, not ported.
• Raw sync_mode at WAL opens — block_manager_open now calls convert_sync_mode internally.
• Unconditional WAL truncate before recovery — tidesdb_create_column_family now validates (preserves) an existing WAL and only truncates a genuinely fresh column family.
• SSTable cursor block_size caching (the v0.2.4 bug #4) — the cursor now caches the real on-disk size only when a block is read from disk, otherwise forces a re-read.

The one fix still required is docker/patches/0001-bloomfix.patch (PR #626): bloom_filter_new free(*bf)'d on its post-malloc failure paths without setting *bf = NULL, leaving a dangling pointer that the unchecked tidesdb_partitioned_merge file_max split path turned into a use-after-free / GPF in bloom_filter_add under heavy compaction.

Per-bug detail and upstream-fix verification: KNOWN-ISSUES.md.

Data directory default moved inside the datadir

The default was a sibling of the datadir (<datadir>/../tidesdb_data) — a MariaDB/TideSQL port artifact that placed engine data where MySQL backup/clone/relocation tooling doesn't expect it. It is now <datadir>/.tidesdb: inside the datadir (same volume/permissions/backup treatment as InnoDB) with a leading dot so it is never mistaken for a schema directory (a bare tidesdb_data would collide with CREATE DATABASE tidesdb_data). tidesdb_data_home_dir still overrides.

Migration: data written by an earlier build lives at the old path; move it into the datadir or set tidesdb_data_home_dir to the old location.

New test gates

• mwbench engine-integrity gate (docker/Dockerfile.mwbench + bench/mwbench/run-mwbench.sh) — builds the upstream mwbench tool against the exact shipped engine and drives heavy ingest + byte-verified concurrent reads + delete/compact, failing on any lost write or corruption. Wired into run-all.sh as a fail-fast step 0.
• Tuned TPC-C throughput profile (bench/hammerdb/run-throughput.sh) — ports tidesdb/hammer's my.cnf.example to MySQL, durability-OFF, with aligned tidesdb/innodb flags for an honest A/B. Not a durability test.

Notes

• perconalab/tidesdb-mysql:latest now points at v0.2.5.
• This remains an experimental build: durable enough that committed data survives a crash, but not something to point production traffic at yet.

v0.2.4 — bug #4 patched: full row recovery on multi-SSTable post-restart

TidesDB MySQL plugin — v0.2.4

Patches the residual durability bug v0.2.3 left localized. All four engine bugs found during the investigation are now fixed.

Tests: 61/61 MTR pass. Post-SIGKILL row recovery, HammerDB WARE=10 BUILDVU=4 RUNVU=4 + 30s NewOrder mix:

Table	v0.2.3 (broken)	v0.2.4 (fixed)	Expected
tpcc__orders per (w,d)	2,017 for d=1, 0 for d=2..10	3,145–3,186 each, all 10 districts	d_next_o_id − 1
tpcc__orders total	~307k (full-scan only)	309,943	~310k
tpcc__order_line total	2,669	3,100,951 (×1,162 recovered)	~3.1M
tpcc__stock total	717	1,000,000 (×1,394 recovered)	1M
tpcc__customer total	300,000	300,000	300k

What changed

Correctness (CRITICAL — partial silent loss on SIGKILL with multi-SSTable level 1)

Bug #4: SSTable cursor cached the wrong block_size. Four sites in tidesdb/src/tidesdb.c (tidesdb_merge_source_advance and two paths in tidesdb_iter_seek_sstable_source_forward) set cursor->current_block_size = bdata_size / = block_data_size plus cursor->block_size_valid = 1 after consuming a cached SSTable block. The value cached was the cache-entry size (decompressed block data plus appended per-entry index entries), not the on-disk block size that block_manager_cursor_next needs to add to current_pos. On the next cursor_next call, current_pos jumped by header + bdata_size + footer — which we measured at 160 MB / 318 MB / 812 MB / 1.2 GB / 1.7 GB per call. After 2–3 calls, current_pos exceeded klog_data_end_offset and the cursor returned TDB_ERR_NOT_FOUND, dropping the rest of the SSTable.

The bug fired only when post-restart level 1 contained two SSTables — one loaded from disk via tidesdb_sstable_load (the pre-kill flushed SSTable) plus one created during recovery via tidesdb_level_add_sstable (the recovery-flushed memtable). Tables with one SSTable were unaffected — that's why customer and (in some test profiles) orders returned correct counts while order_line and stock returned 0.07–0.36% of their rows.

Fix: removed the four wrong current_block_size / block_size_valid = 1 assignments. cursor_next now pread's the real 4-byte on-disk size header per block transition. One syscall per block, in the host page cache anyway — measurable but negligible cost.

Patch grew from 99 → 155 lines

docker/patches/0001-walfix.patch now covers all four engine fixes (v0.2.3's three plus v0.2.4's one), applied to the upstream TidesDB v9.2.0 clone in docker/Dockerfile.mysql before cmake. Removable once equivalent fixes land upstream.

How to verify after a future TidesDB upgrade

cd bench/hammerdb
./recovery-diag.sh
# bench/results/recovery-diag-*/snapshots.txt — post-restart counts
# must match district.d_next_o_id - 1 for each (w, d).

bench/hammerdb/run-all.sh runs the full v0.2.3/v0.2.4 suite (correctness baseline, recovery, VU sweep, head-to-head vs InnoDB, TPROC-H, sustained) and generates a self-contained REPORT.md.

Pull

docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.4

Digest: sha256:b4e226acb6713f43652d676fbf63defd28e7cbd3ed56d9da9b64ea9fb3736740

Also tagged mysql-9.7-tidesdb-latest.

v0.2.3 — WAL durability fix (silent loss on SIGKILL)

TidesDB MySQL plugin — v0.2.3

Critical durability fix on top of v0.2.2. The vendored TidesDB v9.2.0 engine silently lost every committed write on a hard process crash (SIGKILL / power-loss / kernel panic) even with the plugin requesting sync_mode=FULL — pre-kill COUNT showed the data, post-restart COUNT was 0. MySQL's own data dictionary survived; the engine's per-CF WAL was being restored to header-only across the kill+restart cycle.

Tests: 61/61 MTR pass (no regressions from the patch). 5/5 INSERTs survive docker kill -9 + restart; mixed 100-row BEGIN/COMMIT + 5-row autocommit → 105/105 recovered post-SIGKILL.

What changed

Correctness (CRITICAL — silent data loss on crash)

Bundles three stacked engine patches against the vendored TidesDB v9.2.0 as a single unified diff at docker/patches/0001-walfix.patch, applied to the upstream clone in docker/Dockerfile.mysql before the engine cmake. The patch will be removed once these fixes land upstream in TidesDB.

1. src/block_manager.c convert_sync_mode() inverted case logic. The function maps the engine's tidesdb_sync_mode_t enum (NONE=0, INTERVAL=1, FULL=2) to the block manager's two-mode enum (NONE=0, FULL=1). Upstream: case 1 returned BLOCK_MANAGER_SYNC_FULL (mapping INTERVAL → FULL) and case 2 fell through default → BLOCK_MANAGER_SYNC_NONE (mapping FULL → NONE). So a sync request of FULL silently got NONE: the block manager opened the WAL without O_DSYNC and skipped the per-block fdatasync, leaving the write only in the kernel page cache.

2. Multiple WAL block_manager_open sites in src/tidesdb.c passed the raw engine enum. The unified-memtable path (lines 17269, 23298), per-CF path (line 18600), and rotation reopens (lines 19174, 19190) passed config->sync_mode / cf->config.sync_mode / umt_sync_mode directly to block_manager_open without going through convert_sync_mode(). With (1) fixed, these still wired the wrong enum value into the block manager, so the fix had to extend to every WAL-open call site.

3. tidesdb_create_column_family unconditionally truncated the WAL. The function is invoked both for fresh CREATE TABLE and during database open when an existing CF directory is rediscovered on disk. It called block_manager_truncate(new_wal), which wipes the WAL to header-only — running before recover_wals had a chance to replay it. So with sync now working, recovery still saw an empty WAL. Replaced with block_manager_validate_last_block(PERMISSIVE), which writes the header for a 0-byte file, leaves a valid header-only file alone (fresh CF case), and forward-scans + sets current_file_size to the last valid block for an existing WAL with data (recovery case). Same observable behaviour as truncate for fresh, preserves data for recovery.

Pull

docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.3

Digest: sha256:d2a80d515d392f93975bde76ed3d9d19d5c64744521a5114c5bd180e57108341

Also tagged mysql-9.7-tidesdb-latest.

v0.2.2 — silent bulk-load data-loss fix (unified_memtable default OFF)

TidesDB MySQL plugin — v0.2.2

Correctness release on top of v0.2.1. One critical fix that changes a
default to make the engine safe out of the box, one latent silent-loss
fix in the bulk-commit path, and the reverse-ref/OSTAT handler fix that
landed on main between tags.

Tests: 61/61 MTR pass (the tidesdb_unified_memtable test now opts
into the unified path explicitly via a -master.opt so it stays
covered after the default flip). HammerDB 5.0 TPROC-C WARE=20 / BUILDVU=8
/ RUNVU=16 verified PASS out of the box (2511 NOPM, 0 handler-unsupported
errors, schema build + reverse-ref/OSTAT clean).

What changed

Correctness (CRITICAL — silent data loss)

• Default tidesdb_unified_memtable=OFF. The TidesDB v9.2.0 unified
  WAL+memtable path silently loses committed rows under many concurrent
  writers. HammerDB TPROC-C at WARE=20/BUILDVU=8 dropped ~95% of
  bulk-loaded rows (MIN(o_id)=3001 across every district while
  district.d_next_o_id ~ 3000 — 100% of build-phase rows gone, only
  post-build single-row inserts survived). Every tidesdb_txn_commit
  returned TDB_SUCCESS and all loaders reported FINISHED SUCCESS.
  Refuted handler-side hypotheses (skip_list deep-copies key+value,
  ruling out txn_reset-after-commit corruption) and visibility lag
  (stable across 20s of repeated reads). Flipping the same handler
  binary to per-CF memtables produced a 100%-correct build. Until the
  engine fixes the rotation race, default the sysvar to OFF so
  correctness holds out of the box. Opt back in explicitly with
  tidesdb_unified_memtable=ON for low-concurrency multi-table OLTP.

  MTR coverage of the unified path is preserved via a per-test
  tidesdb_unified_memtable-master.opt that forces ON for that one
  test. Four result files were regenerated; their previous baselines
  encoded unified-mode degenerate stats (literal FAIL: DATA_LENGTH is 0 in tidesdb_info_schema, bogus rows=2 after ANALYZE that
  prevented MRR) — per-CF correctly reports non-zero data length,
  accurate row counts, and the optimizer now picks range/MRR scans on
  IN-lists.

Correctness (HIGH — latent silent loss in bulk path)

• maybe_bulk_commit no longer silently swallows a failed mid-batch
  commit. A failed tidesdb_txn_commit in the bulk insert/update/
  delete path was downgraded to an info log; tidesdb_txn_reset then
  discarded the batch's buffered ops, yet the function returned 0, so
  the caller saw success and the loader reported FINISHED SUCCESS
  while up to TIDESDB_BULK_INSERT_BATCH_OPS (500) rows per batch were
  silently lost. The fix retries transient resource errors (TDB_ERR_ CONFLICT / LOCKED / MEMORY_LIMIT) up to 4 attempts with a short
  backoff — tidesdb_txn_commit returns these errors before marking
  the txn aborted, so the buffered ops survive and re-committing is
  safe — then on non-transient errors or exhausted retries returns the
  mapped error without resetting the txn, so the SQL layer rolls the
  statement back instead of corrupting the table.

Correctness (HIGH — handler API)

• Reverse-ref / OSTAT (ER_ILLEGAL_HA on ORDER BY pk DESC LIMIT 1).
  ha_tidesdb now implements index_read_last_map and fixes the
  partial-prefix HA_READ_PREFIX_LAST seek so that
  WHERE w_id=? AND d_id=? ORDER BY o_id DESC LIMIT 1 (the exact
  TPC-C Order-Status query) no longer surfaces error 1031. Verified
  via 6 dedicated MTR cases plus a full HammerDB 5.0 TPROC-C build+run.

Pull

docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.2

Digest: sha256:7dc1d358470ed9c13e5cf31b189e61dc2ae0572a6b8557467454d172a0131bb3

Also tagged mysql-9.7-tidesdb-latest.

v0.2.1 — modular refactor + ADD FULLTEXT back-populate

TidesDB MySQL plugin — v0.2.1

Maintenance release on top of v0.2.0. Large internal restructuring (no
SQL/feature surface change), one user-facing fix, plus a HIGH defect that
fix introduced and which is now resolved before tagging.

Tests: 37/37 hand-rolled + 60/60 MTR pass (2 intentional skips —
MariaDB-only VECTOR, native partitioning). tidesdb_ttl (timing-
sensitive, historically flaky under full-suite pressure) passed in the
release run.

What changed

User-facing fix

• ALTER TABLE … ADD FULLTEXT INDEX now back-populates existing rows.
  Previously the inplace-ALTER populate scan skipped FULLTEXT indexes, so
  adding a FULLTEXT index to a table that already had rows left those rows
  unsearchable — only rows written after the index existed matched.
  The populate scan now tokenizes and indexes existing rows, mirroring the
  write_row FTS path (term/df codec, BM25 meta counters), with correct
  field-pointer rebasing onto the decoded row buffer.

Correctness (HIGH, found and fixed during this cycle)

• F-1: the new back-populate wrote FTS meta counters per-row into the
  data CF in batches. An aborted/KILLed build (rollback drops only the new
  index CF, not the data CF) left inflated total_docs/total_words, so a
  retry double-counted and skewed BM25 ranking. Fixed: per-index
  accumulators flushed in a single post-final-commit txn; aborted builds
  now write no meta and a retry starts clean. New regression test
  tidesdb_fts_add_index_keynr covers DROP+ADD-in-one-ALTER (key-number
  shift) and a DROP/re-ADD identical-ranking check.

Internal restructuring (no behavior change, verified)

ha_tidesdb.cc went from ~12k to ~8.5k lines; cohesive subsystems were
extracted into focused translation units, each behavior-preserving and
regression-clean:

Module	Contents
tidesdb_master_key.{h,cc}	at-rest encryption / master-key subsystem
tidesdb_row_lock.{h,cc}	pessimistic row-lock manager + deadlock walker
tidesdb_engine_context.{h,cc}	engine handle / schema CF / conflict info
tidesdb_fts.{h,cc}	full-text search subsystem
tidesdb_spatial.{h,cc}	Hilbert/MBR spatial index subsystem
tidesdb_portability.{h,cc}	MariaDB→MySQL OS/runtime shims (real impls)
tidesdb_inplace_alter.{h,cc}	online-DDL state machine

tidesdb_compat.h is now exclusively MariaDB→MySQL renames / type
aliases / #define-as-0 stubs.

Known issues (pre-existing, tracked)

A deep multi-agent review of the extractions and the FTS fix
(docs/code-review-A2-A7-report.md) closed F-1 (above) and catalogued 8
lower-severity pre-existing items (TTL not threaded through the
populate loop, populate put best-effort-continue, uint16 tf wrap,
two unsanitized sysvar-update log lines, encrypted-flag INPLACE flip,
unregistered FTS PSI keys, duplicate stop-word, strxnmov off-by-one).
None are regressions in this release; all are documented in the report
with recommended fixes for a follow-up hygiene pass.

Runnable image — published on Docker Hub

docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.1
# or the moving tag
docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-latest

Image digest: sha256:822fe343c7788318a5a1f9a3f6eed0af2bc8349d2005cc5cf033c416848eabe0
(both tags resolve to this digest — bit-identical to the locally
smoke-tested image)
Size: 967 MB
Base: Oracle Linux 9 + gcc-toolset-14 (Stage 1) → mysql:9.7 (Stage 2)

Quick smoke

docker run --rm -d --name tidesdb-v021 -p 3307:3306 \
  -e MYSQL_ALLOW_EMPTY_PASSWORD=1 \
  evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.1

# wait ~10s for mysqld, then:
docker exec -it tidesdb-v021 mysql -uroot -e \
  "SELECT engine, support, transactions FROM information_schema.engines WHERE engine='TidesDB';"
# expect: TidesDB | YES | YES

To rebuild from source: scripts/release-image.sh v0.2.1 (add --push
to republish; without it, builds + smoke-tests locally with no
credentials, tagging tidesdb/mysql:9.7).

v0.2.0 — post-review hardening

TidesDB MySQL plugin — v0.2.0

First release after two full code-review rounds. 50 findings closed across CRITICAL / HIGH / MEDIUM / LOW severity, plus the previously-undocumented "ENGINE_ATTRIBUTE freeze" constraint. All 37/37 hand-rolled and 58/58 MTR tests pass (2 intentional skips unchanged).

Pull

docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.0
# or
docker pull evgeniypatlan/test-images:mysql-9.7-tidesdb-latest

Image digest: sha256:5001022248f83b923b24f971aabd0e8b145f42c549315b5596d841728b8de516
Size: 967 MB
Base: Oracle Linux 9 + gcc-toolset-14 (Stage 1) → mysql:9.7 (Stage 2)

Quick smoke

docker run --rm -d --name tidesdb-v020 -p 3307:3306 \
  -e MYSQL_ROOT_PASSWORD=root \
  evgeniypatlan/test-images:mysql-9.7-tidesdb-v0.2.0

# wait ~10 seconds for mysqld to come up
docker exec -it tidesdb-v020 mysql -uroot -proot -e \
  "SELECT engine, support, transactions FROM information_schema.engines WHERE engine='TidesDB';"
# expect: TidesDB | YES | YES

What changed

The full review reports landed alongside the fixes (docs/code-review-report.md, docs/code-review-followup-report.md). Headline items below.

CRITICAL (3 fixed)

C-1	Encryption silently emitted ciphertext under uninitialized stack key/IV when encryption_key_get or my_random_bytes failed. Now checks both return values, secure-zeroes the stack key on every exit path via tdb_secure_zero (and now explicit_bzero where available).
C-2	tdb_global was a plain pointer cleared on shutdown without synchronization vs concurrent handler threads — null-deref crash on shutdown-under-load. Now std::atomic<tidesdb_t *> accessed via tdb_get_engine() / tdb_set_engine(); shutdown uses exchange(nullptr, acq_rel).
CF-1	The H-3 trx-lifecycle rwlock rollout missed tidesdb_hton_kill_query — a UAF if KILL QUERY raced a connection close. Now takes the read-lock around the trx deref.

HIGH (13 fixed)

• H-1 deadlock detector now re-runs on each cond_wait wakeup (was indefinite hang under specific contention patterns).
• H-2 / HF-2 FTS doc/word counter RMW serialized — now per-share (was process-global mutex serializing unrelated tables).
• H-3 deadlock walker derefs are protected by a global mysql_rwlock_t (g_trx_lifecycle_lock); close_connection takes the write lock around my_free(trx).
• H-4 Hilbert spatial encoder operator-precedence bug — (uint32_t)s << 1 was truncating before shift; widened hilbert_rot's n to uint64_t. Roughly half of all spatial keys were encoded wrong pre-fix.
• H-5 tidesdb_s3_secret_key + tidesdb_s3_access_key hidden via PLUGIN_VAR_NOSYSVAR (no longer visible via SHOW VARIABLES to anyone with SYSTEM_VARIABLES_ADMIN).
• H-6 / HF-3 path-traversal defense on tidesdb_backup_dir / tidesdb_checkpoint_dir: rejects relative paths and .. components; new tidesdb_backup_allowed_root sysvar for full confinement.
• H-9 ENGINE_ATTRIBUTE JSON parser switched to kParseIterativeFlag + 64KB length cap — no stack overflow / heap exhaustion via deeply nested JSON.
• H-10 ICP was advertised but the check stub always returned 0 — secondary-index range scans leaked rows past end_range. Now actually evaluates pushed_idx_cond->val_bool() + compare_key_icp + thd_killed. Pre-fix MTR baselines for tidesdb_sql / tidesdb_stress were encoding the bug.
• HF-1 M-12 stopword-table privilege check now fail-closed on NULL THD.
• HF-4 backup/checkpoint refuse immediately if THD already killed (uncancellable from MySQL side once started).

MEDIUM (21 fixed)

• ENGINE_ATTRIBUTE options cached on the share with atomic publish (was per-call JSON re-parse + 25 THDVAR reads).
• FTS doc/word delta buffered during bulk INSERT, flushed once at end_bulk_insert (was 1 RMW per row).
• TLS-cached master key with generation-counter invalidation — no mutex on the hot decrypt path.
• Master-key page mlock + MADV_DONTDUMP; smart 1-page-vs-2-page based on actual straddle.
• Per-handler scratch buffers via std::unique_ptr thread-local pointer (destructor runs at thread exit; no pooled-thread leak).
• getrandom(2) for IVs (was open/read/close /dev/urandom per row).
• Log-injection defense — tdb_sanitize_for_log on user-supplied strings.
• Stopword loader's table scan moved outside the write-lock.
• key_unpack_scratch_ sized at open() rather than per-seek.
• ... plus a dozen more, see docs/code-review-report.md.

LOW (13 fixed)

S3 endpoint/bucket redacted in error log; explicit_bzero where available; lock-order documentation; null-byte path check; end_bulk_insert propagates flush rc; mlock errno mapped to operator hints; tests renamed from finding-ID to feature-based.

ENGINE_ATTRIBUTE-freeze constraint closed

ALTER TABLE t ENGINE_ATTRIBUTE='{"...":...}' previously appeared to succeed (DD updated, SHOW CREATE TABLE reflected the new value) but the engine kept using the pre-alter parse for the share's lifetime. Now commit_inplace_alter_table computes fresh opts and atomically swaps the share's cached pointer, plus refreshes share->default_ttl / share->isolation_level / share->encrypted. New regression test: tidesdb_alter_engine_attribute.

What's deferred (and why)

Three findings remain open by design, with code comments documenting the rationale:

• M-6 Field::pack virtual-dispatch fast path — codec hot path; correctness risk too high without benchmarks proving the speedup.
• M-10 Lock-entry recycling — bound to the H-3 UAF invariant that lock entries are never freed. Recycling needs extending the rwlock to cover lock-entry lifetime. Memory growth bounded per distinct PK ever locked.
• LF-5 Per-row atomic acquire-load of g_master_key_gen — single mov on x86 with no fence; eliminating it would require API change to plumb a handler-level cache into the free function. Disproportionate effort for negligible gain.

Architectural work outstanding (not bugs, not blocked)

Per the refreshed architect's priority list at the end of docs/code-review-followup-report.md:

1. Extract EngineContext + promote tidesdb_keyring_compat.cc to a real tidesdb_crypto/ module.
2. Extract row-lock manager to tidesdb_row_lock.{h,cc} with an opaque RowLockManager.
3. Extract FTS to tidesdb_fts.{h,cc} (4 duplicated thread_local scratch sites collapse into one FtsIndex).
4. TidesStore abstraction over the TidesDB C API.
5. MySQL 9.7 atomic-DDL participation (SDI callbacks).
6. Delete #if 0 dead corners and MariaDB-only methods.
7. Explicit state machine for inplace ALTER.
8. Audit / shrink tidesdb_compat.h.

Breaking changes

None. v0.2.0 is a drop-in replacement for v0.1.0.

Acknowledgements

Two thorough multi-lens code reviews (C++ correctness / security / performance / architecture) ran across plugin/ and surfaced every finding in this release. The reports themselves (docs/code-review-report.md, docs/code-review-followup-report.md) are committed alongside the code so the audit trail is preserved.