===========================================
MariaDB Foundation: TAF 4.0 Release Announcement
===========================================
The MariaDB Foundation is pleased to announce a major update to the 4.0 version of the Test Automation Framework (TAF).
This release delivers significant improvements in reproducibility, configuration provenance, PostgreSQL support, and test-case portability.
The work represents a substantial step forward in making database performance testing more reliable, transparent, and self-contained.
Major Implementations
1) PostgreSQL Support (Beta)
TAF now includes end-to-end PostgreSQL support:
* PostgreSQL Database Plugin (PR #7 integrated)
* Sysbench-lua updated for PostgreSQL builds
* RUN SQL library updated for PostgreSQL
This expands TAF’s multi-database testing capabilities and ensures consistent behavior across MariaDB, MySQL, and PostgreSQL.
This is beta addition, and all feedback and pull requests around are appreciated!
2) Embedded DB Configuration in User Properties
TAF now supports embedding database configuration directly inside the user properties file using:
Code
[db_config_start]
...
[db_config_end]
Precedence order:
1. CLI overrides
2. Inline DB config block
3. db_config_file reference
The winning configuration is written into the auto-generated test-case properties file, used to generate the temporary DB configuration file, which is archived with results, and passed to reporters and backend ingestion.
This eliminates configuration drift and ensures that a single properties file fully defines the test case.
3) Auto-generated Database Configuration File
After precedence selection, TAF generates a temporary database configuration file at runtime.
This auto-generated file is:
- built directly from the winning configuration
- used to start the database
- archived with the results bundle
- included in the auto-generated test-case properties file
- passed to reporters and backend ingestion
This guarantees that the exact configuration used at runtime is always captured, reproducible, and traceable.
4) Auto-generated Test Case User Properties Files
Each test case now produces a standalone properties file containing:
* original properties (minus db_config_file)
* CLI overrides converted into properties
* the winning DB configuration
This file is archived with results and allows the test case to be rerun independently of the original configuration files. Test cases are now self-contained, reproducible, and diff-friendly.
5) README.txt Improvements
Each test-case README now includes:
* the DB configuration used for the run
* the auto-generated test-case user properties file
Reporters and backend ingestion now rely on run metadata rather than external configuration files, improving provenance and consistency.
6) Recovery Mode Improvements
Recovery mode has been enhanced to re-run iteration 1 of the recovered data store to eliminate skew and ensure synchronization. All changes from PR #9 are included.
7) New TAF Usage Options
New options added:
--test-case-tag / taf.test_case_tag
--dump-run-state / taf.dump_run_state
--exec-script-file-before-tests
--exec-script-file-after-tests
These options improve test documentation, reproducibility, and pre/post-run automation.
8) New Database Configuration Files
New comparable, minimal, default, analytics, and OLTP configuration files have been added for MariaDB, MySQL, and PostgreSQL.
9) TAF Backend and Reporters
- Backend now retrieves DB configuration from run metadata
- Reporters updated to use raw run data for configuration
This resolves long-standing issues around configuration provenance.
10) ResultsCompareRaw.pl (New)
A new tool allows regenerating HTML from a test-case raw results file, targeting metrics other than the suite primary.
Engineers may produce new HTML reports with any metric promoted to primary using:
Code --metric-name=<metric_name>
11) System Saturation Collector Scripts
New scripts added:
- taf_collect_mpstat.sh
- taf_collect_perf_sched.sh
- taf_collect_pidstat.sh
- taf_collect_sarq.sh
- taf_start_collectors.sh
- taf_stop_collectors.sh
12) Test Suite Improvements
All suites:
- added client_cpu_affinity
- hammerdb_tprocc / hammerdb_tproch:
- added state directory cleanup options
- removed timing options not applicable to HammerDB
sysbench_lua:
- added NormalizeDBType
13) New TidesDB Configuration and Test-Case Properties
TidesDB author Alex Padula provided additional TidesDB configuration and test-case properties files for inclusion in TAF 4.0 (PR #8).
Configuration files:
- database_config_files/mariadb/innodb_compare_local.cnf
- database_config_files/mariadb/tides_compare_local.cnf
- database_config_files/mariadb/tides_compare_v10.cnf
Properties files:
- properties/mariadb/sysbench_local_tidesdb_innodb.properties
- properties/mariadb/sysbench_tidesdb10_innodb_rocks_compare.properties
14) RenameResultsTestCaseTag Utility Added
A new post-processing utility, RenameResultsTestCaseTag.sh, normalizes archived result directories and their associated raw, text, CSV, and JSON report files by renaming them according to the authoritative test_case_tag value found in each result’s primary report.
This ensures consistent naming across all generated artifacts and improves downstream comparison, ingestion, and archival workflows.
Patches and Fixes
1) Test Suite Framework
- fixed README metadata timing for multi-test runs
- dynamic fields moved from WriteReadmeStart to WriteReadmeEnd
- ensures each test’s README reflects its correct configuration
2) DatabaseSoftwareInstall
Major hardening improvements: restored correct package-list parsing (scalar → arrayref) tightened error severity (warnings → errors)
refusal to install if target directory already exists hard failure if extraction does not produce valid install_root hard failure on invalid or empty package validation hard failure if install directory name cannot be derived improved ACTIVE selection logic and reprint
These changes improve safety, correctness, and diagnostic clarity.
3) MariaDB / MySQL Startup (SSL)
TAF’s background-spawn wrapper for MariaDB and MySQL was over-hardened (setsid(), FD-closing, STDERR duplication), interfering with WolfSSL/OpenSSL initialization and causing intermittent SSL startup failures.
The daemon wrapper was rewritten to a minimal fork/exec model, restoring reliable SSL startup for both engines.
4) HammerDB TPROC-C SSL Configuration
The TPROC-C suite emitted SSL settings into the wrong dictionary and used incorrect HammerDB keys, causing SSL to be ignored for MariaDB/MySQL and misconfigured for other engines.
SSL configuration was moved to the correct connection dictionary and updated to use the proper HammerDB parameters for each database type.
Contributor Acknowledgements
TAF 4.0 includes contributions from multiple engineers across the MariaDB ecosystem.
The MariaDB Foundation would like to thank the following contributors for their work:
1) Virtuozzo (PR #7 and PR #9)
Lukas Oliva lukas.oliva@virtuozzo.com Contributed the PostgreSQL Database Plugin and integration work included in PR #7.
Viktor Ganeles viktor.ganeles@virtuozzo.com Contributed the recovery-mode resynchronization improvements included in PR #9.
Virtuozzo is a MariaDB Foundation sponsor, and we appreciate their continued support and engineering contributions to TAF.
2) PostgreSQL Testing & Patches
Amrendra Kumar amroo76@gmail.com Performed early PostgreSQL testing and identified issues with PostgreSQL .deb installation, improving TAF’s PostgreSQL installation workflow and validation. He provided 2 different patches, one around TPROC-C setup, and other with Sybench-lua build for PG.
3) TidesDB (PR #8)
Alex Padula me@alexpadula.com Author/Creator of TidesDB provided additional TidesDB configuration files and test-case properties for inclusion in TAF 4.0.
Acknowledgment
TAF 4.0 was engineered, validated, and stress-tested on high-performance hosts provided by MariaDB Foundation sponsor Hetzner.com.
Hetzner.com hardware directly enabled full multi-database workload testing, reproducible performance runs, and the scale required to expose real behavior across MariaDB, MySQL and PostgreSQL.
Hetzner Sponsor Host (hz-bench)
CPU: 13th Gen Intel(R) Core(TM) i5-13500
CPU Count: 20
Core Count: 14
Socket Count: 1
RAM: 62.33 GB
OS: AlmaLinux 9.7 (Moss Jungle Cat)
Kernel: 5.14.0-811.35.1.el9_7.x86_64
Hetzner Sponsor Host (hz-bench2)
CPU: AMD EPYC 9454 48-Core Processor
CPU Count: 96
Core Count: 48
Socket Count: 1
RAM: 124.98 GB
OS: AlmaLinux 9.8 (Olive Jaguar)
Kernel: 5.14.0-687.29.1.el9_8.x86_64
These systems provided the horsepower required to run full TAF validation, expose performance boundaries, and generate reproducible results for MDEV performance engineering.
We appreciate the outstanding support and the quality of the systems Hetzner.com provided throughout development. If you need hosting that’s reliable, affordable, and capable of handling serious engineering workloads, Hetzner.com delivers.
==================================
End of Announcement
==================================
Full Changelog: v4.0...v4.0