benchmark: add R10 + awa, retire legacy /benchmarks/#157
Merged
Conversation
Adds the 8-system R10 round on top of the existing R7/R8/R9 harness: awa as the 8th queue system, driven by its native Python worker (`pip install awa-pg`) so the methodology stays "use each system's native API" that R7-R9 already follow. New artifacts under benchmark/: - install/install_awa.sh — pip install awa-pg awa-cli + awa migrate - producers/producer_awa.py — async batched insert at -R 2000 - consumers/consumer_awa.py — 4 worker tasks; emits NOTICE-format ev/s lines so parse_events_consumed.py works unchanged - runners/run_r10.sh — 50m phase orchestrator (10m clean + 30m idle-in-tx + 10m recovery); branches on awa to dispatch the Python producer/consumer instead of pgbench - tooling/idle_in_tx.sh — psql-based REPEATABLE READ tx holder (no psycopg2 dep; supersedes idle_in_tx.py on bare AMIs) - charts/r10_throughput_chart.py — 8-panel consumer-throughput timeline with per-phase averages - charts/r10_sysmetrics_chart.py — CPU + NVMe write MiB/s + IOPS time-series, all 8 systems overlaid - charts/r10_ash_chart.py — Performance-Insights-style ASH chart using the pg_ash colour convention - charts/r10_summary_table.py — per-system summary (producer/consumer totals, true backlog, peaks) - charts/r10_ash_table.py — per-phase wait-event mix - R10.md — round write-up + headline result README.md updated to mention awa, R10, and the new run_r10.sh entry point; directory layout reflects the new files. Removes the legacy /benchmarks/ directory (4 files: insert_bench.sql, consumer_bench.sql, dead_tuple_check.sql, run_benchmarks.sh). Those were single-system pgque-only psql tests subsumed by the harness in benchmark/ — keeping both directories was just a confusing naming collision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-R 2000, freshi4i.2xlargespot VMs in us-east-2).benchmark/:install_awa.sh,producer_awa.py,consumer_awa.py,run_r10.sh, the four R10 chart/table scripts (r10_throughput_chart.py,r10_sysmetrics_chart.py,r10_ash_chart.py,r10_summary_table.py,r10_ash_table.py), and a psql-basedidle_in_tx.sh(nopsycopg2dep — supersedes the Python version on bare AMIs).benchmark/R10.mdwith the round write-up + headline result.run_r10.shentry point; directory layout reflects the new files./benchmarks/directory — four single-system pgque-only psql tests (insert_bench.sql,consumer_bench.sql,dead_tuple_check.sql,run_benchmarks.sh) subsumed by the multi-system harness inbenchmark/. Keeping both directories was just a confusing naming collision.R10 round results: https://gitlab.com/postgres-ai/postgresql-consulting/tests-and-benchmarks/-/issues/77#note_3304910299
Headline (R10-specific, non-pgque): awa shows TX-avg throughput 68 ev/s (3.4 % of offered) and 5.33 M events outstanding at end of run, with ASH dominated by
Lock:transactionidwaits across all phases. R7-R9 systems reproduce.Test plan
benchmark/runners/run_r10.sh awa 3000exercises the awa branch (validated on a live VM during the round)benchmark/runners/run_r10.sh <other>exercises the pgbench branch unchanged from R9/tmp/bench_r10/<sys>/data produced during the round🤖 Generated with Claude Code