docs: bench methodology + tooling under benchmark/

[NikolayS]

Summary

Adds a complete `benchmark/` directory documenting the methodology, tooling, and operational lessons from the pgque-vs-pgq-vs-pgmq-vs-river-vs-que-vs-pgboss-vs-pgmq-partitioned bench that backs #61 (the held-xmin bloat issue) and #62 (the subscription/tick rotation fix).

The content is strictly additive — no pgque production SQL is touched. Everything new sits under `benchmark/`.

Contents

• `benchmark/README.md` — entry point + quick-start.
• `benchmark/METHODOLOGY.md` — the bench methodology: workload shape, observability stack (pg_ash + pg-flight-recorder), phase scheduler, per-system patterns. Hardware: AWS i4i.2xlarge, PG18, 1.5 h with 30-min held-xmin window.
• `benchmark/OPS_GOTCHAS.md` — 15 operational lessons from running this on AWS (NVMe auto-mount, pg_partman stale `part_config` rows, que `que_validate_tags` leftovers, pgboss covering index, pgq ticker daemon, pgque PG18 xid8 bug, spot reclaim mitigation, ASH prereqs, NOTICE-based instrumentation, etc.).
• `benchmark/HARDWARE.md` — i4i.2xlarge specs, PG tuning, expected microbench baselines.
• `benchmark/tooling/` — 7 samplers + daemons: `bloat_sampler.py`, `sys_metrics_sampler.py` (v2 with IOPS + latency), `pg_stat_statements_snapshot.py`, `parse_events_consumed.py`, `idle_in_tx.py` (the inducer), `pgq_ticker_daemon.py`, `microbench.sh`.
• `benchmark/runners/` — `run_r7.sh` (phase-scheduled orchestrator), `clean_reinstall.sh`, `fix_nvme_mount.sh`.
• `benchmark/consumers/` + `benchmark/producers/` — the 7 instrumented SQL claim patterns (DO + `RAISE NOTICE 'ev ts=% n=%'`), one per queue system.
• `benchmark/install/` — per-system installers + shared `bootstrap.sh` (PGDG PG18 + pg_cron + pg_stat_statements + pg_ash + pgfr) + `install/README.md`.
• `benchmark/charts/` — `r5_analyze.py`, `r6_smoke_chart.py` (linear-scale only, per project rule).
• `benchmark/gifs/` — `r4_gif_v17_solarized.py` (dead-tuples animated), `r4_gif_tps_solarized.py` (TPS/latency animated).

Why now

PR #62 is the rotation fix for the held-xmin bloat pattern from #61. This PR is the evidence PR: the harness that proves #62 works, the ops playbook so a reviewer can reproduce, and the catalog of operational surprises we hit so the next person doesn't rediscover them.

Sanitation

• All [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3} matches in committed files are 127.0.0.1 (DB connections on the VM itself) — no AWS IPs, no public IPs.
• Docs use <pgque-ip>, <your-ssh-key> placeholders.
• No tokens, no AWS creds, no SSH private keys, no DB passwords.
• Scripts that run on the VMs keep their 127.0.0.1 connection strings since that's operationally correct for local psql / pgbench.

Test plan

• Review METHODOLOGY.md cross-references (all in-doc links resolve to files that exist under benchmark/).
• Review OPS_GOTCHAS.md for completeness — any ops lesson missing?
• Spot-check install/bootstrap.sh for any leaked IPs or tokens before merge.
• Sanity-run runners/run_r7.sh pgque on a fresh i4i.2xlarge once PR Rotate subscription and tick tables to avoid held-xmin bloat (#61) #62 lands.

Refs: #61, #62.

[NikolayS]

PR #66 Security Audit — NikolayS/pgque docs/bench-methodology

Audited commit: c8618af (tip of origin/docs/bench-methodology)
Scope: 39 files / 3,366 LOC under benchmark/
Auditor: automated scan (read-only, no modifications)

Summary

PASS — no secrets found. Ready to mark ready-for-review.

Hard secrets scan

Pattern	Findings
AWS access keys (AKIA…)	0
AWS secret key references (aws_secret_access_key / aws_access_key_id)	0
GitHub tokens (ghp_, gho_, ghs_, github_pat_)	0
GitLab tokens (glpat-…)	0
Slack tokens (xoxb-…)	0
Private key headers (BEGIN OPENSSH/RSA/EC/PRIVATE/DSA)	0
PG credential files (pg_service.conf, .pgpass)	0
Quoted password = '...'	0
Postgres URIs with embedded password (postgres://user:pw@…)	0
Bearer … tokens	0
Raw 64-hex tokens	0

Soft leakage

Pattern	Findings	Notes
Public IPv4 addresses	0	Only 127.0.0.1 appears; no RFC1918-external, no public IPs
arn:aws:…	0
12-digit AWS account IDs	0
~/.ssh/… or *.pem paths	0
EC2 public hostnames (ec2-…compute.amazonaws.com)	0
EC2 instance IDs (i-…)	0
PGPASSWORD, export *_TOKEN=…, curl Authorization headers	0
Inline ssh user@host / scp … commands	0
Base64-looking long strings	1 (false positive — Go import path com/riverqueue/river/riverdriver/riverpgxv5 in install/install_river.sh:34)

IPv4 references (all loopback, expected)

All 12 IPv4 hits in the tree are 127.0.0.1 in passwordless local DSNs, consistent with the pg_hba trust-for-localhost bootstrap in install/bootstrap.sh:77. These are expected for single-VM benchmarks:

• tooling/idle_in_tx.py:8, tooling/bloat_sampler.py:10, tooling/pg_stat_statements_snapshot.py:16, tooling/pgq_ticker_daemon.py:9
• install/install_river.sh:13,53, install/install_pgboss.sh:13,61
• install/bootstrap.sh:77
• runners/run_r7.sh:55,62,69

Historical commits

• Commits touching benchmark/ on this branch: 1 (c8618af)
• Removed-but-still-in-history secrets: none
• Full git log -p -- 'benchmark/**' scanned for AKIA|ghp_|gho_|ghs_|glpat|xoxb|BEGIN (OPENSSH|RSA|EC|PRIVATE|DSA)|aws_secret|bearer — zero matches

No squash / force-push required.

File-type specific checks

• Shell scripts (install/*.sh, runners/*.sh, tooling/*.sh): no hardcoded credential env vars
• Python (tooling/*.py, charts/*.py, gifs/*.py): all DSNs are passwordless localhost; no dsn=…password=… or psycopg2.connect(…, password=…)
• Markdown (README.md, METHODOLOGY.md, HARDWARE.md, OPS_GOTCHAS.md, install/README.md): zero hits on AWS_, aws_access_key_id, EC2_, S3_BUCKET
• SQL (consumers/*.sql, producers/*.sql, install/pgmq-partitioned_setup_5min.sql): no credentials

Recommendation

PASS — the branch is safe to mark ready-for-review. No redaction, no force-push needed.

Non-blocking observations (not required for merge):

1. DSNs throughout tooling/*.py and install/*.sh assume user=postgres with trust auth on 127.0.0.1. This is correct for the documented benchmark VM setup but worth a one-line note in benchmark/README.md so that copy-pasters don't run these against a prod-like host. Already partially covered by OPS_GOTCHAS.md — minor.
2. None of the scripts take credentials from env vars (no PGPASSWORD fallbacks). If the methodology is ever extended to RDS/remote PG, reviewers will want os.environ.get("PGPASSWORD") hooks rather than hardcoded DSN strings. Not a security issue today.

---

Scan methodology: hard-pattern regex (AWS/GitHub/GitLab/Slack tokens, private-key headers, quoted passwords, DSN-embedded credentials, Bearer tokens, 64-hex), soft-pattern regex (public IPv4, ARN, 12-digit account IDs, SSH/PEM paths, EC2 hostnames & instance IDs, base64-looking blobs), env-var extraction, and full branch git log -p sweep.

[NikolayS]

REV Review — PR #66 (post-scrub)

CI: 7/8 pass (test 14/15/16/17/18, client-smoke, verify); claude-review pending
Anti-leak scan (independent verification): HITS — 2 file references to private GitLab WI #77 + 1 commit-message leak (note ID + "issue #77" in subject)
Verdict: FIXES NEEDED

Blocking

• [HIGH] anti-leak — benchmark/install/bootstrap.sh:41 and :44 still reference issue #77:

  # ── postgresql.conf tuning (same as issue #77) ───
  # ── pgq bench tuning (issue #77) ─────────────────
  

  GitHub issue small formatting tweak in reference.md #77 in this repo is unrelated (a reference.md formatting tweak by The-Alchemist). These two comments are residue from the private WI small formatting tweak in reference.md #77 in postgres-ai/postgresql-consulting/tests-and-benchmarks and per project rule (MEMORY.md: "WI refactor: rename to XXX; clarify log vs task queue #76/small formatting tweak in reference.md #77 PRIVATE — never link from public") must not appear in the public repo. The scrub commit 03bc107 claimed it cleaned R4/R5/R6/R8 round refs but missed these two bootstrap.sh comments. Suggested fix: drop the (... issue #77) parenthetical entirely — leave just # postgresql.conf tuning / # pgq bench tuning.

• [HIGH] anti-leak — commit be9c9d0 subject + body:

  • Subject: docs: complete bench methodology + tooling + ops gotchas under benchmark/ (issue #77) — leaks WI small formatting tweak in reference.md #77 in the commit subject (permanent in git history once merged).
  • Body: METHODOLOGY.md: adapted from GitLab #77 note 3263767264 — leaks both GitLab #77 and the internal note ID.

  The scrub commit 03bc107 only edited file contents and the PR description; it did not amend the prior commit. Recommended action before merge: rebase + reword commit subjects/bodies of be9c9d0 and 03bc107 to drop GitLab/WI small formatting tweak in reference.md #77 references (the scrub commit's own message also enumerates GitLab URLs, note IDs, R4/R5/R6/R8, etc., which is itself documenting the leak vector). Even if reworded, squash-merging would also flatten this — but a rebase is cleaner.

Non-blocking

• [MED] guidelines — shell style violations. CLAUDE.md requires set -Eeuo pipefail; the e94ece7 polish only fixed 3 of 9 shell scripts. Still non-conformant:

  • benchmark/install/bootstrap.sh:2 — set -euo pipefail
  • benchmark/install/install_pgmq.sh:2 — set -euo pipefail
  • benchmark/install/install_pgq.sh:2 — set -euo pipefail
  • benchmark/install/install_river.sh:2 — set -euo pipefail
  • benchmark/install/install_pgboss.sh:3 — set -euo pipefail
  • benchmark/runners/clean_reinstall.sh:3 — set -uo pipefail (missing -Ee)
  • benchmark/tooling/microbench.sh:3 — set -uo pipefail (missing -Ee)

• [MED] bug — broken in-doc link. benchmark/OPS_GOTCHAS.md:185 references [install/install_pgque.sh](install/install_pgque.sh) which does not exist in the repo. Per benchmark/install/README.md:9, pgque install is driven from AMI userdata, so this script was never created. Suggested fix: rephrase to point at the steps quoted in install/README.md, or change to "re-clone the PR Rotate subscription and tick tables to avoid held-xmin bloat (#61) #62 branch and make install per install/README.md."

• [LOW] guidelines — non-binary unit in shell comment. benchmark/install/install_pgboss.sh:2 — 7.9 GB disk sort pathology should be 7.9 GiB per CLAUDE.md (binary units in prose; PG config is the only exception, and this is a comment, not config).

• [LOW] reproducibility — hardcoded external paths. benchmark/runners/run_r7.sh resolves auxiliary tools from $R7_DIR=/tmp/r7 and $R6_DIR=/tmp/r6 (lines 11–12, 48, 52, 107). Those scripts (sys_metrics_sampler.py, pg_stat_statements_snapshot.py, parse_events_consumed.py) actually live in benchmark/tooling/ in the repo. A fresh clone needs the user to copy/symlink them into /tmp/r6 and /tmp/r7 before run_r7.sh will succeed. Add a note in benchmark/README.md's quick-start, or change the script to fall back on tooling/.

Potential

• [INFO] install_pgboss.sh:6 uses curl … | sudo bash (nodesource installer). This is the standard nodesource pattern and not actionable, just flagging that the PR template's security checklist asked specifically about curl … | sh. No fix needed.
• [INFO] clean_reinstall.sh:32, :38 interpolate $j (a jobname selected from cron.job) directly into a SQL string without quoting. Inside this private bench harness with a controlled DB it's fine, but if anyone copy-pastes the snippet for production cleanup they should escape the value. Out-of-scope for this PR.

Summary

Area	Findings	Potential	Filtered
Anti-leak	2 BLOCKING	0	0
Security	0	1	0
Bugs	1	1	0
Tests	0	0	0
Guidelines	2	0	0
Docs	0	0	0

---

REV-style review (security, bugs, tests, guidelines, docs). SOC2 items skipped per project policy. Anti-leak independently re-verified.

[NikolayS]

REV Review — PR #66 (round 2, post-loop)

CI: 7/8 SUCCESS (test 14/15/16/17/18, client-smoke, verify); claude-review IN_PROGRESS at scan time.
R1 BLOCKING resolution status:

• [HIGH] anti-leak — bootstrap.sh:41,44 (issue #77) residue → addressed (commit d586e35 replaces with neutral wording; current file has no issue #77 hits)
• [HIGH] anti-leak — ee105e8 subject (issue #77) + body GitLab #77 note 3263767264 + sibling commit 916a4f4 body GitLab mentions → addressed (history rewritten: be9c9d0 → ee105e8c, 042833c → 916a4f4; no gitlab / note <id> / issue #77 in git log origin/main..origin/docs/bench-methodology)

Independent anti-leak re-scan: CLEAN

Pattern set (excluding the documented vendored-data exception): issue #?7[67], gitlab\.com, note ?#?[0-9]{8,}, sahmed, artifact[_-]?registry, @AR\b, wi[ -]?#?7[67], hetzner.

Surface	Hits
Commit history (git log origin/main..origin/docs/bench-methodology --format='%H %s%n%b')	0
PR diff (gh pr diff 66)	0
Working tree (benchmark/, docs/, README.md)	0
PR body	0
PR comments (excluding R1 review which itself quotes the prior leaks for record)	0

R[4-9]\b matches in the working tree (9 lines) are all legitimate filenames / vendored data per the documented exception: runners/run_r7.sh, charts/r5_analyze.py, /tmp/bench_r5 path. Flagging for awareness only — not blocking.

Blocking

None.

Non-blocking

1. [LOW] consistency — runner/file naming runners/run_r7.sh, charts/r5_analyze.py, charts/r6_smoke_chart.py, gifs/r4_*.py keep round-numbered filenames. These are now de-coupled from the round refs that were scrubbed from contents (good — anti-leak passed). Optional follow-up: rename to neutral names (run_bench.sh, bloat_analyze.py) so the intent ("these are the bench tools, not a snapshot of round N") is obvious. Non-blocking; current state is internally consistent.
2. [LOW] verify CI for claude-review claude-review was IN_PROGRESS during this scan; user should confirm it lands SUCCESS before merge.

Potential issues

None new in r2.

Summary table

Severity	Count
BLOCKING	0
Non-blocking	2
Potential	0

Anti-leak independent verification

• git log body+subject scan: clean
• gh pr diff 66 scan: clean
• working-tree scan (benchmark/, docs/, README.md): clean
• PR body + comments scan: clean (R1's quoted strings are in R1's review, expected)
• Shell-style: all 7 scripts now have set -Eeuo pipefail; grep -rL "set -Eeuo pipefail" benchmark/ --include="*.sh" returns empty.
• Broken-link fix: grep -rn "install_pgque.sh" benchmark/ returns empty; OPS_GOTCHAS.md:185 now points to install/README.md and benchmark/install/README.md exists.

Verdict

READY FOR USER REVIEW

Both R1 BLOCKING anti-leak findings are resolved at history + working-tree + PR-metadata level. The two BLOCKING items from R1 are gone, the four non-blocking items are addressed (shell-style, broken link, binary units, R-round scrub), and r2's independent re-scan is clean. Pending CI: claude-review should be allowed to complete green before merge.

---

REV-style review (security, bugs, tests, guidelines, docs). SOC2 items skipped per project policy. Anti-leak independently re-verified on commit history, diff, working tree, PR body, and PR comments.