add cmd/binlog-visibility-repro: stand-alone MySQL contract repro

[morgo]

Summary

Adds cmd/binlog-visibility-repro, a stand-alone reproducer for what looks like an upstream MySQL contract violation:

with binlog_order_commits=ON, every row event the binlog streamer observes for a committed transaction must already be visible to a fresh autocommit SELECT on a separate connection.

The tool is a single main.go that uses only database/sql + go-mysql-org/go-mysql/replication. No spirit components. That's the point — when this fails, it can't be attributed to anything in pkg/repl, the delta map, the chunker, the applier, or any other spirit wrapper. The failure attributes to MySQL itself.

This is the smoking gun behind the second bug in #746: spirit's deltaMap.Flush periodically sees affected_rows < num_keys from REPLACE INTO _new ... SELECT FROM original WHERE id IN (...), meaning a key was in the delta map (so the binlog streamer had already delivered an event for it) but the row wasn't visible to a separate-connection SELECT moments later.

How it works

1. Connects to MySQL and creates a small InnoDB table.
2. Spawns N producer goroutines, each looping BEGIN; INSERT; UPDATE; COMMIT; on the table.
3. Opens a binlog replication connection. For every WriteRowsEvent on the test table, immediately runs SELECT id FROM <table> WHERE id = ? on a separate connection.
4. On the first miss, sleeps -recheck-delay (default 100 ms), re-runs the SELECT, and classifies the miss as either permanent (still not visible) or delayed (now visible). Both are violations of the contract; "delayed" is the soft form, "permanent" the hard form.
5. Prints a summary and exits — 1 if a miss was seen, 0 if -max-duration elapses cleanly, 2 for an infrastructure problem.

Bisect — what triggers the race

Local Docker MySQL 8.0.45 (no -race), short runs (~16 k commits / 5 s):

Producer shape	Result
autocommit INSERT … VALUES (…)	does NOT reproduce
BEGIN; INSERT; COMMIT; (explicit txn, single stmt)	does NOT reproduce
BEGIN; INSERT; UPDATE; COMMIT;	reproduces (probabilistic)

The minimum producer shape that surfaces the race is a multi-statement transaction that INSERTs and then UPDATEs the same row before COMMIT. Single-statement transactions, with or without explicit BEGIN/COMMIT, do not reproduce in the same time budget.

How to run it

A repro service has been added to compose/compose.yml and compose/replication-tls/replication-ci.yml. It builds the binary and runs it against the local mysql service. The exact same docker compose commands the existing matrix uses, just targeting mysql repro instead of mysql test:

# MySQL 8.0.28 (Aurora)
cd compose && docker compose -f compose.yml -f 8.0.28.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.0.43 (Oracle Linux 9 base — same userspace as the default tag)
cd compose && docker compose -f compose.yml -f 8.0.43.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.0.43 (Debian base — same MySQL version, different container userspace,
# linux/amd64 only so on Apple Silicon Docker emulates via Rosetta)
cd compose && docker compose -f compose.yml -f 8.0.43-debian.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.0.45 (default)
cd compose && docker compose up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.0.45 RBR-minimal
cd compose && docker compose -f compose.yml -f 8.0.45-rbr-minimal.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.0.42 (Aurora LTS)
cd compose/replication-tls && docker compose -f replication-ci.yml -f 8.0.42.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 8.4 GA
cd compose/replication-tls && docker compose -f replication-ci.yml -f 8.4.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

# MySQL 9.7
cd compose/replication-tls && docker compose -f replication-ci.yml -f 9.7.yml up --build mysql repro --abort-on-container-exit --exit-code-from repro

--exit-code-from repro propagates the binary's exit code to the shell, so a failure shows up as exit 1.

For a faster iteration loop (shorter max-duration, more producers, etc.), run the binary directly:

cd compose && docker compose run --rm --build --entrypoint "" repro \
  bash -c "go build -o /tmp/repro ./cmd/binlog-visibility-repro && \
           /tmp/repro -dsn 'tsandbox:msandbox@tcp(mysql:3306)/' \
                      -max-duration 1m -producers 16"

Flags:

flag	default	meaning
-dsn	tsandbox:msandbox@tcp(127.0.0.1:8033)/	MySQL DSN; the program creates and drops its own database
-max-duration	5m	give up if no miss is seen within this long
-recheck-delay	100ms	how long to wait before re-checking a missed row's visibility
-producers	8	concurrent producer goroutines

Findings so far

• Every MySQL version we test reproduces at least once across short runs: 8.0.28, 8.0.42, 8.0.43, 8.0.45, 8.0.45-rbr-min, 9.7. 8.4 GA hasn't reproduced in our limited samples but we don't have enough data to call it clean.
• Container userspace doesn't matter: mysql:8.0.43 (Oracle Linux 9 base) and mysql:8.0.43-debian (Debian base) both reproduce on the same host, with the same MySQL version. Same kernel, different userspace — same bug.
• Architecture doesn't matter: arm64 native, amd64 native (CI), and amd64 emulated via Rosetta (Apple Silicon running the Debian variant) all reproduce.
• The bug only surfaces on Linux: native macOS MySQL 8.0.43 directly on Apple Silicon (no Docker) does NOT reproduce in short runs, while the same MySQL 8.0.43 inside Docker (which is just a Linux VM on the same host) reproduces in seconds. The most parsimonious explanation is that the race window is much narrower on Darwin's scheduler / pthread / fsync stack than on Linux's — but the bug almost certainly exists on Darwin too, just at a much lower rate.
• Almost every miss is "delayed": the row becomes visible to a fresh SELECT within 100 ms of the binlog event being observed. Only one permanent miss has been seen so far (MySQL 8.0.28).
• The race is sensitive to load: it surfaces in ~1 in 2 k–150 k commits depending on the host. Docker on small CI runners reproduces faster than a beefy local machine.

Why this isn't in CI

The matrix workflow that ran this on every push is intentionally not included. The test is designed to fail under the bug we're hunting; gating PRs on that would be silly. The repro lives as a stand-alone artefact: anyone can run it on demand against any MySQL with the commands above.

Related

• Issue flaky test: TestCutoverAtomicityWithConcurrentWrites #746 — second-bug investigation
• PR check: require binlog_order_commits=ON #819 — adds the binlog_order_commits=ON preflight check (this PR's hypothesis depends on that being honoured)

🤖 Generated with Claude Code