storage/mysql: enable primary-key snapshot parallelism by default

[peterdukelarsen]

Motivation

Part of SS-438

Flips the mysql_source_snapshot_parallelism dyncfg default from false to true, so MySQL source snapshots of tables with a supported single-column primary key are split into per-worker PK ranges out of the box. Tables without a suitable PK continue to fall back to a single-worker whole-table read.

CI has been exercising the enabled path via the mzcompose system parameter override (which also randomizes the flag so the fallback path stays covered). This brings the production default in line with what is tested. The flag remains available to disable splitting for a replica if needed.

Tips for reviewer

One-line default change. The existing test/mysql-cdc/mysql-cdc.td coverage explicitly sets the flag to false for the fallback case and relies on the enabled path elsewhere, so no test changes are needed.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- PK-sampling probe connections skip the wait_timeout guard that every other snapshot connection gets

src/storage/src/source/mysql/snapshot.rs:365

The pooled connections sample_pk_bounds opens are the only snapshot-phase connections that never run SET @@session.wait_timeout, and this default flip is what makes them live long enough for that to matter. On an upstream with a low global wait_timeout, a probe connection that finishes its table early and sits in the pool while a slower table's prefix walk runs can be reaped by the server; the next use of it fails and aborts the whole snapshot as a transient error, which then repeats on every retry.

Details

set_wait_timeout (snapshot.rs:1163) exists precisely for this: "so a lowered global value cannot reap the connection while it sits idle during snapshot setup", backed by DEFAULT_SNAPSHOT_WAIT_TIMEOUT = 48h. It is applied to the leader's lock connection (snapshot.rs:471) and to each worker's snapshot connection (snapshot.rs:832). The probe pool at snapshot.rs:362-378 sets only max_execution_time.

Why the flip matters: with the flag off, a probe connection's work is SELECT table_rows plus (for tables under mysql_source_snapshot_exact_count_max_rows) a COUNT(*), which takes seconds. With it on, an eligible table additionally runs partition_table, which probe.rs itself budgets at up to 5000 prefix probes of a few queries each, and compute_sampled_splits documents as able to "take minutes". buffer_unordered(worker_count) reuses connections immediately while tables remain, so the exposure is the tail: once fewer tables remain than the pool holds, the surplus connections idle for the full duration of the slowest walk. They are then only touched by conn.disconnect().await? at snapshot.rs:443, whose write_command_data + stream.close() on a peer-closed socket propagates an IO error through ? as a TransientError.

Fix: carry snapshot_wait_timeout into the pooled connection setup, next to the existing max_execution_time read:

 let max_execution_time = config
     .config
     .parameters
     .mysql_source_timeouts
     .snapshot_max_execution_time;
+let wait_timeout = config
+    .config
+    .parameters
+    .mysql_source_timeouts
+    .snapshot_wait_timeout;

 None => {
     let mut conn = connection_config
         .connect(task_name, ssh_tunnel_manager)
         .await?;
+    if let Some(timeout) = wait_timeout {
+        set_wait_timeout(&mut *conn, timeout).await?;
+    }
     if let Some(timeout) = max_execution_time {

[peterdukelarsen]

I don't think the wait_timeout edge case here is very severe. Probes don't have that long of a long runtime -- the wait_timeout was set up when we were expecting to spend 8 hours finding boundary samples.