SIGSEGV race in js_native_call_method via @perryts/mysql nativeScramble during concurrent Pool.acquire handshakes (0.5.1029 / c720d2a4)

[proggeramlug]

Summary

Follow-on from #1767 (fixed — thanks; sequential parameterized queries work now). With that fixed, sequential POST ingest is stable, but under concurrent load the @perryts/mysql pool opens additional connections (Pool.acquire), and the crash moves to the auth handshake: a SIGSEGV in js_native_call_method during @perryts/mysql's nativeScramble (mysql_native_password).

It's a race / heisenbug: it's masked by gdb and by low-load --debug-symbols builds, and only surfaced under heavy concurrency (~ab -c 100). I captured it via a core dump of a --debug-symbols build, so the backtrace is symbolized.

Environment

• perry 0.5.1029 @ c720d2a4, Linux x86-64 (Ubuntu 24.04)
• @perryts/mysql 0.1.4, MariaDB 10.11 (mysql_native_password)
• Sequential ingest (ab -c 1, single pooled connection) is stable. The crash needs concurrency high enough to make the pool open new connections (so the handshake/auth runs repeatedly/concurrently).
• NOTE: this is the mysql_native_password path. I have not yet tested caching_sha2_password (will follow up).

Symbolized backtrace (from core dump)

Thread 1 SIGSEGV
#0  js_native_call_method ()
#1  @perryts/mysql auth/native_password.ts: nativeScramble ()
#2  @perryts/mysql auth/native_password.ts: __obj_method_initialResponse_2 ()
#3  js_native_call_method ()
#4  @perryts/mysql connection.ts: sendAuthResponseForPlugin ()
#5  @perryts/mysql connection.ts: handleHandshakeV10 ()
#6  @perryts/mysql connection.ts: handlePacket ()
#7  perry_closure connection.ts
#8  js_net_process_pending ()
#9  js_stdlib_process_pending ()
#10 js_run_stdlib_pump ()
#11 @perryts/mysql pool.ts: Pool.acquire ()
#12 @perryts/mysql pool.ts: Pool.query ()
#13 chirp db.ts: query ()
#14 js_async_first_call ()
...
#19 perry_stdlib::fastify::server::call_closure2_catching ()
#20 perry_stdlib::fastify::server::js_fastify_process_pending ()

Faults in js_native_call_method while nativeScramble builds the auth response during a new connection's handleHandshakeV10, reached from Pool.acquire under concurrency.

Repro

Compile a Fastify + @perryts/mysql service whose POST handler runs parameterized queries (so the pool must grow under load), point it at a mysql_native_password server, and hit it concurrently:

perry compile --debug-symbols src/main.ts -o app    # (also crashes in default/stripped builds)
ulimit -c unlimited; PORT=3199 ... ./app &
ab -n 8000 -c 100 -p body.json -T application/json -H 'X-Chirp-Key: <key>' http://127.0.0.1:3199/api/v1/event
# -> SIGSEGV in js_native_call_method (nativeScramble); core dumped

Sequential (-c 1) is fine. Because it's a race, instrumented/low-opt builds hide it — a core dump from an optimized run is the reliable way to see it.

I can provide the full failing source, the --debug-symbols binary, or the core. Cross-ref: #1663, #1767.

[proggeramlug]

Follow-up on the caching_sha2_password question (the OP was tested against mysql_native_password).

I set up a MySQL 8 with caching_sha2_password (the auth our production uses) and re-ran the concurrent ingest test. The concurrency problem is not mysql_native_password-specific — the optimized build misbehaves on caching_sha2 too:

• Optimized / stripped build (--no-auto-optimize, no debug symbols): under concurrency, ingest hangs / stops inserting (0 rows inserted at -c 10, process alive but stuck) — i.e. the connection pool wedges rather than SIGSEGV'ing as it does on native_password.
• --debug-symbols build: survives (processed -c 100/150/200, pool-rate-limited at max:10 but no crash/hang).

So it reproduces only in optimized builds (consistent heisenbug/race) — and unfortunately that's what a production deploy runs. I could not get a clean symbolized trace for the caching_sha2 variant because the --debug-symbols build masks it; the symbolized nativeScramble backtrace above (native_password) remains the concrete artifact. Net: the underlying concurrency bug in connection-establishment / the pool appears auth-plugin-agnostic and affects the caching_sha2 path as well.

Both auth paths share: new pooled connections being opened concurrently under load (Pool.acquire → handshake → auth). Happy to provide the failing binary, or run a specific instrumented build if that helps catch the optimized-only race.