File-backed Turso can crash under concurrent single-event appends that bypass the write gate

[nikstern]

Problem

A modest batch of concurrent entity transitions can crash the Temper process when using file-backed Turso/libSQL.

Before the native crash, libSQL returned:

SQLite failure: `bad parameter or other API misuse`

The process subsequently aborted with:

malloc: *** error for object <address>: pointer being freed was not allocated
malloc: *** set a breakpoint in malloc_error_break to debug

This appears related to concurrent single-event appends bypassing the Turso write gate. The exact native failure has not yet been reduced to a standalone libSQL reproducer.

Impact

• A valid, bounded Temper workload can terminate the server process.
• The failure interrupts active entity transitions and reaction chains.
• A native allocator failure presents potential memory-safety and data-integrity concerns beyond an ordinary transient storage error.

Environment

• Temper commit: a28fdb2ed8b0e1e0ea15a99c3b4a1dfba20e160e
• Storage: local file-backed Turso
• libsql: 0.9.29
• Platform: macOS arm64

What we know

The failure occurred under a bounded workflow that:

1. Read several independent entities concurrently.
2. Created independent entities and files concurrently.
3. Dispatched single-event transitions to those entities concurrently.
4. Caused ordinary follow-up reactions and projection writes.

The WASM host implements http_call_batch using join_all, so requests within a batch execute concurrently.

No shared entity was intentionally mutated by multiple requests in the same batch.

A transition first returned a managed persistence error:

persistence failed: storage error:
SQLite failure: `bad parameter or other API misuse`

Continued or retried processing then terminated the native process with an invalid-free allocator error. The crash was reproduced in separate server processes and at different points in ordinary entity and reaction persistence. It is timing-dependent.

Expected behavior

Concurrent bounded actions should complete successfully, queue behind a bounded write gate, or return a recoverable storage or admission error. They must not terminate the native process.

Suspected mechanism — unverified

Most Temper actions append exactly one event. The Turso event store currently uses:

let bypass_write_gate = events.len() == 1;

Single-event appends therefore do not acquire the process-local write semaphore.

At the same time, the WASM host's http_call_batch executes requests concurrently. A bounded batch of independent transitions can consequently open many concurrent local libSQL write paths despite the configured Turso write concurrency.

This behavior was introduced intentionally by:

• [codex] Fast path single-event Turso appends #186 — Fast path single-event Turso appends
• Let atomic appends bypass Turso write gate #187 — Let atomic appends bypass Turso write gate

Those changes addressed remote Turso latency and write-lane pressure. The current failure may indicate that bypassing the gate is unsafe for file-backed local libSQL under concurrent writes.

This remains a hypothesis until confirmed with a native backtrace or minimal store-level reproducer.

Investigation plan

1. Add a kernel-level stress test that creates a file-backed TursoEventStore and issues concurrent single-event appends to distinct persistence IDs.
2. Repeat while ordinary projection or reaction writes are active.
3. Run the same workload with single-event appends acquiring the write gate.
4. Progressively increase permitted local write concurrency to identify the safe boundary.
5. Run the reproducer under ASan or LLDB to capture the invalid-free frame.
6. Compare local file-backed behavior with remote Turso.
7. Check newer libSQL releases for relevant concurrency fixes.

Setting TEMPER_TURSO_WRITE_CONCURRENCY=1 may not affect the current single-event path because that path appears to bypass the semaphore.

Acceptance criteria

• A regression test covers concurrent single-event appends against file-backed Turso.
• The workload completes or returns a bounded recoverable error without native process termination.
• Local and remote write-gate behavior is explicit and independently tested.
• The regression test is clean under ASan or an equivalent native memory diagnostic.

Related

• [codex] Fast path single-event Turso appends #186 — Fast path single-event Turso appends
• Let atomic appends bypass Turso write gate #187 — Let atomic appends bypass Turso write gate
• ADR-0068 — Turso write-gate retrospective