Add thread-local random number generator for thread safety

[lenzo-ka]

Make random number generator thread-safe using thread-local storage

Summary

This PR makes the Mersenne Twister random number generator (RNG) in PocketSphinx thread-safe by using thread-local storage (TLS) for its state variables. This eliminates race conditions and data corruption when multiple threads use the RNG concurrently.

Problem

The RNG implementation used global static variables (mt[] array and mti index) that were shared across all threads, causing:

• Race conditions in multi-threaded applications
• Non-deterministic behavior
• Potential data corruption
• Thread safety issues in speech recognition pipelines

Solution

• Add platform-agnostic thread-local storage abstraction (PS_THREAD_LOCAL macro)
• Make RNG state variables thread-local, giving each thread its own independent RNG state
• Maintain complete backward compatibility - no API changes required

Key Features

✅ Fully backward compatible - No code changes required for users
✅ Thread-safe by default - Each thread gets independent RNG state
✅ Cross-platform support - Works with C11, C++11, GCC, Clang, MSVC
✅ Opt-out available - Can be disabled with -DPS_THREAD_LOCAL_RNG=OFF
✅ Comprehensive testing - Includes thread safety verification tests

Implementation Details

The implementation adds PS_THREAD_LOCAL before the static RNG state:

PS_THREAD_LOCAL static unsigned long mt[N];  /* the array for the state vector */
PS_THREAD_LOCAL static int mti = N + 1;      /* mti==N+1 means mt[N] is not initialized */

When PS_USE_THREAD_LOCAL_RNG is defined (default), the macro expands to the appropriate thread-local keyword for the compiler. When disabled, it expands to nothing, reverting to the original behavior.

Testing

New tests verify:

• Thread-local storage compilation and functionality
• RNG thread safety (no collisions between threads)
• Preservation of deterministic behavior within each thread
• Cross-platform thread utilities work correctly

Run thread safety tests:

./build/test_genrand_thread_tls

Build Configuration

The feature is enabled by default but can be disabled:

cmake -DPS_THREAD_LOCAL_RNG=OFF ..

Commits

1. feat: Add thread-local storage abstraction header - Platform-agnostic TLS support
2. feat: Make random number generator thread-safe using TLS - Core RNG changes
3. test: Add cross-platform thread utilities for testing - Testing infrastructure
4. test: Add comprehensive tests for thread-local RNG - Thread safety verification
5. build: Add CMake support for thread-local RNG feature - Build system integration
6. docs: Add thread safety documentation - User documentation

Impact

• Single-threaded applications: No change in behavior
• Multi-threaded applications: Now thread-safe (fixes existing race conditions)
• Performance: Minimal overhead from TLS access
• Memory: Each thread allocates ~2.5KB for RNG state (624 * 4 bytes)

This is a bug fix that makes the code thread-safe without breaking any properly-functioning existing code.

Files Changed

• src/util/thread_local.h - New TLS abstraction header
• src/util/genrand.c - Added TLS to RNG state variables
• src/util/genrand.h - Added thread safety documentation
• CMakeLists.txt - Added PS_THREAD_LOCAL_RNG option
• config.h.in - Added PS_USE_THREAD_LOCAL_RNG configuration
• test/unit/ - Added comprehensive thread safety tests
• docs/thread_safety.md - New documentation for thread safety

---

[dhdaines]

Good idea. Of course, Arthur's comment implies that we are already 11 years overdue for replacing the RNG with another algorithm!

Do you know if this works on musl libc based systems? I think we can use docker to test this in the CI...