Document determinism guidelines

[mcches]

Turmoil is built on the concept of deterministic execution. Using structures such as HashMap initialize with non-deterministic RandomState. Both the internals of turmoil and applications using it need to buy in.

e.g. HashMap, HashSet, tokio::select!, etc.

Document the guidelines.

[mcches]

Related: #16

[dtwitty]

Here are some I ran into in my own testing:

• Make sure your servers are each using their own RNG seed. Otherwise, they will behave identically and often fail to make progress, elect a leader, etc.
• Be wary of libraries that might include randomness. For example, many client libraries contain "exponential backoff with random jitter". You'll need to make sure those RNGs are properly seeded.
• Be wary of hashmaps. Initialize hashmaps with a deterministic hasher, or use ones with deterministic iteration order (indexmap)
  • This one may be hard to accomplish because hashmaps are ubiquitous in library code.

[shikhar]

Our experience jives with @dtwitty points. Ultimately overriding certain randomness & time-related libc fns proved necessary to ensure perfect determinism (shared some thoughts here https://x.com/shikhrr/status/1818476801657598128)

To be specific, we override: clock_gettime, getrandom, and getentropy (just uses getrandom). Simply having those symbols defined in DST mode is sufficient to override libc, as noted by the author of madsim here madsim-rs/madsim#159