Adding options to enable some determinism related configs

[githubsgi]

Wondering if it would be ok to add enable options for the following determinism related knobs ?

1. preserve_rng_state=True in ptd_checkpoint_wrapper()
2. torch.use_deterministic_algorithms(True, warn_only=True)

[wwwjn]

For 2, we already supported in titan with config --training.deterministic:

torchtitan/torchtitan/config/job_config.py

Line 239 in 476a965

deterministic: bool = False

[githubsgi]

@wwwjn , thanks for pointing that out. So we need an option for preserve_rng_state=True, right ?

This is where looks like the torch.use_deterministic_algorithms(True) is invoked. It is missing the warn_only=True arguments though.

[githubsgi]

Would a PR be entertained for this ?

[fegin]

If we set warn_only=True by default, people may assume everything works correctly. People sometimes don't look into the warning carefully. While I don't want another option, but if we want to add warn_only option, I prefer making it configurable and default is False.