[RFC][inductor][index_put] fallback to aten in torch deterministic mode

[yuguo68]

Fixes #93537
fallback to aten for index_put and scatter ops in torch deterministic mode

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96898

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• CI signal is delayed due to GitHub outage

✅ No Failures

As of commit 580c3a5:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[yuguo68]

working on failed tests

[ngimel]

wires crossed, #96695 is doing same thing pretty much, cc @colesbury

[yuguo68]

wires crossed, #96695 is doing same thing pretty much, cc @colesbury

oh, thanks for sharing! #96695 has dynamo support including guards. This PR only has inductor part for scatter_ops and index_put. I still see test failures and will look into them.

[colesbury]

@yuguo68 Let me know what your plans are. Do you want to combine this with #96695? Or land this first and then I'll land the guards?

[yuguo68]

@yuguo68 Let me know what your plans are. Do you want to combine this with #96695? Or land this first and then I'll land the guards?

Thanks for asking! May I land this and then you land the dynamo part? I think I am close to fix the test errors.

[yuguo68]

@ngimel , it seems the index_put fallback will cause benchmarks/dynamo/huggingface.py failure due to cudagraph and sorting in the deterministic implementation of index_put with accumulate=True. Any suggestions for this? Shall we disable cudagraph or torch.use_deterministic_algorithms for this model? Thanks!

+ python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --output /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv
cuda train AlbertForMaskedLM                   ERROR:common:CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 336, in cudagraphify_impl
    static_outputs = model(list(static_inputs))
  File "/tmp/tmpce9dhnjz/6x/c6xuuwpof6ncjaongbprvu332nxskskhnd7pdmcgpfw4smch44ft.py", line 3373, in call
    aten.index_put_(buf608, [slice_2], buf609, True)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 528, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: CUDA error: operation not permitted when stream is capturing
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

...
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 265, in cudagraphify
    return cudagraphify_impl(model, inputs, static_input_idxs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 335, in cudagraphify_impl
    with torch.cuda.graph(graph, stream=stream):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 173, in __exit__
    self.cuda_graph.capture_end()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/graphs.py", line 79, in capture_end
    super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture

[ngimel]

Yeah, deterministic index_put is a synchronizing operation, so we should not run it with cuda graphs, it's fine to run accuracy jobs that have index_put without cudagraphs.

[yuguo68]

drop the change to scatter/scatter_reduce fallback since it causes numerical issues, 1. we might need a ir.ScatterReduceFallback similar to ir.IndexPutFallback. 2. currently scatter/scatter_reduce do not have a deterministic impl anyway, whereas scatter_add does and we can utilize it.

[yuguo68]

@pytorchbot merge

[pytorchmergebot]

Merge failed

Reason: 1 mandatory check(s) are pending/not yet run. The first few are:

• EasyCLA

Dig deeper by viewing the pending checks on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

[yuguo68]

/easycla

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here