[DebugMode] torch.hash_tensor option #167486
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/167486
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (2 Unrelated Failures)As of commit a1e8beb with merge base 2984331 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
ciflow/inductor
oncall: distributed
…debug_hash_tensor
…debug_hash_tensor
…pytorch into pianpwk/debug_hash_tensor
pianpwk
changed the title
| # special case op, we don't need to propagate for local | ||
| # scalar. TODO: figure out a better way to handle this | ||
| if op_schema.op is aten._local_scalar_dense.default: | ||
| if op_schema.op in (aten._local_scalar_dense.default, aten.hash_tensor.default): |
There was a problem hiding this comment.
imo, we just need to explicitly hash_tensor(dtensor.to_local()). _local_scalar_dense is here because it's called a lot internally. but hash_tensor is new and user invoked
…debug_hash_tensor
…debug_hash_tensor
…pytorch into pianpwk/debug_hash_tensor
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Adds
torch.hash_tensor(#154149) as tensor hashing variant; allows tuple of hashes in log annotations for more info (e.g.with DebugMode.log_tensor_hashes(hash_fn=["norm", "hash_tensor"]): ...)also fixes some corner cases around norm hashing (preserves NaNs/infs, avoids erroring on smaller dtypes)
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci