WIP debug why timing=true is crashing

[wconstab]

Stack from ghstack (oldest at bottom):

• [C10D] Fix coalescedCollective op Flight Recording #120430
• wip add test for individual p2p and start fixing input/output sizes for non coalescedo ps #120410
• -> WIP debug why timing=true is crashing #120350
• [C10D] Fix pointToPoint op Flight Recording #120270
• [C10D] Add test case for crashing isend_irecv dump_entries combo #119757

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @yf225

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/120350

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 6 Unrelated Failures

As of commit a38f196 with merge base cfddfce ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner-noclang / linux-job (gh)
  >>> Lint for test/distributed/test_c10d_nccl.py:
• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) (gh)
  distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced
• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) (gh)
  distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl
• pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) (gh)
  distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager
• pull / linux-jammy-cuda11.8-cudnn8-py3.8-clang12 / build (gh)
  /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2722:10: error: unused variable 'trace_id' [-Werror,-Wunused-variable]
• pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, linux.4xlarge) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated
• pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated
• pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: can't create new thread at interpreter shutdown
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  dynamo/test_trace_rules.py::TraceRuleTests::test_torch_name_rule_map_updated

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[wconstab]

squashed