distributed/debug: add an HTTP server for debugging running jobs

[d4l3k]

This adds a debug HTTP server for debugging stuck or slow jobs. It runs the WorkerServer on every worker and then launches a separate flask process on rank 0 to have users connect to for debugging.

This can easily be improved to trigger profilers as well as visualize the data much better.

Initial handlers:

• pytorch profiler
• FlightRecorder data
• Python stacks

os.environ["TORCH_NCCL_TRACE_BUFFER_SIZE"] = "2000"

from torch.distributed.debug import enable_debug_server

enable_debug_server()

Test plan:

torchrun --nnodes 1 --nproc_per_node=gpu ~/scripts/debug_test.py

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @pragupta @msaroufim @dcci

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/167395

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures

As of commit d9bd0f2 with merge base bfddfde ():

NEW FAILURES - The following jobs have failed:

• docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-executorch) (gh)
  Final attempt failed. Child_process exited with error code 1
• docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-clang21, linux.arm64.m7g.4xlarge) (gh)
  Final attempt failed. Child_process exited with error code 1
• docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-gcc13-inductor-benchmarks, linux.arm64.m7g.4xlar... (gh)
  Final attempt failed. Child_process exited with error code 1
• docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-gcc13, linux.arm64.m7g.4xlarge) (gh)
  Final attempt failed. Child_process exited with error code 1
• pull / linux-jammy-py3.10-clang12-onnx / test (default, 1, 2, lf.linux.c7i.2xlarge) (gh)
  Process completed with exit code 2.
• pull / linux-jammy-py3.10-clang12-onnx / test (default, 2, 2, lf.linux.c7i.2xlarge) (gh)
  Process completed with exit code 2.
• trunk / linux-jammy-py3-clang12-executorch / build (gh)
  Final attempt failed. Child_process exited with error code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[fduwjj]

Overall lgtm.

[d4l3k]

@atalman these seem like broken infr and unrelated to this PR, good to land?

[d4l3k]

From HUD those issues are broken on trunk

[d4l3k]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

Merge failed

Reason: 2 mandatory check(s) failed. The first few are:

• pull / linux-jammy-py3.10-clang12-onnx / test (default, 1, 2, linux.c7i.2xlarge)
• pull / linux-jammy-py3.10-clang12-onnx / test (default, 2, 2, linux.c7i.2xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

[d4l3k]

@pytorchbot merge -i

[pytorchmergebot]

Merge started

Your change will be merged while ignoring the following 7 checks: Build manywheel docker images / manylinuxaarch64-builder:cuda13.0, docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-gcc13, linux.arm64.m7g.4xlarge), docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-executorch), docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-gcc13-inductor-benchmarks, linux.arm64.m7g.4xlar..., docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-clang21, linux.arm64.m7g.4xlarge), pull / linux-jammy-py3.10-clang12-onnx / test (default, 1, 2, linux.c7i.2xlarge), pull / linux-jammy-py3.10-clang12-onnx / test (default, 2, 2, linux.c7i.2xlarge)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-py3-clang12-executorch / build

Details for Dev Infra team

Raised by workflow job

[d4l3k]

@pytorchbot merge -f "infra failures"

[pytorchmergebot]

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorch-auto-revert]

@pytorchbot revert -m "Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable" -c autorevert

This PR is attributed to have caused regression in:

• pull: linux-jammy-py3.10-clang12-onnx / test (hud)

Please investigate and fix the issues.

[pytorchmergebot]

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

[pytorchmergebot]

@d4l3k your PR has been successfully reverted.