debug: TCP bailing of new Torch [wip] #20392

Borda · 2024-11-04T16:27:51Z

What does this PR do?

the case:

FAILED strategies\launchers\test_multiprocessing.py::test_fit_twice_raises - torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\torch\multiprocessing\spawn.py", line 68, in _wrap
    fn(i, *args)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\pytorch\strategies\launchers\multiprocessing.py", line 173, in _wrapping_function
    results = function(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 937, in _run
    self.strategy.setup_environment()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\pytorch\strategies\ddp.py", line 154, in setup_environment
    self.setup_distributed()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\pytorch\strategies\ddp.py", line 203, in setup_distributed
    _init_dist_connection(self.cluster_environment, self._process_group_backend, timeout=self._timeout)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\lightning\fabric\utilities\distributed.py", line 297, in _init_dist_connection
    torch.distributed.init_process_group(torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\torch\distributed\c10d_logger.py", line 86, in wrapper
    func_return = func(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\torch\distributed\distributed_c10d.py", line 1177, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\torch\distributed\rendezvous.py", line 246, in _env_rendezvous_handler
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\torch\distributed\rendezvous.py", line 174, in _create_c10d_store
    return TCPStore(
           ^^^^^^^^^
torch.distributed.DistNetworkError: Unknown error

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--20392.org.readthedocs.build/en/20392/

github-actions · 2024-11-04T16:28:16Z

⛈️ Required checks status: Has failure 🔴

Warning
This job will need to be re-run to merge your PR. If you do not have write access to the repository, you can ask Lightning-AI/lai-frameworks to re-run it. If you push a new commit, all of CI will re-trigger.

Groups summary

🔴 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-13, lightning, 3.9, 2.1, oldest)	failure	❌
pl-cpu (macOS-14, lightning, 3.10, 2.1)	failure	❌
pl-cpu (macOS-14, lightning, 3.11, 2.2)	failure	❌
pl-cpu (macOS-14, lightning, 3.11, 2.3)	failure	❌
pl-cpu (macOS-14, lightning, 3.12, 2.4)	failure	❌
pl-cpu (ubuntu-20.04, lightning, 3.9, 2.1, oldest)	failure	❌
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	failure	❌
pl-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	failure	❌
pl-cpu (ubuntu-20.04, lightning, 3.11, 2.3)	failure	❌
pl-cpu (ubuntu-20.04, lightning, 3.12, 2.4)	failure	❌
pl-cpu (windows-2022, lightning, 3.9, 2.1, oldest)	failure	❌
pl-cpu (windows-2022, lightning, 3.10, 2.1)	failure	❌
pl-cpu (windows-2022, lightning, 3.11, 2.2)	failure	❌
pl-cpu (windows-2022, lightning, 3.11, 2.3)	failure	❌
pl-cpu (windows-2022, lightning, 3.12, 2.4)	failure	❌
pl-cpu (macOS-14, pytorch, 3.9, 2.1)	failure	❌
pl-cpu (ubuntu-20.04, pytorch, 3.9, 2.1)	failure	❌
pl-cpu (windows-2022, pytorch, 3.9, 2.1)	failure	❌
pl-cpu (macOS-13, pytorch, 3.10, 2.1)	failure	❌
pl-cpu (ubuntu-22.04, pytorch, 3.10, 2.1)	failure	❌
pl-cpu (windows-2022, pytorch, 3.10, 2.1)	failure	❌

These checks are required after the changes to tests/tests_pytorch/callbacks/test_prediction_writer.py, tests/tests_pytorch/helpers/advanced_models.py, tests/tests_pytorch/trainer/connectors/test_data_connector.py, tests/tests_pytorch/trainer/test_dataloaders.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs) (testing Lightning \| latest)	success	✅
pytorch-lightning (GPUs) (testing PyTorch \| latest)	success	✅

These checks are required after the changes to tests/tests_pytorch/callbacks/test_prediction_writer.py, tests/tests_pytorch/helpers/advanced_models.py, tests/tests_pytorch/trainer/connectors/test_data_connector.py, tests/tests_pytorch/trainer/test_dataloaders.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to tests/parity_fabric/models.py, tests/parity_pytorch/models.py.

🔴 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-13, lightning, 3.9, 2.1, oldest)	success	✅
fabric-cpu (macOS-14, lightning, 3.10, 2.1)	success	✅
fabric-cpu (macOS-14, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-14, lightning, 3.11, 2.3)	success	✅
fabric-cpu (macOS-14, lightning, 3.12, 2.4)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 2.1, oldest)	failure	❌
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.3)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.12, 2.4)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 2.1, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	failure	❌
fabric-cpu (windows-2022, lightning, 3.11, 2.3)	success	✅
fabric-cpu (windows-2022, lightning, 3.12, 2.4)	success	✅
fabric-cpu (macOS-14, fabric, 3.9, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.9, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.9, 2.1)	success	✅
fabric-cpu (macOS-13, fabric, 3.10, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.10, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.10, 2.1)	success	✅

These checks are required after the changes to tests/tests_fabric/utilities/test_data.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs) (testing Fabric \| latest)	success	✅
lightning-fabric (GPUs) (testing Lightning \| latest)	success	✅

These checks are required after the changes to tests/tests_fabric/utilities/test_data.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

Borda · 2024-11-04T18:00:12Z

related cases:

codecov · 2024-11-04T18:19:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76%. Comparing base (3627c5b) to head (24fb2c5).
Report is 69 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (3627c5b) and HEAD (24fb2c5). Click for more details.

HEAD has 579 uploads less than BASE

Flag BASE (3627c5b) HEAD (24fb2c5)

cpu 147 19

lightning 104 15

pytest 84 23

python3.9 45 5

lightning_fabric 25 7

python3.11 41 5

python3.10 45 6

python3.12 16 3

pytorch2.1 43 0

pytest-full 67 0

pytorch_lightning 22 1

pytorch2.2 8 0

pytorch2.3 12 0

pytorch2.4 4 0

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #20392     +/-   ##
=========================================
- Coverage      89%      76%    -14%     
=========================================
  Files         267      267             
  Lines       23065    23065             
=========================================
- Hits        20574    17456   -3118     
- Misses       2491     5609   +3118

Borda requested review from lantiga, tchaton and justusschock as code owners November 4, 2024 16:27

github-actions bot added the pl Generic label for PyTorch Lightning package label Nov 4, 2024

Borda mentioned this pull request Nov 4, 2024

Releasing update [rebase & merge] #20381

Merged

7 tasks

try: persistent_workers=True

fed9783

Borda force-pushed the debug/tcp branch from 3e68f66 to fed9783 Compare November 4, 2024 17:14

github-actions bot added the fabric lightning.fabric.Fabric label Nov 4, 2024

pre-commit-ci bot and others added 4 commits November 4, 2024 17:14

[pre-commit.ci] auto fixes from pre-commit.com hooks

d9a620a

for more information, see https://pre-commit.ci

update

8bbe907

[pre-commit.ci] auto fixes from pre-commit.com hooks

61cfca7

for more information, see https://pre-commit.ci

Merge branch 'master' into debug/tcp

24fb2c5

Borda closed this Nov 11, 2024

Borda deleted the debug/tcp branch June 20, 2025 12:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

debug: TCP bailing of new Torch [wip] #20392

debug: TCP bailing of new Torch [wip] #20392

Uh oh!

Borda commented Nov 4, 2024 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Nov 4, 2024 •

edited

Loading

Uh oh!

Borda commented Nov 4, 2024

Uh oh!

codecov bot commented Nov 4, 2024 •

edited

Loading

Uh oh!

Uh oh!

debug: TCP bailing of new Torch [wip] #20392

debug: TCP bailing of new Torch [wip] #20392

Uh oh!

Conversation

Borda commented Nov 4, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

PR review

Uh oh!

github-actions bot commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⛈️ Required checks status: Has failure 🔴

Groups summary

Uh oh!

Borda commented Nov 4, 2024

Uh oh!

codecov bot commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Borda commented Nov 4, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 4, 2024 •

edited

Loading

codecov bot commented Nov 4, 2024 •

edited

Loading