Fix XPU CI UT test_circular_dependencies #158189

guangyey · 2025-07-12T16:39:17Z

Stack from ghstack (oldest at bottom):

-> Fix XPU CI UT test_circular_dependencies #158189

Motivation

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @gujinghui @fengyuan14

pytorch-bot · 2025-07-12T16:39:20Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158189

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 508cfff with merge base 5221448 ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh) (#153987)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 0a00b5a Pull Request resolved: #158189

guangyey · 2025-07-12T16:43:15Z

@Skylion007 Thanks very much~

guangyey · 2025-07-12T16:46:00Z

cc @malfet
This test case, test_circular_dependencies, fails on my local machine with CUDA enabled, but it doesn’t seem to be caught by the CUDA CI jobs. I'm not sure where the gap is.

>>> import torch
>>> import torch._inductor
>>> import torch._inductor.jagged_lowerings
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/guangyey/pytorch/torch/_inductor/jagged_lowerings.py", line 9, in <module>
    from .lowering import fallback_handler, is_integer_type, register_lowering
  File "/data/guangyey/pytorch/torch/_inductor/lowering.py", line 7167, in <module>
    jagged_lowerings.register_jagged_ops()
AttributeError: partially initialized module 'torch._inductor.jagged_lowerings' has no attribute 'register_jagged_ops' (most likely due to a circular import)
>>>

Skylion007 · 2025-07-12T19:08:06Z

cc @malfet This test case, test_circular_dependencies, fails on my local machine with CUDA enabled, but it doesn’t seem to be caught by the CUDA CI jobs. I'm not sure where the gap is.

>>> import torch
>>> import torch._inductor
>>> import torch._inductor.jagged_lowerings
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/guangyey/pytorch/torch/_inductor/jagged_lowerings.py", line 9, in <module>
    from .lowering import fallback_handler, is_integer_type, register_lowering
  File "/data/guangyey/pytorch/torch/_inductor/lowering.py", line 7167, in <module>
    jagged_lowerings.register_jagged_ops()
AttributeError: partially initialized module 'torch._inductor.jagged_lowerings' has no attribute 'register_jagged_ops' (most likely due to a circular import)
>>>

This is a might be bug caused by subdirectories without init.py? I know that has slightly different behaviorn when using setup.py develop vs pip install?

[ghstack-poisoned]

guangyey · 2025-07-13T06:11:48Z

@Skylion007 I am not sure, but it still fails with pytorch-cuda-nightly that I installed via pip

(ygy_2.7) pt-gpu@mlp-milan-02:/data/guangyey/pytorch/test$ python
Python 3.10.18 | packaged by conda-forge | (main, Jun  4 2025, 14:45:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__file__
'/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/__init__.py'
>>> torch.__version__
'2.9.0.dev20250712+cu128'
>>> exit()
(ygy_2.7) pt-gpu@mlp-milan-02:/data/guangyey/pytorch/test$ python test_testing.py TestImports.test_circular_dependencies
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch._functorch.pytree_hacks` is deprecated and will be removed in a future release. Please `use torch.utils._pytree` instead.
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._shard.checkpoint` will be deprecated, use `torch.distributed.checkpoint` instead
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._sharding_spec` will be deprecated, use `torch.distributed._shard.sharding_spec` instead
  return _bootstrap._gcd_import(name[level:], package, level)
E
======================================================================
ERROR: test_circular_dependencies (__main__.TestImports)
Checks that all modules inside torch can be imported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/distributed/_tools/sac_ilp.py", line 11, in <module>
    from pulp import (  # type: ignore[import-untyped,import-not-found]
ModuleNotFoundError: No module named 'pulp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/guangyey/pytorch/test/test_testing.py", line 2402, in test_circular_dependencies
    mod = importlib.import_module(mod_name)
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/distributed/_tools/sac_ilp.py", line 24, in <module>
    raise ImportError(
ImportError: Please install pulp package. See: https://github.com/coin-or/pulp.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3223, in wrapper
    method(*args, **kwargs)
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1916, in wrapper
    return fn(*args, **kwargs)
  File "/data/guangyey/pytorch/test/test_testing.py", line 2404, in test_circular_dependencies
    raise RuntimeError(f"Failed to import {mod_name}: {e}") from e
RuntimeError: Failed to import torch.distributed._tools.sac_ilp: Please install pulp package. See: https://github.com/coin-or/pulp.

To execute this test, run the following from the base repo dir:
    python test/test_testing.py TestImports.test_circular_dependencies

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 1 test in 0.992s

FAILED (errors=1)

guangyey · 2025-07-13T06:12:19Z

Anyway, CI pass, land this PR first.
@pytorchbot merge

pytorchmergebot · 2025-07-13T06:14:55Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

guangyey · 2025-07-13T06:21:22Z

@Skylion007 I am not sure, but it still fails with pytorch-cuda-nightly that I installed via pip

(ygy_2.7) pt-gpu@mlp-milan-02:/data/guangyey/pytorch/test$ python
Python 3.10.18 | packaged by conda-forge | (main, Jun  4 2025, 14:45:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__file__
'/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/__init__.py'
>>> torch.__version__
'2.9.0.dev20250712+cu128'
>>> exit()
(ygy_2.7) pt-gpu@mlp-milan-02:/data/guangyey/pytorch/test$ python test_testing.py TestImports.test_circular_dependencies
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch._functorch.pytree_hacks` is deprecated and will be removed in a future release. Please `use torch.utils._pytree` instead.
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._shard.checkpoint` will be deprecated, use `torch.distributed.checkpoint` instead
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
  return _bootstrap._gcd_import(name[level:], package, level)
/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py:126: DeprecationWarning: `torch.distributed._sharding_spec` will be deprecated, use `torch.distributed._shard.sharding_spec` instead
  return _bootstrap._gcd_import(name[level:], package, level)
E
======================================================================
ERROR: test_circular_dependencies (__main__.TestImports)
Checks that all modules inside torch can be imported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/distributed/_tools/sac_ilp.py", line 11, in <module>
    from pulp import (  # type: ignore[import-untyped,import-not-found]
ModuleNotFoundError: No module named 'pulp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/guangyey/pytorch/test/test_testing.py", line 2402, in test_circular_dependencies
    mod = importlib.import_module(mod_name)
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/distributed/_tools/sac_ilp.py", line 24, in <module>
    raise ImportError(
ImportError: Please install pulp package. See: https://github.com/coin-or/pulp.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3223, in wrapper
    method(*args, **kwargs)
  File "/data/wengshiy/conda/envs/ygy_2.7/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1916, in wrapper
    return fn(*args, **kwargs)
  File "/data/guangyey/pytorch/test/test_testing.py", line 2404, in test_circular_dependencies
    raise RuntimeError(f"Failed to import {mod_name}: {e}") from e
RuntimeError: Failed to import torch.distributed._tools.sac_ilp: Please install pulp package. See: https://github.com/coin-or/pulp.

To execute this test, run the following from the base repo dir:
    python test/test_testing.py TestImports.test_circular_dependencies

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 1 test in 0.992s

FAILED (errors=1)

cc @malfet

malfet · 2025-07-13T06:39:26Z

What makes it XPU specific? And test plan doesn't sound reasonable

guangyey · 2025-07-14T01:52:15Z

What makes it XPU specific? And test plan doesn't sound reasonable

@malfet It seems that test_circular_dependencies would fail on CUDA if without this PR. I'm not sure why this wasn't caught by the CUDA CI job.

pytorch-bot bot added ciflow/inductor module: inductor labels Jul 12, 2025

guangyey added a commit that referenced this pull request Jul 12, 2025

Fix XPU CI UT test_circular_dependencies

006ced2

ghstack-source-id: 0a00b5a Pull Request resolved: #158189

Skylion007 approved these changes Jul 12, 2025

View reviewed changes

guangyey added this to PyTorch Intel Jul 12, 2025

guangyey added module: xpu Intel XPU related issues ciflow/xpu Run XPU CI tasks labels Jul 12, 2025

guangyey added the topic: not user facing topic category label Jul 12, 2025

pytorchbot added the open source label Jul 12, 2025

guangyey requested a review from malfet July 12, 2025 16:51

Update

508cfff

[ghstack-poisoned]

cyyever approved these changes Jul 13, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 13, 2025

pytorchmergebot added the merging label Jul 13, 2025

pytorchmergebot closed this in c68af9a Jul 13, 2025

pytorchmergebot added the Merged label Jul 13, 2025

github-project-automation bot moved this to Done in PyTorch Intel Jul 13, 2025

pytorchmergebot removed the merging label Jul 13, 2025

github-actions bot deleted the gh/guangyey/167/head branch August 13, 2025 02:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix XPU CI UT test_circular_dependencies #158189

Fix XPU CI UT test_circular_dependencies #158189

guangyey commented Jul 12, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jul 12, 2025 •

edited

Loading

Uh oh!

guangyey commented Jul 12, 2025

Uh oh!

guangyey commented Jul 12, 2025 •

edited

Loading

Uh oh!

Skylion007 commented Jul 12, 2025

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

pytorchmergebot commented Jul 13, 2025

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

malfet commented Jul 13, 2025

Uh oh!

guangyey commented Jul 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

Fix XPU CI UT test_circular_dependencies #158189

Fix XPU CI UT test_circular_dependencies #158189

Conversation

guangyey commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Uh oh!

pytorch-bot bot commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158189

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

guangyey commented Jul 12, 2025

Uh oh!

guangyey commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Skylion007 commented Jul 12, 2025

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

pytorchmergebot commented Jul 13, 2025

Merge started

Uh oh!

guangyey commented Jul 13, 2025

Uh oh!

malfet commented Jul 13, 2025

Uh oh!

guangyey commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

guangyey commented Jul 12, 2025 •

edited

Loading

pytorch-bot bot commented Jul 12, 2025 •

edited

Loading

guangyey commented Jul 12, 2025 •

edited

Loading

guangyey commented Jul 14, 2025 •

edited

Loading