Skip to content

Conversation

@Gamrix
Copy link
Contributor

@Gamrix Gamrix commented Jul 12, 2022

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jul 12, 2022

🔗 Helpful links

❌ 4 New Failures

As of commit 86a20a3 (more details on the Dr. CI page):

Expand to see more
  • 4/4 failures introduced in this PR

🕵️ 4 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages

See GitHub Actions build pull / linux-bionic-cuda11.6-py3.7-gcc7 / test (default, 1, 4, linux.4xlarge.nvidia.gpu) (1/4)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-07-12T19:26:40.1448098Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:26:40.1443529Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1185, in set_rng_seed
2022-07-12T19:26:40.1443922Z     torch.manual_seed(seed)
2022-07-12T19:26:40.1444383Z   File "/opt/conda/lib/python3.7/site-packages/torch/random.py", line 40, in manual_seed
2022-07-12T19:26:40.1444765Z     torch.cuda.manual_seed_all(seed)
2022-07-12T19:26:40.1445735Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
2022-07-12T19:26:40.1446135Z     _lazy_call(cb, seed_all=True)
2022-07-12T19:26:40.1446613Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
2022-07-12T19:26:40.1446945Z     callable()
2022-07-12T19:26:40.1447375Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in cb
2022-07-12T19:26:40.1447749Z     default_generator.manual_seed(seed)
2022-07-12T19:26:40.1448098Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:26:40.1448722Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-07-12T19:26:40.1449207Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-07-12T19:26:40.1449426Z 
2022-07-12T19:26:40.1449706Z ----------------------------------------------------------------------
2022-07-12T19:26:40.1450024Z Ran 154 tests in 92.766s
2022-07-12T19:26:40.1450188Z 
2022-07-12T19:26:40.1450350Z FAILED (errors=1, skipped=9, expected failures=3)
2022-07-12T19:26:40.1450554Z 
2022-07-12T19:26:40.1450677Z Generating XML reports...
2022-07-12T19:26:40.1641807Z Generated XML report: test-reports/python-unittest/test_ops/TEST-TestCommonCUDA-20220712192506.xml

See GitHub Actions build pull / linux-bionic-cuda11.6-py3.7-gcc7 / test (default, 2, 4, linux.4xlarge.nvidia.gpu) (2/4)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-07-12T19:28:38.2672869Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:28:38.2668684Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1185, in set_rng_seed
2022-07-12T19:28:38.2669067Z     torch.manual_seed(seed)
2022-07-12T19:28:38.2669504Z   File "/opt/conda/lib/python3.7/site-packages/torch/random.py", line 40, in manual_seed
2022-07-12T19:28:38.2669879Z     torch.cuda.manual_seed_all(seed)
2022-07-12T19:28:38.2670368Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
2022-07-12T19:28:38.2670906Z     _lazy_call(cb, seed_all=True)
2022-07-12T19:28:38.2671379Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
2022-07-12T19:28:38.2671718Z     callable()
2022-07-12T19:28:38.2672126Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in cb
2022-07-12T19:28:38.2672504Z     default_generator.manual_seed(seed)
2022-07-12T19:28:38.2672869Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:28:38.2673345Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-07-12T19:28:38.2673781Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-07-12T19:28:38.2674000Z 
2022-07-12T19:28:38.2674272Z ----------------------------------------------------------------------
2022-07-12T19:28:38.2674605Z Ran 150 tests in 37.538s
2022-07-12T19:28:38.2674771Z 
2022-07-12T19:28:38.2674893Z FAILED (errors=1, expected failures=3)
2022-07-12T19:28:38.2675087Z 
2022-07-12T19:28:38.2675211Z Generating XML reports...
2022-07-12T19:28:38.2856872Z Generated XML report: test-reports/python-unittest/test_decomp/TEST-TestDecompCUDA-20220712192800.xml

See GitHub Actions build pull / linux-bionic-cuda11.6-py3.7-gcc7 / test (default, 3, 4, linux.4xlarge.nvidia.gpu) (3/4)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-07-12T19:46:44.9252555Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:46:44.9248048Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1185, in set_rng_seed
2022-07-12T19:46:44.9248444Z     torch.manual_seed(seed)
2022-07-12T19:46:44.9248904Z   File "/opt/conda/lib/python3.7/site-packages/torch/random.py", line 40, in manual_seed
2022-07-12T19:46:44.9249281Z     torch.cuda.manual_seed_all(seed)
2022-07-12T19:46:44.9249781Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
2022-07-12T19:46:44.9250155Z     _lazy_call(cb, seed_all=True)
2022-07-12T19:46:44.9250603Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
2022-07-12T19:46:44.9251361Z     callable()
2022-07-12T19:46:44.9251832Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in cb
2022-07-12T19:46:44.9252211Z     default_generator.manual_seed(seed)
2022-07-12T19:46:44.9252555Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:46:44.9253032Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-07-12T19:46:44.9253484Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-07-12T19:46:44.9253704Z 
2022-07-12T19:46:44.9253957Z ----------------------------------------------------------------------
2022-07-12T19:46:44.9254489Z Ran 213 tests in 37.347s
2022-07-12T19:46:44.9254658Z 
2022-07-12T19:46:44.9254822Z FAILED (errors=1, skipped=34, expected failures=3)
2022-07-12T19:46:44.9255028Z 
2022-07-12T19:46:44.9255153Z Generating XML reports...
2022-07-12T19:46:44.9493906Z Generated XML report: test-reports/python-unittest/test_sparse/TEST-TestSparseCUDA-20220712194607.xml

See GitHub Actions build pull / linux-bionic-cuda11.6-py3.7-gcc7 / test (default, 4, 4, linux.4xlarge.nvidia.gpu) (4/4)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-07-12T19:24:01.7405668Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:24:01.7401427Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1185, in set_rng_seed
2022-07-12T19:24:01.7401842Z     torch.manual_seed(seed)
2022-07-12T19:24:01.7402310Z   File "/opt/conda/lib/python3.7/site-packages/torch/random.py", line 40, in manual_seed
2022-07-12T19:24:01.7402802Z     torch.cuda.manual_seed_all(seed)
2022-07-12T19:24:01.7403319Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 113, in manual_seed_all
2022-07-12T19:24:01.7403701Z     _lazy_call(cb, seed_all=True)
2022-07-12T19:24:01.7404149Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 156, in _lazy_call
2022-07-12T19:24:01.7404502Z     callable()
2022-07-12T19:24:01.7404942Z   File "/opt/conda/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in cb
2022-07-12T19:24:01.7405299Z     default_generator.manual_seed(seed)
2022-07-12T19:24:01.7405668Z RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-12T19:24:01.7406158Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-07-12T19:24:01.7406615Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-07-12T19:24:01.7406834Z 
2022-07-12T19:24:01.7407088Z ----------------------------------------------------------------------
2022-07-12T19:24:01.7407438Z Ran 150 tests in 17.477s
2022-07-12T19:24:01.7407607Z 
2022-07-12T19:24:01.7407752Z FAILED (errors=1, expected failures=3)
2022-07-12T19:24:01.7407943Z 
2022-07-12T19:24:01.7408076Z Generating XML reports...
2022-07-12T19:24:01.7598132Z Generated XML report: test-reports/python-unittest/test_meta/TEST-TestMetaCUDA-20220712192343.xml

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

Gamrix added a commit that referenced this pull request Jul 12, 2022
@facebook-github-bot facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Jul 12, 2022
Copy link
Contributor

@davidberard98 davidberard98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, this looks accurate to me

@Gamrix
Copy link
Contributor Author

Gamrix commented Jul 12, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Merge failed due to Refusing to merge as mandatory check(s) pull failed for rule superuser
Raised by https://github.com/pytorch/pytorch/actions/runs/2658624717

@Gamrix
Copy link
Contributor Author

Gamrix commented Jul 14, 2022

@pytorchbot merge --force

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Contributor

Hey @Gamrix.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Jul 15, 2022
#81341)

Summary:
Pull Request resolved: #81341
Approved by: https://github.com/davidberard98

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/cf6499e5e89cb7bc443c1b11e9a1c3f9986356fc

Reviewed By: DanilBaibak

Differential Revision: D37876471

Pulled By: Gamrix

fbshipit-source-id: b624260db1129dbc70a1d70beab68d34743161b7
@facebook-github-bot facebook-github-bot deleted the gh/gamrix/83/head branch July 18, 2022 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed Merged oncall: jit Add this issue/PR to JIT oncall triage queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants