DISABLED test_coalesce_reference_cycle_cpu_float64 (__main__.TestSparseCPU)

[pytorch-bot]

Platforms: dynamo

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

Debugging instructions (after clicking on the recent samples link):
DO NOT BE ALARMED IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:

1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for test_coalesce_reference_cycle_cpu_float64
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.

2022-11-20T01:57:36.5512960Z   test_coalesce_reference_cycle_cpu_float64 (__main__.TestSparseCPU) ... [2022-11-20 01:57:12,155] torch._dynamo.convert_frame: [ERROR] WON'T CONVERT test_coalesce_reference_cycle test_sparse.py line 272 
2022-11-20T01:57:36.5513300Z due to: 
2022-11-20T01:57:36.5513473Z Traceback (most recent call last):
2022-11-20T01:57:36.5513866Z   File "/opt/conda/lib/python3.7/site-packages/torch/_dynamo/variables/builder.py", line 820, in wrap_fx_proxy_cls
2022-11-20T01:57:36.5514214Z     + f"{typestr(example_value)} {proxy.node.op} {proxy.node.target}"
2022-11-20T01:57:36.5514633Z AssertionError: torch.* op returned non-Tensor _WeakTensorRef call_function <class 'torch._C._WeakTensorRef'>
2022-11-20T01:57:36.5514849Z 
2022-11-20T01:57:36.5514920Z from user code:
2022-11-20T01:57:36.5515153Z    File "test_sparse.py", line 278, in test_coalesce_reference_cycle
2022-11-20T01:57:36.5515399Z     t_ref = torch._C._WeakTensorRef(t)
2022-11-20T01:57:36.5515527Z 
2022-11-20T01:57:36.5515642Z Set torch._dynamo.config.verbose=True for more information
2022-11-20T01:57:36.5515799Z 
2022-11-20T01:57:36.5515803Z 
2022-11-20T01:57:36.5516068Z [2022-11-20 01:57:12,164] torch._dynamo.convert_frame: [ERROR] WON'T CONVERT test_sparse_sum test_sparse.py line 284 
2022-11-20T01:57:36.5516328Z due to: 
2022-11-20T01:57:36.5516511Z Traceback (most recent call last):
2022-11-20T01:57:36.5516853Z   File "/opt/conda/lib/python3.7/site-packages/torch/_dynamo/utils.py", line 1093, in run_node
2022-11-20T01:57:36.5517126Z     return node.target(*args, **kwargs)
2022-11-20T01:57:36.5517668Z RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.
2022-11-20T01:57:36.5517964Z 
2022-11-20T01:57:36.5518089Z The above exception was the direct cause of the following exception:
2022-11-20T01:57:36.5518255Z 
2022-11-20T01:57:36.5518347Z Traceback (most recent call last):
2022-11-20T01:57:36.5518698Z   File "/opt/conda/lib/python3.7/site-packages/torch/_dynamo/utils.py", line 1104, in run_node
2022-11-20T01:57:36.5518942Z     ) from e
2022-11-20T01:57:36.5519550Z RuntimeError: Failed running call_function <built-in method sparse_coo_tensor of type object at 0x7fa2bdc44d08>(*(FakeTensor(FakeTensor(..., device='meta', size=(2, 1), dtype=torch.int64), cpu), FakeTensor(FakeTensor(..., device='meta', size=(1, 1, 4), dtype=torch.float64), cpu)), **{}):
2022-11-20T01:57:36.5520294Z The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.
2022-11-20T01:57:36.5520690Z (scroll up for backtrace)
2022-11-20T01:57:36.5520809Z 
2022-11-20T01:57:36.5520978Z The above exception was the direct cause of the following exception:
2022-11-20T01:57:36.5521145Z 
2022-11-20T01:57:36.5521224Z Traceback (most recent call last):
2022-11-20T01:57:36.5521589Z   File "/opt/conda/lib/python3.7/site-packages/torch/_dynamo/utils.py", line 1072, in get_fake_value
2022-11-20T01:57:36.5521873Z     raise TorchRuntimeError() from e
2022-11-20T01:57:36.5522112Z torch._dynamo.exc.TorchRuntimeError: 
2022-11-20T01:57:36.5522244Z 
2022-11-20T01:57:36.5522315Z from user code:
2022-11-20T01:57:36.5522530Z    File "test_sparse.py", line 288, in test_sparse_sum
2022-11-20T01:57:36.5522765Z     S = torch.sparse_coo_tensor(i, v)
2022-11-20T01:57:36.5522879Z 
2022-11-20T01:57:36.5523008Z Set torch._dynamo.config.verbose=True for more information
2022-11-20T01:57:36.5523164Z 
2022-11-20T01:57:36.5523168Z 
2022-11-20T01:57:36.5523236Z FAIL (0.015s)

cc @nikitaved @pearu @cpuhrsch @amjames @bhosmer @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire

[pytorch-bot]

Hello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disable a test in PyTorch CI. The information I have parsed is below:

• Test name: test_coalesce_reference_cycle_cpu_float64 (__main__.TestSparseCPU)
• Platforms for which to skip the test: dynamo

Within ~15 minutes, test_coalesce_reference_cycle_cpu_float64 (__main__.TestSparseCPU) will be disabled in PyTorch CI for these platforms: dynamo. Please verify that your test name looks correct, e.g., test_cuda_assert_async (__main__.TestCuda).

To modify the platforms list, please include a line in the issue body, like below. The default action will disable the test for all platforms if no platforms list is specified.

Platforms: case-insensitive, list, of, platforms

We currently support the following platforms: asan, dynamo, linux, mac, macos, rocm, win, windows.

[cpuhrsch]

@ezyang - do you know what could have happened here? For reference, the test is

pytorch/test/test_sparse.py

Lines 272 to 296 in 098cbe2

	@dtypes(torch.double)
	def test_coalesce_reference_cycle(self, device, dtype):
	# Test coalesce doesn't create autograd graph cycles (gh-52253)
	# Sanity check that the helper class works as expected
	t = torch.rand(2)
	t_ref = torch._C._WeakTensorRef(t)
	self.assertFalse(t_ref.expired())
	del t
	self.assertTrue(t_ref.expired())
	def test_sparse_sum():
	i = torch.tensor([[0], [4]], dtype=torch.long, device=device)
	v = torch.tensor([[[-0.4567, -1.8797, 0.0380, 1.4316]]],
	dtype=dtype, device=device)
	S = torch.sparse_coo_tensor(i, v)
	S = S.coalesce()
	S.requires_grad_(True)
	S2 = S.coalesce()
	self.assertTrue(S2.is_coalesced())
	return torch._C._WeakTensorRef(S2)
	ref = test_sparse_sum()
	self.assertTrue(ref.expired())

[ezyang]

No, the test looks very simple and should work.

[cpuhrsch]

@amjames - can you take a look?

[amjames]

I think someone from dynamo needs to look at this. I was able to reproduce locally, but I don't have the slightest idea what is going wrong.

simple stand alone reproduction

If you do not optimize the test function with dynamo everything works fine.

[pytorch-bot]

Resolving the issue because the test is not flaky anymore after 206 reruns without any failures and the issue hasn't been updated in 14 days. Please reopen the issue to re-disable the test if you think this is a false positive

[huydhn]

Reopen this as this is not fixed yet https://hud.pytorch.org/pytorch/pytorch/commit/0d0ebcdfe5697936014d5f2b5fed533c086eab65