Transform output on Linux doesn't match its JIT-scripted version

[datumbox]

🐛 Bug

The latest PyTorch nightly on Linux (1.10.0.dev20210825) broke the test_adjust_hue test of TorchVision. The test fails when we compare the output of the transform with its JIT scripted version:

vision/test/test_functional_tensor.py

Line 655 in a9b38db

assert out_fn_t.allclose(out_scripted, atol=atol)

The values are uint8 and as we can see from the output they are not the same. See #4313

Traceback (most recent call last):
  File "/root/project/test/test_functional_tensor.py", line 820, in test_adjust_hue
    agg_method="max"
  File "/root/project/test/test_functional_tensor.py", line 655, in check_functional_vs_PIL_vs_scripted
    assert out_fn_t.allclose(out_scripted, atol=atol)
AssertionError: assert False
 +  where False = <built-in method allclose of Tensor object at 0x7fd7c72f3b38>(tensor([[[**138**,   0, 115,  ..., 209,  40,  71],\n         [139, 245, 116,  ...,   7, 219,  89],\n         [142,  40, 146,...32],\n         [220, 137, 118,  ..., 255,  28,  68],\n         [140, 144,  41,  ..., 191,  38,  40]]], dtype=torch.uint8), atol=1e-06)
 +    where <built-in method allclose of Tensor object at 0x7fd7c72f3b38> = tensor([[[**137**,   0, 115,  ..., 209,  40,  71],\n         [139, 245, 116,  ...,   7, 219,  89],\n         [142,  40, 146,...32],\n         [220, 137, 118,  ..., 255,  28,  68],\n         [140, 144,  41,  ..., 191,  38,  40]]], dtype=torch.uint8).allclose

The test only fails on CPU not on GPU.

Possibly caused by one of these commits.

cc @vfdev-5 @datumbox

[datumbox]

@bertmaher as you suspected:

Setting torch._C._jit_override_can_fuse_on_cpu(False) fixes the issue. See PR #4317

Some PRs that could have affected this:

• Re-apply: [nnc] Support thread level parallelism in fused kernels pytorch#63776
• Add prim::VarStack and pass that transforms aten::stack to prim::VarStack pytorch#63578

The test in question is the following:

vision/test/test_functional_tensor.py

Lines 808 to 821 in 7947fc8

	@pytest.mark.parametrize('device', cpu_and_gpu())
	@pytest.mark.parametrize('dtype', (None, torch.float32, torch.float64))
	@pytest.mark.parametrize('config', [{"hue_factor": f} for f in [-0.45, -0.25, 0.0, 0.25, 0.45]])
	def test_adjust_hue(device, dtype, config):
	check_functional_vs_PIL_vs_scripted(
	F.adjust_hue,
	F_pil.adjust_hue,
	F_t.adjust_hue,
	config,
	device,
	dtype,
	tol=16.1,
	agg_method="max"
	)

Worth noting that the hue method is using stack, which is updated by one of the above PRs:

vision/torchvision/transforms/functional_tensor.py

Lines 372 to 375 in 7947fc8

	a1 = torch.stack((v, q, p, p, t, v), dim=-3)
	a2 = torch.stack((t, v, v, q, p, p), dim=-3)
	a3 = torch.stack((p, p, t, v, v, q), dim=-3)
	a4 = torch.stack((a1, a2, a3), dim=-4)

[bertmaher]

pytorch/pytorch#63776 is almost certainly the problem since it effectively turns on fusion for some workloads where it used to be disabled. Would you like me to land a PR to turn off fusion upstream, or land the setting to this repo temporarily?

[bertmaher]

I believe I've found the issues here. (1) is that the jit was doing tensor-scalar math in higher precision than eager mode (double, rather than float), and (2) remainder was subtly broken. With the linked PR the test_adjust_hue tests pass.

[datumbox]

@fmassa @NicolasHug Any idea why the TorchVision cpu tests don't run when PyTorch core commits? As far as I see, when the PR that broke vision was merged (GH pytorch/pytorch#63776 / D30484555), only pytorch_vision_gpu-buck was run. Now that the GPU runs don't execute CPU-only tests, can we make sure that we also run the CPU versions on FBcode?

[malfet]

can we make sure that we also run the CPU versions on FBcode?

@datumbox distance between TorchVision and PyTorch is more than 2 in fbcode project dependency tree, which means tests would not always run.

[bertmaher]

I've worked around the dependency distance before by adding //caffe2:libtorch as a direct dependence of the test targets I want to run (see D28908516). That just covers the "torch" part of pytorch (so not ATen, c10), but it's better than missing it.