test_grid_sample (from TestNN) fails on POWER

[Flamefire]

🐛 Bug

When running the tests I get the following error:

FAIL: test_grid_sample (__main__.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_nn.py", line 7447, in test_grid_sample
    self.assertEqual(grid.grad, groundtruth,
  File "/tmp/easybuild-tmp/eb-JA1XOe/tmpRJ107i/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1136, in assertEqual
    self.assertTrue(result, msg=msg)
AssertionError: False is not true : gradient groundtruth comparison failed for mode=bilinear, padding_mode=border

To Reproduce

Steps to reproduce the behavior:

1. Compile
2. python run_test.py

Expected behavior

Tests pass

Environment

• PyTorch Version (e.g., 1.0): 1.7.1
• OS (e.g., Linux): RHEL 7
• How you installed PyTorch (conda, pip, source): source
• Build command you used (if compiling from source): PYTORCH_BUILD_VERSION=1.7.1 PYTORCH_BUILD_NUMBER=1 MAX_JOBS=176 BLAS=Eigen USE_FFMPEG=1 BUILD_CUSTOM_PROTOBUF=0 USE_IBVERBS=1 USE_CUDA=0 USE_NNPACK=0 USE_QNNPACK=0 USE_PYTORCH_QNNPACK=0 USE_XNNPACK=0 USE_METAL=0 python setup.py build
• Python version: 3.8.6
• CUDA/cuDNN version: None
• GPU models and configuration: None
• Any other relevant information: GCC 10.2 on ppc9le

Additional context

Maybe related to #54535 as e.g. MKL is not available on POWER?

cc @mruberry @VitalyFedyunin @walterddr

[Flamefire]

FWIW: I printed the values of x and y for the failing test and got:

AssertionError: False is not true : gradient groundtruth comparison failed for mode=bilinear, padding_mode=border
tensor([[[[ -0.0000e+00,  -0.0000e+00],
          [-2.4113e-314,   0.0000e+00],
          [  0.0000e+00,   0.0000e+00],
          [  0.0000e+00,   0.0000e+00]],

         [[  0.0000e+00,   0.0000e+00],
          [  0.0000e+00,   0.0000e+00],
          [  0.0000e+00,   0.0000e+00],
          [  0.0000e+00,   0.0000e+00]]]]) != 
tensor([[[[-0., -0.],
          [-0., 0.],
          [2., 0.],
          [2., 0.]],

         [[0., 0.],
          [0., 0.],
          [2., 0.],
          [2., 0.]]]])

I.e. the result is all zeros which looks like a serious bug to me