"No kernel image is available for execution on the device Error" when running test_spatial_narrow_as_op.py

[FduJyy]

Expected results

Pass the test.

Actual results

Failed the test.

(caffe) jyy@jyy-OptiPlex-9020:~/Detectron$ python ./tests/test_spatial_narrow_as_op.py
E0309 14:17:00.375676  3086 init_intrinsics_check.cc:59] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0309 14:17:00.375697  3086 init_intrinsics_check.cc:59] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0309 14:17:00.375700  3086 init_intrinsics_check.cc:59] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
Found Detectron ops lib: /home/jyy/anaconda3/envs/caffe/lib/libcaffe2_detectron_ops_gpu.so
F.E
======================================================================
ERROR: test_small_forward_and_gradient (__main__.SpatialNarrowAsOpTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./tests/test_spatial_narrow_as_op.py", line 59, in test_small_forward_and_gradient
    self._run_test(A, B, check_grad=True)
  File "./tests/test_spatial_narrow_as_op.py", line 49, in _run_test
    res, grad, grad_estimated = gc.CheckSimple(op, [A, B], 0, [0])
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/caffe2/python/gradient_checker.py", line 284, in CheckSimple
    outputs_with_grads
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/caffe2/python/gradient_checker.py", line 201, in GetLossAndGrad
    workspace.RunOperatorsOnce(grad_ops)
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/caffe2/python/workspace.py", line 184, in RunOperatorsOnce
    success = RunOperatorOnce(op)
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/caffe2/python/workspace.py", line 179, in RunOperatorOnce
    return C.run_operator_once(StringifyProto(operator))
RuntimeError: [enforce fail at context_gpu.h:171] . Encountered CUDA error: no kernel image is available for execution on the device Error from operator: 
input: "A" input: "B" input: "C_grad" output: "A_grad" name: "" type: "SpatialNarrowAsGradient" device_option { device_type: 1 cuda_gpu_id: 0 } is_gradient_op: true

======================================================================
FAIL: test_large_forward (__main__.SpatialNarrowAsOpTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./tests/test_spatial_narrow_as_op.py", line 68, in test_large_forward
    self._run_test(A, B)
  File "./tests/test_spatial_narrow_as_op.py", line 54, in _run_test
    np.testing.assert_allclose(C, C_ref, rtol=1e-5, atol=1e-08)
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/numpy/testing/nose_tools/utils.py", line 1396, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/jyy/anaconda3/envs/caffe/lib/python2.7/site-packages/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-08

(mismatch 100.0%)
 x: array([[[[0., 0., 0., ..., 0., 0., 0.],
         [0., 0., 0., ..., 0., 0., 0.],
         [0., 0., 0., ..., 0., 0., 0.],...
 y: array([[[[ 1.707480e+00,  1.710607e+00,  1.279160e+00, ...,
          -9.014695e-01, -1.781531e+00,  4.036736e-01],
         [ 1.895508e+00, -3.324545e-01,  3.578335e-01, ...,...

----------------------------------------------------------------------
Ran 3 tests in 0.557s

FAILED (failures=1, errors=1)

Detailed steps to reproduce

python ./tests/test_spatial_narrow_as_op.py

System information

• Operating system: Ubuntu 16.04
• Compiler version: ?
• CUDA version: 9.0
• cuDNN version: 7.0
• NVIDIA driver version: 390.25
• GPU models (for all devices if they are not all the same): GTX 1060
• PYTHONPATH environment variable: None
• python --version output: Python 2.7.14 :: Anaconda, Inc.
• Anything else that seems relevant: ? Caffe2 was installed via pre-built binaries.

[rbgirshick]

Duplicate of #260.

[manoshape]

how to fix it ? @rbgirshick

[yfzon]

I met the similar error when I want to use the tensorflow op compiled by nvcc: Could not launch cub::DeviceSegmentedRadixSort::SortPairsDescending to sort input, temp_storage_bytes: 599295, status: no kernel image is available for execution on the device. I found this issue and knew that it's caused by the gpu compute capability. I use Tesla40 and add -gencode arch=compute_61,code=compute_61 my compile file. Solved it finally. Hope it can help you.

[ghost]

F0710 09:54:00.477679 28834 clip_layer.cu:33] Check failed: error == cudaSuccess (48 vs. 0) no kernel image is available for execution on the device

[ghost]

I met the similar error when I want to use the tensorflow op compiled by nvcc: Could not launch cub::DeviceSegmentedRadixSort::SortPairsDescending to sort input, temp_storage_bytes: 599295, status: no kernel image is available for execution on the device. I found this issue and knew that it's caused by the gpu compute capability. I use Tesla40 and add -gencode arch=compute_61,code=compute_61 my compile file. Solved it finally. Hope it can help you.

我的 GPU 是 Tegra X1，算力是多少，你这张图的链接发一下啊