int8 is not supported in GpuAdvancedIncSubtensor

[akhavr]

I'm working on https://github.com/pcyin/NL2code and it is failing with certain options on GPU (pcyin/NL2code#6):

Traceback (most recent call last):
  File "code_gen.py", line 154, in <module>
    model.build()
  File "/home/ubuntu/NL2code/model.py", line 209, in build
    updates=updates)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
    output_keys=output_keys)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1841, in orig_function
    fn = m.create(defaults)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/vm.py", line 1091, in make_all
    impl=impl))
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 866, in make_thunk
    on_unused_input='ignore')
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
    output_keys=output_keys)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1841, in orig_function
    fn = m.create(defaults)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/vm.py", line 1091, in make_all
    impl=impl))
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/op.py", line 955, in make_thunk
    no_recycling)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1144, in module_from_key
    src_code = lnk.get_src_code()
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1496, in get_src_code
    mod = self.get_dynamic_module()
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1540, in get_dynamic_module
    self.code_gen()
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 796, in code_gen
    name))
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gpuarray/basic_ops.py", line 400, in c_support_code_apply
    kernels = self.gpu_kernels(node, name)
  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gpuarray/subtensor.py", line 1179, in gpu_kernels
    tc=CHARMAP[dtype_x])
KeyError: ('The following error happened while compiling the node', forall_inplace,gpu,scan_fn}(Elemwise{Composite{minimum(minimum(i0, i1), i2)}}.0, InplaceGpuDimShuffle{0,x,x}.0, GpuElemwise{sub,no_inplace}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, Elemwise{Composite{minimum(minimum(i0, i1), i2)}}.0, decoder_lstm_p19, decoder_lstm_p25, decoder_lstm_p21, decoder_lstm_p23, decoder_lstm_p29, decoder_lstm_p22, decoder_lstm_p13, decoder_lstm_p15, decoder_lstm_p17, decoder_lstm_p16, decoder_lstm_p1, decoder_lstm_p3, decoder_lstm_p5, decoder_lstm_p4, decoder_lstm_p7, decoder_lstm_p9, decoder_lstm_p11, decoder_lstm_p10, GpuJoin.0, GpuElemwise{Composite{(i0 - EQ(i1, i2))}}[]<gpuarray>.0, GpuElemwise{add,no_inplace}.0, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, decoder_lstm_p30, decoder_lstm_p32, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, decoder_lstm_p27, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0), '\n', 'The following error happened while compiling the node', GpuAdvancedIncSubtensor1_dev20{inplace=True, set_instead_of_inc=True}(InplaceGpuDimShuffle{1,0}.0, GpuArrayConstant{[[1]]}, GpuFromHost<None>.0), '\n', 'int8')

Obvious fix like in https://github.com/akhavr/Theano/blob/master/theano/gpuarray/subtensor.py doesn't work because of type mismatch

  File "/home/ubuntu/NL2code/.env/local/lib/python2.7/site-packages/theano/gof/cc.py", line 1637, in cthunk_factory
    *(in_storage + out_storage + orphd))
RuntimeError: ('The following error happened while compiling the node', forall_inplace,gpu,scan_fn}(Elemwise{Composite{minimum(minimum(i0, i1), i2)}}.0, InplaceGpuDimShuffle{0,x,x}.0, GpuElemwise{sub,no_inplace}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, Elemwise{Composite{minimum(minimum(i0, i1), i2)}}.0, decoder_lstm_p19, decoder_lstm_p25, decoder_lstm_p21, decoder_lstm_p23, decoder_lstm_p29, decoder_lstm_p22, decoder_lstm_p13, decoder_lstm_p15, decoder_lstm_p17, decoder_lstm_p16, decoder_lstm_p1, decoder_lstm_p3, decoder_lstm_p5, decoder_lstm_p4, decoder_lstm_p7, decoder_lstm_p9, decoder_lstm_p11, decoder_lstm_p10, GpuJoin.0, GpuElemwise{Composite{(i0 - EQ(i1, i2))}}[]<gpuarray>.0, GpuElemwise{add,no_inplace}.0, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, decoder_lstm_p30, decoder_lstm_p32, InplaceGpuDimShuffle{x,x,0}.0, GpuFromHost<None>.0, decoder_lstm_p27, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0, GpuSubtensor{int64}.0), '\n', 'The following error happened while compiling the node', GpuAdvancedIncSubtensor1_dev20{inplace=True, set_instead_of_inc=True}(InplaceGpuDimShuffle{1,0}.0, GpuArrayConstant{[[1]]}, GpuFromHost<None>.0), '\n', 'GpuKernel_init error 3: nvrtcCompileProgram: NVRTC_ERROR_COMPILATION')

[abergeron]

Yes, that doesn't work. There are no atomic int8 operations so we probably will not support it.

The fix would be to use a larger dtype or run that node on the CPU.

[abergeron]

Although, we should probably teach the optimizer to not move those cases to the GPU so I'll leave the issue open until that is fixed.

[abergeron]

Does #6628 fix your issue?

Although it might be better in some cases to leave the operation on the CPU since it could be faster sometimes. The non-dev20 version of the kernel can be quite slow.

[akhavr]

@abergeron, thanks. Will check #6628 in next few days.

[akhavr]

Thanks, it worked! Should I close the issue now?

[abergeron]

No, that's ok, we will close it when the fix is merged.

[nouiz]

The fix is merged. thanks.