CUDA error 8: invalid device function

[RdlP]

I am using dockers from: https://hub.docker.com/r/caffe2ai/caffe2/

I have a custom network model in caffe2 (written by me), and it works perfectly in CPU. Now I am trying to run it over GPU. In order to do that I defined device_opts as follow

device_opts = caffe2_pb2.DeviceOption()
device_opts.device_type = caffe2_pb2.CUDA
device_opts.cuda_gpu_id = 0

and pass it to the blobs

workspace.FeedBlob("x_train", x_train, device_opts)
workspace.FeedBlob("y_train", y_train, device_opts)

finally to predict I use RunNet function instead of Predictor interface (because I read that it doesn't work with GPU)

workspace.RunNet(model.net)

When I run the code I get many cuda errors

CUDA error 8 [/caffe2/third_party/cub/cub/block/../iterator/../util_device.cuh, 151]: invalid device function
CUDA error 8 [/caffe2/third_party/cub/cub/device/dispatch/dispatch_reduce.cuh, 697]: invalid device function
CUDA error 8 [/caffe2/third_party/cub/cub/block/../iterator/../util_device.cuh, 151]: invalid device function
CUDA error 8 [/caffe2/third_party/cub/cub/device/dispatch/dispatch_reduce.cuh, 697]: invalid device function

Anybody knows about these errors?

[Yangqing]

@slayton58 @lukeyeager do you guys have a guess? :) This looks a bit new to me...

[slayton58]

Could be compiled for the wrong GPU architecture(s), but I don't have enough details to say for sure.

[pietern]

@RdlP What kind of GPU are you trying to run this on?

@slayton58 I think I've seen this as well when the CUDA ARCHs didn't match the GPU arch.

[RdlP]

@pietern I am trying to run on GeForce GTX 750 Ti with 2GB

[slayton58]

That's a Maxwell card, so it should normally be covered - @Yangqing @pietern what architectures does the docker build use? My reading of what I can find suggests sm50 and 52 should be covered...