cudaGetLastError raises "invalid device ordinal" when running multiple containers on one host

[tgaddair]

Environment:

1. Framework: any
2. Framework version:
3. Horovod version: e4554de
4. MPI version:
5. CUDA version: 10.0
6. NCCL version: 2.6.4
7. Python version:
8. Spark / PySpark version:
9. OS and version:
10. GCC version:

When running a Horovod job that spans multiple containers that reside on the same host (with isolated GPUs), the following error is raised:

RuntimeError: ScaleBufferCudaImpl failed: invalid device ordinal

Through debugging, it was determined this was introduced in #1949 with the line ErrorCheck("ScaleBufferCudaImpl", cudaGetLastError());.

It does appear that this is somehow specific to Horovod, as it does not occur in pure MPI + CUDA. However, it is not known if this is a CUDA version specific issue. By introducing a similar call to the commit just before, this error also occurs, so it is not specific to other changes introduced in that commit.

As a workaround, we will remove this check until the underlying cause of this error is known.

cc @romerojosh