CUDA cudaDeviceSynchronize error

[alvin319]

I followed the README and built a Linux GPU version of the binary, and everything works pretty well for most images, but then all of the sudden I ran into the following errors. I'm using a V100 GPU and unfortunately, I didn't manage to obtain the GPU memory usage at the time.

CUDA error at patch_based_synthesis/src/ebsynth_cuda.cu:598 code=2 "cudaDeviceSynchronize()"
CUDA error at patch_based_synthesis/src/ebsynth_cuda.cu:598 code=2 "cudaDeviceSynchronize()"
CUDA error at patch_based_synthesis/src/ebsynth_cuda.cu:598 code=2 "cudaDeviceSynchronize()"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:283 code=2 "cudaMemcpy2D((void*)dst.data(), dstPitchInBytes, src.tmp_data, src.tmp_pitch, src.width*N*sizeof(T), src.height, cudaMemcpyDeviceToHost)"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:107 code=2 "cudaDestroyTextureObject(texObj)"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:108 code=2 "cudaFree(data)"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:196 code=2 "cudaFree(tmp_data)"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:238 code=2 "cudaMemcpy2D(dst.tmp_data, dst.tmp_pitch, src.data(), srcPitchInBytes, srcWidthInBytes, src.height(), cudaMemcpyHostToDevice)"
CUDA error at patch_based_synthesis/src/ebsynth_cuda.cu:923 code=2 "cudaDeviceSynchronize()"
CUDA error at patch_based_synthesis/src/ebsynth_cuda_texarray2.h:238 code=2 "cudaMemcpy2D(dst.tmp_data, dst.tmp_pitch, src.data(), srcPitchInBytes, srcWidthInBytes, src.height(), cudaMemcpyHostToDevice)"

[OndrejTexler]

Hello Alvin.

From the report you attached, it is really hard to tell what went wrong (I know, it is hard to get better report).

This might be a memory issue, but it could be something else. Try smaller images and make sure that the images are correct (try to open them in some paint tool). Unfortunately, this is the only advice/idea I have.

Anyway, it has been 10 days already (sorry for a late response), did you manage to find what was the problem?

[alvin319]

Thanks for the response! It's very likely due to the out-of-memory error.

[OndrejTexler]

Well, if it is due to the out-of-memory issue, there is not much you can do about it. The best advice is that you run it on CPU,
--patch_based_backend "CPU".

Apart from that, you can work with smaller images, you can reimplement the code so that it uses less memory, or you can get a better GPU :-)

Anyway, I am closing this issue. If you want to discuss it further, feel free to open the issue or create a new one!