OutOfMemoryError when generate on Azure ND6 VM

[taurenshaman]

Info about Azure ND6 VM

NVIDIA Tesla P40 x1：6 cores；RAM 112.00 GiB；Storage 336 GiB；$2.07/hour
Architecture: Pascal
3840 CUDA cores; 1303/1531 MHz
GPU Memory: 24 GB GDDR5；384 bit

the picture info about WP_20170302_15_57_12_Rich.jpg

resolution ratio: 2960x5258
size: 3.63MB

Error when using GPU

`
jerin@mosmoms:/mnt/mosp$ python generate.py 20180321/WP_20170302_15_57_12_Rich.jpg -m models/composition.model -o 20180321/WP_20170302_15_57_12_Rich-composition.jpg -g 0
Traceback (most recent call last):
File "cupy/cuda/memory.pyx", line 623, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc
File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init
File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc
File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "cupy/cuda/memory.pyx", line 629, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc
File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init
File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc
File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "cupy/cuda/memory.pyx", line 635, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc
File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init
File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc
File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "generate.py", line 45, in
y = model(x)
File "/mnt/mosp/net.py", line 63, in call
h = self.b4(F.elu(self.d1(h)))
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/functions/activation/elu.py", line 85, in elu
return ELU(alpha=alpha)(x)
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 212, in call
ret = node.apply(inputs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function_node.py", line 222, in apply
outputs = self.forward(in_data)
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 117, in forward
return self._function.forward(inputs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 319, in forward
return self.forward_gpu(inputs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/functions/activation/elu.py", line 32, in forward_gpu
x[0], self.alpha)
File "cupy/core/elementwise.pxi", line 559, in cupy.core.core.ElementwiseKernel.call
File "cupy/core/elementwise.pxi", line 402, in cupy.core.core._get_out_args_with_params
File "cupy/core/core.pyx", line 93, in cupy.core.core.ndarray.init
File "cupy/cuda/memory.pyx", line 392, in cupy.cuda.memory.alloc
File "cupy/cuda/memory.pyx", line 809, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 830, in cupy.cuda.memory.MemoryPool.malloc
File "cupy/cuda/memory.pyx", line 593, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
File "cupy/cuda/memory.pyx", line 641, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
cupy.cuda.memory.OutOfMemoryError: out of memory to allocate 1048919040 bytes (total 23823933440 bytes)
`

successful when using CPU

cost: 205 seconds