ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[25088,4096]

[why4101130]

我使用faster rcnn tensorflow 进行训练时，出现了这个错误：
D:\python\program\python.exe "D:/python/project/Faster-RCNN-TensorFlow-Python3.5-master (2)/Faster-RCNN-TensorFlow-Python3.5-master/train.py"
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:135] successfully opened CUDA library cublas64_80.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:135] successfully opened CUDA library cudnn64_5.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:135] successfully opened CUDA library cufft64_80.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:135] successfully opened CUDA library nvcuda.dll locally
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\dso_loader.cc:135] successfully opened CUDA library curand64_80.dll locally
Loaded dataset voc_2007_trainval for training
Set proposal method: gt
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from D:\python\project\Faster-RCNN-TensorFlow-Python3.5-master (2)\Faster-RCNN-TensorFlow-Python3.5-master\data\cache\voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1060 3GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 3.00GiB
Free memory: 2.44GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:906] DMA: 0
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:916] 0: Y
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 3GB, pci bus id: 0000:01:00.0)
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots
D:\python\program\lib\site-packages\tensorflow\python\ops\gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Loading initial model weights from ./data/imagenet_weights/vgg16.ckpt
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (32768): Total Chunks: 1, Chunks in use: 0 53.8KiB allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (33554432): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (67108864): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:643] Bin (268435456): Total Chunks: 1, Chunks in use: 0 374.51MiB allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:660] Bin for 392.00MiB was 256.00MiB, Chunk State:
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:666] Size: 374.51MiB | Requested Size: 0B | in_use: 0, prev: Size: 72.0KiB | Requested Size: 72.0KiB | in_use: 1
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400000 of size 1280
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400500 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400600 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400A00 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400C00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400D00 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503400F00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401000 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401400 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401500 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401A00 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401E00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503401F00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503402700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503402800 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503403000 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503403100 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503403900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503403A00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503404200 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503404300 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503404B00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503404C00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503405400 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503405500 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503405600 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503405E00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503405F00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406000 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406100 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406200 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406300 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406400 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503406500 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340A500 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340A600 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340E600 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340E700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340E800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340E900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340EA00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340EB00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340EC00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340ED00 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340F100 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050340F900 of size 36864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503418900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503418A00 of size 73728
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050342AA00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050342AB00 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050342EB00 of size 49152
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050343AB00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050343AC00 of size 196608
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346AC00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346AD00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346AE00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346AF00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B000 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B100 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B200 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B300 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B400 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B500 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B600 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346B700 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050346F700 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503473700 of size 49152
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050347F700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000050347F800 of size 196608
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034AF800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034AF900 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034AFA00 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034AFE00 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B0200 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B0600 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B0E00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B1600 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B1E00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B2600 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B2E00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B3600 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034B3E00 of size 36864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034BCE00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034BCF00 of size 73728
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034CEF00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034CF000 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034D3000 of size 16384
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034D7000 of size 49152
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E3000 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E3100 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E3200 of size 6912
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E4D00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E4E00 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E4F00 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E5100 of size 512
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E5300 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E5700 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E5B00 of size 1024
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E5F00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E6700 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E6F00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E7700 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E7F00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E8700 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E8F00 of size 2048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034E9700 of size 36864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034F2700 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 00000005034F2800 of size 256
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 00000005034F2900 of size 55040
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503600000 of size 2097152
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503800000 of size 4194304
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000503C00000 of size 8388608
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000504400000 of size 16777216
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000505400000 of size 536870912
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000525400000 of size 67108864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000529400000 of size 469762048
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000545600000 of size 67108864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000549600000 of size 1179648
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000549720000 of size 2359296
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000549960000 of size 2359296
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000549BA0000 of size 4718592
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054A020000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054A920000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054B220000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054BB20000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054C420000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054CD20000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000054D620000 of size 411041792
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000565E20000 of size 67108864
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000569E20000 of size 196608
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000569E50000 of size 147456
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000569E74000 of size 294912
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000569EBC000 of size 589824
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 0000000569F4C000 of size 1179648
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056A06C000 of size 2359296
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056A2AC000 of size 2359296
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056A4EC000 of size 4718592
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056A96C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056B26C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056BB6C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056C46C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056CD6C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056D66C000 of size 9437184
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:678] Chunk at 000000056DF6C000 of size 73728
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:687] Free at 000000056DF7E000 of size 392699904
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:693] Summary of in-use Chunks by size:
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 57 Chunks of size 256 totalling 14.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 4 Chunks of size 512 totalling 2.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 10 Chunks of size 1024 totalling 10.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.3KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 22 Chunks of size 2048 totalling 44.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 6912 totalling 6.8KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 7 Chunks of size 16384 totalling 112.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 36864 totalling 108.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 49152 totalling 144.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 73728 totalling 216.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 147456 totalling 144.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 196608 totalling 576.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 294912 totalling 288.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 589824 totalling 576.0KiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 2 Chunks of size 1179648 totalling 2.25MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 2097152 totalling 2.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 4 Chunks of size 2359296 totalling 9.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 4194304 totalling 4.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 2 Chunks of size 4718592 totalling 9.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 8388608 totalling 8.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 12 Chunks of size 9437184 totalling 108.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 16777216 totalling 16.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 3 Chunks of size 67108864 totalling 192.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 411041792 totalling 392.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 469762048 totalling 448.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 536870912 totalling 512.00MiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:700] Sum Total of in-use chunks: 1.66GiB
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:702] Stats:
Limit: 2300588851
InUse: 1787234560
MaxInUse: 1787234560
NumAllocs: 147
MaxAllocSize: 536870912

W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:274] xxxxxx******************************__________________
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\bfc_allocator.cc:275] Ran out of memory trying to allocate 392.00MiB. See logs for memory state.
W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Resource exhausted: OOM when allocating tensor with shape[25088,4096]
Traceback (most recent call last):
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 1022, in _do_call
return fn(*args)
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 1004, in _run_fn
status, run_metadata)
File "D:\python\program\lib\contextlib.py", line 66, in exit
next(self.gen)
File "D:\python\program\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096]
[[Node: vgg_16/fc6/weights/Initializer/random_uniform/mul = Mul[T=DT_FLOAT, _class=["loc:@vgg_16/fc6/weights"], _device="/job:localhost/replica:0/task:0/gpu:0"](vgg_16/fc6/weights/Initializer/random_uniform/RandomUniform, vgg_16/fc6/weights/Initializer/random_uniform/sub)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/python/project/Faster-RCNN-TensorFlow-Python3.5-master (2)/Faster-RCNN-TensorFlow-Python3.5-master/train.py", line 210, in
train.train()
File "D:/python/project/Faster-RCNN-TensorFlow-Python3.5-master (2)/Faster-RCNN-TensorFlow-Python3.5-master/train.py", line 120, in train
sess.run(tf.variables_initializer(variables, name='init'))
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 767, in run
run_metadata_ptr)
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "D:\python\program\lib\site-packages\tensorflow\python\client\session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096]
[[Node: vgg_16/fc6/weights/Initializer/random_uniform/mul = Mul[T=DT_FLOAT, _class=["loc:@vgg_16/fc6/weights"], _device="/job:localhost/replica:0/task:0/gpu:0"](vgg_16/fc6/weights/Initializer/random_uniform/RandomUniform, vgg_16/fc6/weights/Initializer/random_uniform/sub)]]

Caused by op 'vgg_16/fc6/weights/Initializer/random_uniform/mul', defined at:
File "D:/python/project/Faster-RCNN-TensorFlow-Python3.5-master (2)/Faster-RCNN-TensorFlow-Python3.5-master/train.py", line 210, in
train.train()
File "D:/python/project/Faster-RCNN-TensorFlow-Python3.5-master (2)/Faster-RCNN-TensorFlow-Python3.5-master/train.py", line 85, in train
layers = self.net.create_architecture(sess, "TRAIN", self.imdb.num_classes, tag='default')
File "D:\python\project\Faster-RCNN-TensorFlow-Python3.5-master (2)\Faster-RCNN-TensorFlow-Python3.5-master\lib\nets\network.py", line 295, in create_architecture
rois, cls_prob, bbox_pred = self.build_network(sess, training)
File "D:\python\project\Faster-RCNN-TensorFlow-Python3.5-master (2)\Faster-RCNN-TensorFlow-Python3.5-master\lib\nets\vgg16.py", line 39, in build_network
cls_score, cls_prob, bbox_pred = self.build_predictions(net, rois, is_training, initializer, initializer_bbox)
File "D:\python\project\Faster-RCNN-TensorFlow-Python3.5-master (2)\Faster-RCNN-TensorFlow-Python3.5-master\lib\nets\vgg16.py", line 166, in build_predictions
fc6 = slim.fully_connected(pool5_flat, 4096, scope='fc6')
File "D:\python\program\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 177, in func_with_args
return func(*args, **current_args)
File "D:\python\program\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1409, in fully_connected
outputs = layer.apply(inputs)
File "D:\python\program\lib\site-packages\tensorflow\python\layers\base.py", line 303, in apply
return self.call(inputs, **kwargs)
File "D:\python\program\lib\site-packages\tensorflow\python\layers\base.py", line 269, in call
self.build(input_shapes[0])
File "D:\python\program\lib\site-packages\tensorflow\python\layers\core.py", line 123, in build
trainable=True)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 988, in get_variable
custom_getter=custom_getter)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 890, in get_variable
custom_getter=custom_getter)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 341, in get_variable
validate_shape=validate_shape)
File "D:\python\program\lib\site-packages\tensorflow\python\layers\base.py", line 258, in variable_getter
variable_getter=functools.partial(getter, **kwargs))
File "D:\python\program\lib\site-packages\tensorflow\python\layers\base.py", line 208, in _add_variable
trainable=trainable and self.trainable)
File "D:\python\program\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1310, in layer_variable_getter
return _model_variable_getter(getter, *args, **kwargs)
File "D:\python\program\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1299, in _model_variable_getter
custom_getter=getter)
File "D:\python\program\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 177, in func_with_args
return func(*args, **current_args)
File "D:\python\program\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 268, in model_variable
partitioner=partitioner, custom_getter=custom_getter)
File "D:\python\program\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 177, in func_with_args
return func(*args, **current_args)
File "D:\python\program\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 225, in variable
partitioner=partitioner)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 333, in _true_getter
caching_device=caching_device, validate_shape=validate_shape)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 684, in _get_single_variable
validate_shape=validate_shape)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variables.py", line 197, in init
expected_shape=expected_shape)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variables.py", line 274, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 673, in
shape.as_list(), dtype=dtype, partition_info=partition_info)
File "D:\python\program\lib\site-packages\tensorflow\contrib\layers\python\layers\initializers.py", line 145, in _initializer
dtype, seed=seed)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\random_ops.py", line 245, in random_uniform
return math_ops.add(rnd * (maxval - minval), minval, name=name)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\math_ops.py", line 794, in binary_op_wrapper
return func(x, y, name=name)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1015, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "D:\python\program\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1625, in _mul
result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
File "D:\python\program\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 763, in apply_op
op_def=op_def)
File "D:\python\program\lib\site-packages\tensorflow\python\framework\ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "D:\python\program\lib\site-packages\tensorflow\python\framework\ops.py", line 1226, in init
self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[25088,4096]
[[Node: vgg_16/fc6/weights/Initializer/random_uniform/mul = Mul[T=DT_FLOAT, _class=["loc:@vgg_16/fc6/weights"], _device="/job:localhost/replica:0/task:0/gpu:0"](vgg_16/fc6/weights/Initializer/random_uniform/RandomUniform, vgg_16/fc6/weights/Initializer/random_uniform/sub)]]

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_gpu_executor.cc:637] Deallocating stream with pending work

Process finished with exit code 1