OOM on GTX 1080

[matthowland]

Similar to the issue #9 I'm hitting a ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[20,65536,64]

This is with GTX1080 card (8gb, 7.4GB available to TF) Cuda 8, CUDNN 5. Tryed training with a smaller image (100k and a small style 50k), will add logs below.

[matthowland]

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
Train set has been trimmed slightly..
(1, 652, 512, 3)
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.66GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
UID: 5
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 2.81GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.15GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.15GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 2.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512): Total Chunks: 1, Chunks in use: 0 768B allocated for chunks. 160B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048): Total Chunks: 1, Chunks in use: 0 3.2KiB allocated for chunks. 160B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152): Total Chunks: 1, Chunks in use: 0 2.47MiB allocated for chunks. 3.6KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608): Total Chunks: 2, Chunks in use: 0 26.71MiB allocated for chunks. 20.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432): Total Chunks: 2, Chunks in use: 0 120.00MiB allocated for chunks. 50.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864): Total Chunks: 2, Chunks in use: 0 160.00MiB allocated for chunks. 120.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728): Total Chunks: 1, Chunks in use: 0 160.00MiB allocated for chunks. 160.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 320.00MiB was 256.00MiB, Chunk State:
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000000 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000500 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b000f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b001800 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b009200 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b01b200 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b063200 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b0f3200 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b0fac00 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b102600 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b10a000 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b111f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b112000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b112100 of size 6912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b113c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b113d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b113e00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b114000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b114200 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b114600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b114a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b114e00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b115200 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b115a00 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b116a00 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b117200 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b117a00 of size 2048
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b118f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b119a00 of size 116224
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b136600 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b148600 of size 147456
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b16c600 of size 368640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c6600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c6800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c6a00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c6c00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c6e00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7a00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7c00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b1c7e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b257e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b2e7e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b377e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b407e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b497e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b527e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b5b7e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b647e00 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b68fe00 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b6d7e00 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b767e00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768a00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768c00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b768e00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769a00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769c00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b769e00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76a900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b76aa00 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b772c00 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b77a600 of size 31232
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b782000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b782100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b782200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b782300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b782400 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b794400 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7a6400 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7b8400 of size 73728
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7ca400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7ca500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7ca600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7ca700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b7ca800 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b812800 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b85a800 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8a2800 of size 294912
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8ea800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8eaa00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8eac00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8eae00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b8eb000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020b97b000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020ba0b000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020ba9b000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bb2b000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bbbb000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bc4b000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bc4b200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bc4b400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bc4b600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bc4b800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bcdb800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bd6b800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bd6ba00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bd6bc00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bd6be00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bd6c000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bdfc000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020be8c000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020be8c200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020be8c400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020be8c600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020be8c800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bf1c800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bfac800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bfaca00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bfacc00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bface00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020bfad000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c03d000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c0cd000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c0cd200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c0cd400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c0cd600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c0cd800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c15d800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c1ed800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c1eda00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c1edc00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c1ede00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c1ee000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c27e000 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c30e000 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c30e200 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c30e400 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c30e600 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c30e800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c39e800 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c42e800 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c42ea00 of size 512
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c42ec00 of size 989696
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c520600 of size 589824
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c5b0600 of size 1179648
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c6d0600 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020c910600 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020cb50600 of size 2359296
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020cd90600 of size 4718592
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020d210600 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020db10600 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020e410600 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020ed10600 of size 9437184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f610600 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f614600 of size 65536
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f624600 of size 262144
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f664600 of size 1048576
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f764600 of size 1048576
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f864f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f865f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1020f866d00 of size 15728896
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10210766e00 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211666e00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211669600 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021166be00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021166e600 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211670e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211670f00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211673700 of size 12544
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211676800 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211679000 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021167b800 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021167e000 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211680800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211680900 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211683100 of size 12544
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211686200 of size 19456
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168ae00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168af00 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168b900 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168c300 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168cd00 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168d700 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168e100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168e200 of size 5632
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021168f800 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211690200 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211691600 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211692a00 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211693e00 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211695200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211695300 of size 11264
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211697f00 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211699300 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021169bb00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021169e300 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116a0b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116a0c00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116a3400 of size 12544
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116a6500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116a8d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116ab500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116add00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116b0500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116b2d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116b5500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116b7d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116ba500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116bcd00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116bf500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116c1d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116c4500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116c6d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116c9500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116cbd00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116ce500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116d0d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116d3500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116d5d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116d8500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116dad00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116dd500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116dfd00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116e2500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116e4d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116e7500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116e9d00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116ec500 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116ed900 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116eed00 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f0100 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f1500 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f3d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f3e00 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f4800 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f5200 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f5c00 of size 3584
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f6a00 of size 5120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f7e00 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f8d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f9100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102116f9f00 of size 2560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211972200 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974e00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211974f00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211975000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211975100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211975200 of size 8192
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211977200 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211979a00 of size 12800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021197cc00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021197f400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211981c00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211984400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211986c00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211989400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021198bc00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021198e400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211990c00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211993400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211995c00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10211998400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021199ac00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021199d400 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021199fc00 of size 10240
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10212566e00 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1021c566e00 of size 183532288
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1022746e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10229c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1022c46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1022ec6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1023146e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10233c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1023646e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10238c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1023b46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1023dc6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1024046e900 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1024546e900 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1024a46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1024cc6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1024f46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10251c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1025446e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10256c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1025946e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1025bc6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1025e46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10260c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1026346e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10265c6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1026846e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1026ac6e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1026d46e900 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1026fc6e900 of size 44574720
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102726f1100 of size 81261312
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10277470400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1027c470400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10281470400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1028b470400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10295470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10297c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1029a470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1029cc70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1029f470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102a1c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102a4470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102a6c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102a9470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102abc70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102ae470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102b0c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102b3470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102b5c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102b8470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102bac70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102bd470400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102be370400 of size 26214400
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102bfc70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102c2470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102c4c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102c9c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102cec70400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102cfb70400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102d0a70400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102d2861400 of size 21032960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102d3c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102d8c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102ddc70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102e7c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102f1c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102f6c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102f9470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x102fbc70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10305c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1030fc70400 of size 335544320
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10323c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1032dc70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10337c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1034bc70400 of size 335544320
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1035fc70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10369c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10373c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10378c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1037dc70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10382c70400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10389070400 of size 20971520
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1038a470400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1038cc70400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x1038db70400 of size 15728640
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10392670400 of size 62914560
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10396270400 of size 41943040
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x10398a70400 of size 125829120
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x103a5270400 of size 335544320
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x103b9270400 of size 599011328
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x102116f8e00 of size 768
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x102116f9200 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x102116fa900 of size 2586880
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x102119a2400 of size 12339712
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x102d1970400 of size 15667200
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x10341c70400 of size 167772160
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x10364c70400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x10385470400 of size 62914560
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x1038ea70400 of size 62914560
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x103a0270400 of size 83886080
I tensorflow/core/common_runtime/bfc_allocator.cc:689] Summary of in-use Chunks by size:
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 138 Chunks of size 256 totalling 34.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 70 Chunks of size 512 totalling 35.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 1024 totalling 4.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 2 Chunks of size 1280 totalling 2.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 2048 totalling 10.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 11 Chunks of size 2560 totalling 27.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 3584 totalling 3.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 10 Chunks of size 5120 totalling 50.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 5632 totalling 5.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 6912 totalling 6.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 8192 totalling 8.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 60 Chunks of size 10240 totalling 600.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 11264 totalling 11.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 12544 totalling 36.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 12800 totalling 12.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 16384 totalling 16.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 19456 totalling 19.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 8 Chunks of size 31232 totalling 244.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 65536 totalling 64.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 6 Chunks of size 73728 totalling 432.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 116224 totalling 113.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 147456 totalling 144.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 262144 totalling 256.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 7 Chunks of size 294912 totalling 1.97MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 368640 totalling 360.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 31 Chunks of size 589824 totalling 17.44MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 989696 totalling 966.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 2 Chunks of size 1048576 totalling 2.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 1179648 totalling 1.12MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 2359296 totalling 6.75MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 4718592 totalling 4.50MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 9437184 totalling 36.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 7 Chunks of size 15728640 totalling 105.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 15728896 totalling 15.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 20971520 totalling 20.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 21032960 totalling 20.06MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 26214400 totalling 25.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 48 Chunks of size 41943040 totalling 1.88GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 44574720 totalling 42.51MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 62914560 totalling 60.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 81261312 totalling 77.50MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 13 Chunks of size 83886080 totalling 1.02GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 125829120 totalling 120.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 11 Chunks of size 167772160 totalling 1.72GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 183532288 totalling 175.03MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 335544320 totalling 960.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 599011328 totalling 571.26MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 6.82GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats:
Limit: 7815770932
InUse: 7323799808
MaxInUse: 7815768832
NumAllocs: 1475
MaxAllocSize: 5157690368

W tensorflow/core/common_runtime/bfc_allocator.cc:270] ******************************************_______xxx
W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 320.00MiB. See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:968] Resource exhausted: OOM when allocating tensor with shape[20,65536,64]
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 1.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
Traceback (most recent call last):
File "style.py", line 166, in
main()
File "style.py", line 146, in main
for preds, losses, i, epoch in optimize(_args, **kwargs):
File "src/optimize.py", line 113, in optimize
train_step.run(feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1619, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3796, in _run_using_default_session
session.run(operation, feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[20,65536,64]
[[Node: gradients/transpose_grad/transpose = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/BatchMatMul_grad/tuple/control_dependency, gradients/transpose_grad/InvertPermutation)]]

Caused by op u'gradients/transpose_grad/transpose', defined at:
File "style.py", line 166, in
main()
File "style.py", line 146, in main
for preds, losses, i, epoch in optimize(_args, *_kwargs):
File "src/optimize.py", line 90, in optimize
train_step = tf.train.AdamOptimizer(learning_rate).minimize(loss)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 196, in minimize
grad_loss=grad_loss)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 253, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients.py", line 469, in gradients
in_grads = _AsList(grad_fn(op, *out_grads))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_grad.py", line 345, in _TransposeGrad
return [array_ops.transpose(grad, array_ops.invert_permutation(p)), None]
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1100, in transpose
ret = gen_array_ops.transpose(a, perm, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2903, in transpose
result = _op_def_lib.apply_op("Transpose", x=x, perm=perm, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in init
self._traceback = _extract_stack()

...which was originally created as op u'transpose', defined at:
File "style.py", line 166, in
main()
[elided 0 identical lines from previous traceback]
File "style.py", line 146, in main
for preds, losses, i, epoch in optimize(_args, *_kwargs):
File "src/optimize.py", line 73, in optimize
feats_T = tf.transpose(feats, perm=[0,2,1])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1100, in transpose
ret = gen_array_ops.transpose(a, perm, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2903, in transpose
result = _op_def_lib.apply_op("Transpose", x=x, perm=perm, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in init
self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[20,65536,64]
[[Node: gradients/transpose_grad/transpose = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/BatchMatMul_grad/tuple/control_dependency, gradients/transpose_grad/InvertPermutation)]]

[lengstrom]

What batch size are you using?

[matthowland]

Currently using 20

python style.py --style examples/style/the_scream.jpg --checkpoint-dir saver --test examples/content/thecity.jpg --test-dir test --content-weight 1.5e1 --checkpoint-iterations 1000 --batch-size 20

[matthowland]

Good callout! My bad I copied and pasted from the readme, should have taken a closer look. Modified to smaller batch and seems to be working. Sorry 'bout that.

[anishathalye]

Yeah, the batch size of 20 fits on the GTX Titan X, which has 12 GB of VRAM, but the GTX 1080 has 8 GB I think. A batch size of ~ 13 is probably the largest that will work.