Resource exhausted: OOM when allocating tensor with shape[255,256,256,69] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

[Linamiao-1998]

I meet an error:

2022-01-30 10:37:36.925076: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 1074790400 memory_limit_: 3059430195 available bytes: 1984639795 curr_region_allocation_bytes_: 1073741824
2022-01-30 10:37:36.925383: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit: 3059430195
InUse: 621716992
MaxInUse: 621716992
NumAllocs: 1595
MaxAllocSize: 33554432

2022-01-30 10:37:36.925822: W tensorflow/core/common_runtime/bfc_allocator.cc:424] **********************************************************__________________________________________
2022-01-30 10:37:36.926021: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[255,256,256,69] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Could you please help me solve this problem

[bubbliiiing]

reduce batch_size