Run out of memory

[jinyu121]

Well, I have a voc-like dataset with 7000 classes. So I use the following config:

    GPU_COUNT = 2
    IMAGES_PER_GPU = 1
    STEPS_PER_EPOCH = 150
    BASE_EPOCH = 10
    NUM_CLASSES = 1 + 7000  # WARNING: This dataset has  7000 classes
    MAX_GT_INSTANCES = 50
    POST_NMS_ROIS_TRAINING = 1000
    POST_NMS_ROIS_INFERENCE = 500
    DETECTION_MAX_INSTANCES = 50

Other config is just as the default config.

And~ I only take less than 15 objects in one image.

I run it on 2 Titan X, each of which has 12 GB memory. But still run out of memory during training:

41/150 [=======>......................] - ETA: 1:12 - loss: 3.6104 - rpn_class_loss: 0.0231 - rpn_bbox_loss: 0.7184 - mrcnn_class_loss: 1.5067 - mrcnn_bbox_loss: 0.6695 - mrcnn_mask_loss: 0.6922
42/150 [=======>......................] - ETA: 1:11 - loss: 3.6236 - rpn_class_loss: 0.0239 - rpn_bbox_loss: 0.7198 - mrcnn_class_loss: 1.5192 - mrcnn_bbox_loss: 0.6680 - mrcnn_mask_loss: 0.6921


2017-11-13 16:05:25.086033: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.62GiB.  Current allocation summary follows.
2017-11-13 16:05:25.086152: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): 	Total Chunks: 276, Chunks in use: 275. 69.0KiB allocated for chunks. 68.8KiB in use in bin. 11.9KiB client-requested in use in bin.
2017-11-13 16:05:25.086173: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): 	Total Chunks: 45, Chunks in use: 44. 23.2KiB allocated for chunks. 22.5KiB in use in bin. 22.1KiB client-requested in use in bin.
.....
.....

So, anyone can tell me how to prevent it?

---

OK~ set TRAIN_ROIS_PER_IMAGE = 32, goes well.