Memory cost too much

[SuperWWF]

Hi, I try to learn to train pspnet on cityscapes(only train dataset), but after several iters , always ResourceExhaustedError, I try to reduce batch_size and train_size from 864 to 512.and I use
4*TITAN-XP(12G). Could you give to some advice? Thank you so much!
python ./run.py --network 'resnet_v1_101' --visible_gpus '0,1,2,3' --reader_method 'queue' --batch_size 4 --poly_lr 1 --lrn_rate 0.01 --momentum 0.9 --weight_decay_mode 0 --weight_decay_rate 0.0001 --weight_decay_rate2 0.001 --database 'Cityscapes' --subsets_for_training 'train' --train_image_size 512 --snapshot 10000 --train_max_iter 50000 --test_image_size 512 --random_rotate 0 --fine_tune_filename './z_pretrained_weights/resnet_v1_101.ckpt'

[holyseven]

• try batch size 2 (total batch size 8), it decreases the performance but not much.
• try crop size 816, 768, 720 etc... in fact, train crop size and batch size affects each other.
• try resnet_v1_50.

[erichhhhho]

• try batch size 2 (total batch size 8), it decreases the performance but not much.
• try crop size 816, 768, 720 etc... in fact, train crop size and batch size affects each other.
• try resnet_v1_50.

Hi, I was wondering what is the effect of training on cropped training data(E.g. 720×720) and then testing on the larger images(E.g. 1024×2048). I saw the original implementation of PSPNet only train images in 713*713(cityscape) and then test the model in the full resolution images, is it correct? Thus, the reported mIoU inference result in here are in which image size exactly?

Also, are you loading the pretrained weight of res101 before training from scratch here?

Thank you

[holyseven]

Hi,

I saw the original implementation of PSPNet only train images in 713*713(cityscape) and then test the model in the full resolution images, is it correct? Thus, the reported mIoU inference result in here are in which image size exactly?

No... pspnet trains with 713713 crops and tests with 713713 crops, see this. and here 864*864 crops were used.

Also, are you loading the pretrained weight of res101 before training from scratch here?

Yes.

[erichhhhho]

@holyseven Thx for your reply.

But I saw the cityscape benchmark submission page saying that the reported mIoU are under full resolution, see here(As the given test set is 2048 x 1024 pixels )

• single zip archive
• maximum 100 MB
• Result files with filename "berlin_000123_000019*.ext" where ext is png for pixel-level/panoptic and txt for instance-level semantic labeling. The files can be in arbitrary sub folders for pixel- and instance-level.
• For panoptic, additionally a .json file in COCO panoptic segmentation format. The prediction pngs need to be in a subfolder that matches the .json filename without extension.
• Exactly one result file for each test image
• Result image size must be equal to the input image size, i.e. 2048 x 1024 pixels
• Labels must be encoded by labelIDs, not trainIDs, e.g. a car should have ID 26

[holyseven]

@erichhhhho One whole image of 2048x1024 is split into 713x713 crops with a certain way, see the code https://github.com/hszhao/PSPNet/blob/master/evaluation/scale_process.m,
https://github.com/hszhao/PSPNet/blob/master/evaluation/eval_sub.m#L48.

Similar operations are done here.