-
Notifications
You must be signed in to change notification settings - Fork 545
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
will you provide the training script? #12
Comments
same |
I am also regrading to the question. Thanks |
same |
Has anyone re-trained successfully? |
same |
1 similar comment
same |
Hi, for the training, the issues are mainly related to bn layer:
|
@hszhao : Thanks for your information. I am working in same cityscapes dataset. I am using 1 GPU TitanX Pascal. Is it possible to run your training model in my computer? If not, could you reduce the Resnet depth layer such as 54? I also a beginner of caffer, so I do not know how can I make the training model from your first point |
@hszhao I am training a 713 resolution pspnet on 2 x 12gb titan x with batch size 1, and it seems almost all memories are used. So I guess training with batchsize 16 would require about 32 titan x cards (12gb memory) ? I cannot find details about how many gpus are used in the paper, so I want to confirm that how many gpus are required to train with batchsize 16 according to your experience ? I really wonder what is the quantitative performance improvement between batchsize 16 and batchsize 1, because in the paper and this thread you emphasize that batchsize matters yet in deeplab-v2 (and according to my own experience) training with batchsize 1 also works (to some extent). Do I really need to use batchsize 16 (and potentially 32 cards ?) to achieve ideal performance ? ... |
@Fromandto If your batchsize is 1, the batch normalization layer may be not work. However, the bn layer seems important to the performance of PSPNet. |
@huaxinxiao yes, this is exactly what i am concerned ... but I just don't have 32 gpus (or is there anything wrong with my setting so that 4 gpus are enough to train 16 batch ?) |
@Fromandto Smaller crop size (<321) will work in 4 gpus. Besides, you should use OpenMPI-based Multi-GPU caffe to gather the bn parameters. |
@Fromandto Could you share your training script? |
@suhyung I am using the training script of deeplab-v2. it is compatible. |
@Fromandto @hszhao Could you tell me some details for training? I'm using deeplab-v2 caffe, and I'm ready to train a model with my own python script. But I don't have any proper initial weights for pspnet101-VOC2012.prototxt. I tried to use the initial parameters from deeplab-v2, the layer names are different. Should I train a network for pre-trained model on ImageNet by myself? |
@SoonminHwang you can either transfer the weights over by matcaffe/pycaffe, or you can replace the ResNet part of the PSPNet prototxt with the DeepLab version. |
@SoonminHwang Did you have init weights file? |
What init weights should we use for training the cityscapes model for pspnet? |
@huaxinxiao can you train this pspnet with SyncBN? |
I've implemented sync batch normalization in pure tensorflow, which makes possible to train and reproduce the performance of PSPNet: batch norm across GPUs. |
No description provided.
The text was updated successfully, but these errors were encountered: