How to achieve the high performance?

[ivy94419]

I have trained the fc_model for 30 epochs, and the hyper-parameters are set the same as the default.
But when testing, I only achieve Bleu-4 = 0.25 in my 1000 validation images in COCO, but I used your provided pre-trained model and achieved Bleu-4=0.324

Bleu_4 0.25786956526235716
METEOR 0.221778680264552
CIDEr 0.785944241965
ROUGE_L 0.494435436025
Bleu_2 0.5063064959709507
Bleu_1 0.6833226219296441
Bleu_3 0.3629853091429978

How to reproduce the high performance as you achieved?

[ruotianluo]

Use the full 5000 val images?

[ivy94419]

@ruotianluo No，for simplify, I selected a part of 1000 val images and test them on your pretrained model and my model. The gap of two performance is huge.

[ruotianluo]

I'm not sure. I haven't touched fc model for a while. Have you tried attention-based model?

[ivy94419]

@ruotianluo You mean show_attend_tell model? I haven't try it. I will try it now.
All parameters are set as default, but some parameters I still need to confirm, batchsize=10, ss_prob=0.0, att_size=7 (Since att_size=14 results a 368GB feats_att.h5, too big)

[ruotianluo]

Try att2in2. att_size 7 is fine.

[ivy94419]

@ruotianluo Training att2in2 model also follow the the training method? I mean I only need to run train.py and do not need any steps during training process?

[ruotianluo]

Yes. Just change the caption_model arg

[ivy94419]

@ruotianluo Thanks, I will try it now, reply to you soon.

[ivy94419]

@ruotianluo
I have trained att2in2 for 14 epochs for now. But the performance is not inspiring.
Epoch 11 model test result

Epoch 13 model test result

Epoch 13 validation loss

Epoch 14 model test result

It seems hard to more improve space.
But I achieved the following scores on att2in2 pretrained model on the same 1000 images

[ruotianluo]

To be honest, I have no idea.