Reproducing validation results

[JHLee0513]

Hello,

Thanks for the great work! I was interested in reproducing the transformer network with frozen GPT-2, and achieved slightly lower performance on COCO so far:

Metric	reported	reproduced
Bleu@4	33.53	31.0
METEOR	27.45	27.1
CIDEr	113.08	105.7
SPICE	21.05	20.4

I was wondering if the provided code should be able to reproduce the validation scores or if I am missing something?

[rm-rf-me]

Hello, I am also going to try to train and test this model recently, but I am not very familiar with these evaluation indicators. Do you mainly rewrite the code when you are doing evaluation or refer to some external libraries?

[JHLee0513]

Hi @rm-rf-me , we used COCO caption evaluation tool for our evaluation to get the above numbers

[rm-rf-me]

got it, thanks a lot !

[rmokady]

Hi @JHLee0513,
We have not validated that the code reproduce the exact numbers, but we did publish the model weights so you could check your evaluation script.
Please also note we have used Karpathy et el. script and the evaluation code of Oscar.

Is it possible that a bug or slightly different hyperparameters were inserted when publishing the code.
Sorry for not able to provide a solution to this issue.

[JHLee0513]

Hi @rmokady,
Thanks for the reply, I believe that's a good idea. It seems like there's only the pretrained model on resnet encoder, is there also one with ViT?

[rmokady]

Sorry, at this point we are not planning to release additional models

[JHLee0513]

Got it, thanks for the info!