How to caculate the scores on MSCOCO 1k test images?

[jamiechoi1995]

I have read some image-text matching papers but I still have no idea about how to select 1k images from the 5k test images, how can I get the ID list of the 1k test images?

[fartashf]

The 1K results we reported on the result of cross-validation over 5 folds of the test set. Similar to prior works and their open-source codes, we chose 5-folds sequentially without random permutation.

For implementation see here:

vsepp/evaluation.py

Lines 169 to 189 in 226688a

	else:
	# 5fold cross-validation, only for MSCOCO
	results = []
	for i in range(5):
	r, rt0 = i2t(img_embs[i * 5000:(i + 1) * 5000],
	cap_embs[i * 5000:(i + 1) *
	5000], measure=opt.measure,
	return_ranks=True)
	print("Image to text: %.1f, %.1f, %.1f, %.1f, %.1f" % r)
	ri, rti0 = t2i(img_embs[i * 5000:(i + 1) * 5000],
	cap_embs[i * 5000:(i + 1) *
	5000], measure=opt.measure,
	return_ranks=True)
	if i == 0:
	rt, rti = rt0, rti0
	print("Text to image: %.1f, %.1f, %.1f, %.1f, %.1f" % ri)
	ar = (r[0] + r[1] + r[2]) / 3
	ari = (ri[0] + ri[1] + ri[2]) / 3
	rsum = r[0] + r[1] + r[2] + ri[0] + ri[1] + ri[2]
	print("rsum: %.1f ar: %.1f ari: %.1f" % (rsum, ar, ari))
	results += [list(r) + list(ri) + [ar, ari, rsum]]

[chinmay5]

Just to confirm, we evaluate on the 5 folds and report the best result on one of these samples(the best performing 1k sample) or we report the average value for all 5 sets?

[fartashf]

See #26