What is the experimental protocol for evaluating on ImageNet-based benchmark?

[aosokin]

Hi Joseph,
I'm a bit confused about the experimental protocol for the ImageNet-based benchmark used in your work and would be very grateful for clarification.
Specifically, when detecting a class (in some episode) do you detect this class on the query images corresponding only to the same class (option 1) or on all query images of the episode (option 2)?

The RepMet paper (the second paragraph of Section 5.2) makes me believe in option 2. File data/Imagenet_LOC/episodes/epi_inloc_in_domain_1_5_10_500.pkl (1-shot 5-way setting) that you kindly provided supports the same conclusion as all the query image of the episode are glued together to one list.

However, the code seems to support option 1.
Function test_model from few_shot_benchmark seems to detect a class only on the images corresponding to the same category query_images[cat]:

RepMet/fpn/few_shot_benchmark.py

Lines 668 to 693 in 9bdc3f2

	def test_model(perf_stats,epi_cats,query_images,cat_indices,roidb,d,sym_ext, arg_params, aux_params,epi_root,epi_num,epi_cats_names, display=0):
	for cat in epi_cats:
	if args.nqc>0 and args.nqc< len(query_images[cat]):
	q_images = query_images[cat][0:args.nqc]#random.sample(query_images[cat],args.nqc)
	else:
	q_images = query_images[cat]
	for nImg in q_images:
	img_fname = roidb[nImg]['image']
	gt_classes = roidb[nImg]['gt_classes']
	if display==1:
	img_cats = [epi_cats_names[i] for i in range(args.Nway) if epi_cats[i] in gt_classes]
	q_dets = run_detection(sym_ext, arg_params, aux_params, img_fname, img_cats[0], cat_indices, epi_cats_names, epi_root, nImg, epi_num)
	else:
	q_dets = run_detection(sym_ext, arg_params, aux_params, img_fname, '', cat_indices, [], epi_root, nImg, epi_num)
	gt_boxes = np.copy(roidb[nImg]['boxes'])
	# legacy from Pascal dataset
	gt_boxes_test = []
	gt_classes_test = []
	for gt_box, gt_class in zip(gt_boxes, gt_classes):
	if gt_class in epi_cats:
	gt_boxes_test += [gt_box]
	gt_classes_test += [gt_class]
	gt_classes_test = np.asarray(gt_classes_test)
	gt_boxes_test = np.asarray(gt_boxes_test)
	d = perf_stats.comp_epi_stats_m(d, q_dets, gt_boxes_test, gt_classes_test, epi_cats, args.ovthresh)
	return d

The code uses another file from your package:
output/RepMet_inloc_1shot_5way_10qpc_500epi_episodes.npz, which has all the query images separated by category.

Could you please clarify which protocol was used to obtain the results of the paper?

Best,
Anton

[jshtok]

Hi Anton, Thank you for your interest in our work. The code searches for objects of all classes in all query images. The query images are separated by category for convenience. I agree that the loop on the classes is a bit misleading. However, please notice that the variable 'cat', denoting the category the query belongs to, is not used. The predictions produced by the model are for all classes. For a simple check, replace the loop by your own where the query images are loaded in random. Regards, Joseph

…

On Tue, May 26, 2020 at 1:33 PM Anton Osokin ***@***.***> wrote: Hi Joseph, I'm a bit confused about the experimental protocol for the ImageNet-based benchmark used in your work and would be very grateful for clarification. Specifically, when detecting a class (in some episode) do you detect this class on the query images corresponding only to the same class (option 1) or on all query images of the episode (option 2)? The RepMet paper (the second paragraph of Section 5.2) makes me believe in option 2. File data/Imagenet_LOC/episodes/epi_inloc_in_domain_1_5_10_500.pkl (1-shot 5-way setting) that you kindly provided supports the same conclusion as all the query image of the episode are glued together to one list. However, the code seems to support option 1. Function test_model from few_shot_benchmark seems to detect a class only on the images corresponding to the same category query_images[cat]: https://github.com/jshtok/RepMet/blob/9bdc3f20ff08a8b3ce005af327aba6bf0bb71213/fpn/few_shot_benchmark.py#L668-L693 The code uses another file from your package: output/RepMet_inloc_1shot_5way_10qpc_500epi_episodes.npz, which has all the query images separated by category. Could you please clarify which protocol was used to obtain the results of the paper? Best, Anton — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOBU6RDDVZG32DCAGWIAP3RTOLHXANCNFSM4NKF4AJQ> .

[aosokin]

Joseph, thanks a lot for a quick reply!
I guess I just got lost in the code.
Best,
Anton