About filtering duplicate relations for VG dataset

[coldmanck]

It seems to me the following code snippet doesn't work as expected:

Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/data/datasets/visual_genome.py

Lines 148 to 156 in d0ffa40

	if self.filter_duplicate_rels:
	# Filter out dupes!
	assert self.split == 'train'
	old_size = relation.shape[0]
	all_rel_sets = defaultdict(list)
	for (o0, o1, r) in relation:
	all_rel_sets[(o0, o1)].append(r)
	relation = [(k[0], k[1], np.random.choice(v)) for k,v in all_rel_sets.items()]
	relation = np.array(relation, dtype=np.int32)

I was thinking filtering out duplicate relations means for those exactly repeated relation triplets (i.e., not only subject and object are the same but also the predicate); however, this snippet seems to preserve only a single predicate for each object pair (with a higher chance for those occurring more times to be chosen). This seems unreasonable for me and makes the following snippet redundant:

Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/data/datasets/visual_genome.py

Lines 162 to 164 in d0ffa40

	if relation_map[int(relation[i,0]), int(relation[i,1])] > 0:
	if (random.random() > 0.5):
	relation_map[int(relation[i,0]), int(relation[i,1])] = int(relation[i,2])

To accommodate multiple labels for each object pair, I think we have to change L148-L156 to the following:

if self.filter_duplicate_rels:
    # Filter out dupes!
    assert self.split == 'train'
    old_size = relation.shape[0]
    all_rel_sets = defaultdict(set)
    for (o0, o1, r) in relation:
        all_rel_sets[(o0, o1)].add(r)
    relation = [(k[0], k[1], v) for k, vs in all_rel_sets.items() for v in vs]
    relation = np.array(relation, dtype=np.int32)

[coldmanck]

Hi @KaihuaTang 麻烦花时间看下哈，我认为是个蛮明显的 Bug，我也提供修复 patch 开了 PR 了，帮助之后的人不犯相同错误

[KaihuaTang]

@coldmanck Thank you for your advice. I also doubted it when I saw these codes in the previous implementations. However, on second thought, I realize that the current SGG setting doesn't allow a pair of objects to take more than one predicates, because if a pair of objects have more than one predicate, its prediction will always be penalized by cross-entropy loss.

The current version is not an ideal implementation, but your alternative code also doesn't fit the multi-class setting unless you change the SGG to the multi-label classification task and replace all the CE losses.

[coldmanck]

Hi @KaihuaTang Thank you for your reply. Actually, the current version (of randomly sampling a predicate if a pair got multiple predicates; L162-164) is just nice. A recent NeurIPS 2020 paper [1] shows that, to optimize Recall@K given multilabel classification, for reduction of multilabel to binary/multi-class classification problem we should choose either (a) One-versus-all normalized, (b) Pick-all-labels normalized, or (c) Pick-one-label. Among them (c) is exactly what has been done by default in this SGG implementation. We don't necessarily need a multilabel loss (such as (a)); instead, a multiclass cross-entropy (such as (b) or (c)) also works! :)

By the way, my code fixes the bug at L148-156 and still keeps the default setting because of Line 162-164. My PR paves the right way for other people trying to extend into a multi-label setting.

[1] Menon, Aditya K., et al. "Multilabel reductions: what is my loss optimising?." Advances in Neural Information Processing Systems 32 (2019): 10600-10611.