Question about the dataloader

[lailvlong]

Hello! Thanks for sharing your nice project.
I have some uncertainties about the data loader.

FPSG/src/datasets/modelnet.py

Lines 134 to 136 in 238b1de

	for eachFile in os.listdir(self.auxiliary_dir):
	if not eachFile.endswith('.txt'):
	continue

In my understanding, the above codes load the samples of all the categories as training data, including those for evaluation. Do i make a wrong comprehension？

[voidstrike]

@lailvlong Hey, thanks for your interest in our project!

The short answer to your question is Yes, but it doesn't violate the evaluation protocol.
If you look at

FPSG/src/datasets/modelnet.py

Lines 110 to 112 in 238b1de

	def __getitem__(self, index):
	img_path, pc_path = self.data_corpus[index].split('\t')
	data_instance_class = img_path.split('/')[-4]

You will see that we access the reference tensors (containing both TRAIN & TEST split) based on"data_corpus", which is generated in the constructor according to the input "config_path". Look back to the main training script
https://github.com/voidstrike/FPSG/blob/main/src/trainNetwork.py#L85-L90
you will see the "config_path" are indeed different.
Moreover, noted that the base classes and novel classes are mutually exclusive. We use all data from base classes to train the model and test the model on novel classes, it's not a standard 80/20 split.

[lailvlong]

@lailvlong Hey, thanks for your interest in our project!

The short answer to your question is Yes, but it doesn't violate the evaluation protocol. If you look at

FPSG/src/datasets/modelnet.py

Lines 110 to 112 in 238b1de

	def __getitem__(self, index):
	img_path, pc_path = self.data_corpus[index].split('\t')
	data_instance_class = img_path.split('/')[-4]

You will see that we access the reference tensors (containing both TRAIN & TEST split) based on"data_corpus", which is generated in the constructor according to the input "config_path". Look back to the main training script
https://github.com/voidstrike/FPSG/blob/main/src/trainNetwork.py#L85-L90
you will see the "config_path" are indeed different.
Moreover, noted that the base classes and novel classes are mutually exclusive. We use all data from base classes to train the model and test the model on novel classes, it's not a standard 80/20 split.

Thanks for your reply.
Yes, the "config_path" for the train and test dataloaders are different, but the “reference_path” of them are the same.

FPSG/src/trainNetwork.py

Lines 85 to 87 in 238b1de

	if opt.dataset == 'modelnet':
	ds = FewShotModelNet(config_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs)
	ds_test = FewShotModelNet(test_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs)

And the “reference_path” contains the files of both the base and novel categories. Thus, for the train dataloader, data of novel categories would also be loaded into "self.img_corpus" and "self.pc_corpus".

FPSG/src/datasets/modelnet.py

Lines 130 to 153 in 238b1de

	def _build_reference(self, ):
	assert self.auxiliary_dir is not None, 'Auxiliary folder is not generated yet!!!'
	tmp_img_list, tmp_pc_list = list(), list()
	for eachFile in os.listdir(self.auxiliary_dir):
	if not eachFile.endswith('.txt'):
	continue
	class_name = eachFile.split('.')[0].split('+')[1]
	# self.reference[class_name] = dict()
	print(f'Building Reference dataset for {class_name} ...')
	class_ds = FewShotSubModelNet(os.path.join(self.auxiliary_dir, eachFile), transform=self.tfs, tgt_transform=self.tgt_tfs)
	loader = DataLoader(class_ds, batch_size=len(class_ds), shuffle=False)
	for stacked_img, stacked_pc in loader:
	self.reference[class_name]['imgs'] = stacked_img
	self.reference[class_name]['pcs'] = stacked_pc
	tmp_img_list.append(stacked_img)
	tmp_pc_list.append(stacked_pc)
	break # Follow the protonet, only need one sample because batch_size equal to the dataset length
	self.img_corpus = torch.cat(tmp_img_list, dim=0)
	self.pc_corpus = torch.cat(tmp_pc_list, dim=0)

During iterating the train dataloader, data in the "self.img_corpus" and "self.pc_corpus" would be fetched as "ans[xad]" and "ans[pcad]", including those of novel categories.

FPSG/src/datasets/modelnet.py

Lines 110 to 126 in 238b1de

	def __getitem__(self, index):
	img_path, pc_path = self.data_corpus[index].split('\t')
	data_instance_class = img_path.split('/')[-4]
	query_matrix = {
	'class': data_instance_class,
	'img_data': self.reference[data_instance_class]['imgs'],
	'pc_data': self.reference[data_instance_class]['pcs'],
	}
	ans = extract_episode(self.n_support, self.n_query, query_matrix) # Original episode
	# tgt_path = self.reference[data_instance_class]['imgs'][ans['tmp']]
	# print(f'{index}: {tgt_path}')
	example_idx = torch.randperm(self.item_len)[:self.n_support] # Adding additional img-pc pairs to avoid model collapse
	ans['xad'] = self.img_corpus[example_idx]
	ans['pcad'] = self.pc_corpus[example_idx]

When computing the intra_support loss during training, there are two options, namely "option 2" and "option 1".

FPSG/src/models/few_shot.py

Lines 63 to 73 in 238b1de

	def loss(self, sample):
	# Gather input
	xs, xq, pcs, pcq = Variable(sample['xs']), Variable(sample['xq']), Variable(sample['pcs']), Variable(sample['pcq'])
	# NOTE: following 2 options are interchangeable
	# xad, pcad = xs, pcs # Option 1
	xad, pcad = Variable(sample['xad']), Variable(sample['pcad']) # Option 2
	return self._loss_single_class(xs, xq, xad, pcs, pcq, pcad)

The codes select "option 2", which uses the "ans[xad]" and "ans[pcad]" and may involves the data of the novel categories. This is the point that i feel puzzled.
In contrast, the "option 1" that uses the "ans[xs]" and "ans[pcs]" should have been adopted. Moreover, i have tried both "option 1" and "option 2". With my observation, in ModelNet dataset, the performance of "option 1" (avg_cd=7.+) are much worser than that of "option 2" (avg_cd=2.+). It is very weried that the two options have such different performances. Or maybe that "option 2" has used the evaluation data for training.

[voidstrike]

@lailvlong Oh my god, I see, that's definitely an information leak.

The reason we adopt "option 2" is we found that "option 1" will give us almost the same point cloud during the test phase. We thought it was a model collapse because the model observes one and only one class per episode, therefore, we tried to make some variants by sampling random (IMG, PC) pairs from the dataset (so each episode contains more than one class). We just use "randperm" and forgot "self.img_corpus" and "self.pc_corpus" contain evaluation points.

Thanks for your question and it shows that the problem is solved by information leak rather than bringing more classes in. Really sorry for the mistake.

Last but not least, could you please somehow try to modify the code such that keeps random sampling from "self.img_corpus" and "self.pc_corpus" but avoid information leak? This one is absolutely based on your interest and feel free to ignore it. Sorry again for the bug.

[lailvlong]

@voidstrike Yes, i also find that, during the test phase, the generated point clouds of different query images are very similar in an episode. In my opinion, it is not caused by model collapse but by that the support prototypes (class-specific prior) dominates the generation. Since reconstructing the shape of query image is much harder than restoring the shape of the support prototype (mean shape of the support point clouds). Meanwhile, restoring the shape of the support prototype can also offer a low loss. In this way, the model prefers to ignore the query image and output the same results.
To enable the model to consider more categories in one gradient descent step, maybe we can construct a mini-batch with several episodes. In your project, we can set "n_way>1 " and make other corresponding modifications.

[voidstrike]

@lailvlong A good perspective! Thanks for your investigation & suggestions, considering more categories in one gradient step definitely sounds promising and makes sense to me. Unfortunately, I already left the university and I'm no longer working in this area.
Please feel free to modify the code if you want and looking forward to your paper. :)

[BTAROCKET]

@lailvlong @voidstrike >
Hello! I get the same question. Could you please share the solution about it. I am also confused about how to test on base class. I have tried but got bad result which is about 10 times as long as the traing phase CD. Looking forward to your reply. Thank you!