Background Data Missing

[z7r7y7]

Thank you for providing the code！As shown below, I encountered an issue when training the model with LMO data.

Traceback (most recent call last):
File "/home/zhongry/code/poet/main.py", line 394, in
main(args)
File "/home/zhongry/code/poet/main.py", line 338, in main
train_stats = train_one_epoch(
File "/home/zhongry/code/poet/engine.py", line 49, in train_one_epoch
prefetcher = data_prefetcher(data_loader, device, prefetch=True)
File "/home/zhongry/code/poet/data_utils/data_prefetcher.py", line 29, in init
self.preload()
File "/home/zhongry/code/poet/data_utils/data_prefetcher.py", line 33, in preload
self.next_samples, self.next_targets = next(self.loader)
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in next
data = self._next_data()
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
return self._process_data(data)
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
data.reraise()
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zhongry/anaconda3/envs/ffbpy38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zhongry/code/poet/data_utils/pose_dataset.py", line 62, in getitem
img, target = super(PoseDataset, self).getitem(idx)
File "/home/zhongry/code/poet/data_utils/torchvision_datasets/coco.py", line 136, in getitem
background_img = self.get_background(img.size)
File "/home/zhongry/code/poet/data_utils/torchvision_datasets/coco.py", line 84, in get_background
n_background_images = len(self.synthetic_background)
TypeError: object of type 'NoneType' has no len()

I believe the issue arises from not properly loading the background data, with the default value of 'None' for 'synt_background' in the main.py file.
parser.add_argument('--synt_background', default=None, type=str, help="Directory containing the background images from which to sample")
I noticed in the documentation that 'class 0' is referred to as the background class, but I'm unsure if this represents the background data itself. How should the path for the background data be configured? Or could you please clarify if the background data needs to be downloaded separately?

Looking forward to your response. Thank you very much.

[tgjantos]

Dear @z7r7y7 ,

you are confusing two things. The argument synt_background refers to images that you can sample from for synthetic images to replace their background. If you are working with the LMO dataset you have images that are synthetically rendered and thus have no background, i.e. just a black background. For the sake of data augmentation we allowed for the data sampler to sample an image and use it as a background instead of a blank background.

To summarize, the background data for synt_background should be a directory containing random images from which you can sample. For example, I use the validation set of the MSCOCO dataset.

However, you found a bug where it tries to sample from the background images, but it can't if there none provided. I will fix it immediately to raise a warning in that case.

Best,
Thomas

[z7r7y7]

Thank you very much for your response, and the issue regarding the background has been resolved!

While running the code for the object detection model using Mask R-CNN, I encountered another problem in the "poet/models/backbone_maskrcnn.py" file, specifically in lines 98-112, where the variable "img_predictions" is generated containing information about the bounding boxes, labels, and scores.
I noticed that in lines 101-106:
if self.obj_id_map is not None: if cls.item() in self.obj_id_map.keys(): new_cls = self.obj_id_map[cls.item()] box = torch.hstack((box, torch.tensor(new_cls, dtype=torch.float32, device=device))) else: box = torch.hstack((box, cls)) img_predictions.append(box)

when adding the label information (cls) to the boxes, if the "obj_id_map" is not empty but the label does not belong to the "obj_id_map", resulting in a 1x5-dimensional "box" without label information, while in other cases, it is a 1x6-dimensional "box" with label information. This dimension mismatch can lead to an error in torch.hstack in line 112:
img_predictions = torch.stack(img_predictions)
Therefore, I think it may be necessary to add a conditional statement to collect only the box information with dimensions of 1x6.

[tgjantos]

You are definitely right. The model might predict objects that are not in the object_id_map.

With the most recent commit, I implemented it as a skip if the detected object is not of "interest".

Hope it works now and thank you for raising this issue!

Best,
Thomas

[tgjantos]

Closed due to inactivity. Feel free to open the issue again.