Some doubts about pseudo labels

[Jianghui-Wang]

Hi, I am pseudo-tagging the imagenet-1k, and encountering some difficulties.

Firstly, I wonder what would happen if the classes of semeg are more than 255? How to use one channel depth png image to represent them? (Although COCO datasets is only 80 classes, the imagenet is more than 255 classes when fine-tuning)

Secondly, on the example of Colab notebook, the rgb2depth model of DPT could not input any size of imagenet pictures. How could we save all the pseudo labels down before the data augmentation cutting it into 224*224? We need to align the original images with the pseudo labeled image should we?

Thank you for any help.

[roman-bachmann]

Hi @Chianghui-Wong!

If you need to save more than 255 classes, you could consider saving them as 16 bit PNGs or just as a numpy array. I would recommend the PNG option, since semantic segmentation images are often highly compressible. To do that, you can simply convert a uint16 numpy array to a PIL.Image and save it as a PNG:
Image.fromarray(semseg.astype(np.uint16)).save('semseg.png')

Regarding saving depth images: DPT needs all images to have side lengths that are a multiple of 32 pixels. We therefore resize the image height and width to the closest multiples of 32, process them with the DPT and then resize them back to their original resolution. We also limit the minimum and maximum side lengths to be between 32 and 768 for the DPT input images. During MultiMAE pre-training, we select the same random crops from the RGB, depth and semseg images and resize each to 224x224.

Best, Roman

[Jianghui-Wang]

Hi @Chianghui-Wong!

If you need to save more than 255 classes, you could consider saving them as 16 bit PNGs or just as a numpy array. I would recommend the PNG option, since semantic segmentation images are often highly compressible. To do that, you can simply convert a uint16 numpy array to a PIL.Image and save it as a PNG: Image.fromarray(semseg.astype(np.uint16)).save('semseg.png')

Regarding saving depth images: DPT needs all images to have side lengths that are a multiple of 32 pixels. We therefore resize the image height and width to the closest multiples of 32, process them with the DPT and then resize them back to their original resolution. We also limit the minimum and maximum side lengths to be between 32 and 768 for the DPT input images. During MultiMAE pre-training, we select the same random crops from the RGB, depth and semseg images and resize each to 224x224.

Best, Roman

I’ve been stuck for so long! Your advice finally opened the door I needed to move forward.
Thank you for your time and guidance.❤️