Open set inference without training?

[Colin97]

Thanks for the great work. It seems that the released model is trained on the ADE dataset. If we want to test on other text descriptions (classes), we must re-train the model? Is there any other way to do the openest inference without training? For example, can I directly utilize the pre-trained CLIP model to calculate the pixel-wise dot product? Do you have such kind of code support?

[Mark-Dou]

Hello! If I want to re-train the model on the ADE dataset, how can I organize it, I mean whether I need to download it or something else, It is because i can not run the train code successfully. Looking forward to your reply! Thanks a lot!

[raoyongming]

@Colin97 We have tried directly using the pre-trained CLIP models in our early experiments. Unfortunately, we find the locality of the pre-trained CLIP model is not very satisfactory (i.e., the local feature on the feature map may not perfectly reflect the semantic information of the corresponding region/patch). I found that it is necessary to tune the pre-trained parameters on the dense prediction tasks to align the feature map and the input image.

[raoyongming]

@Mark-Dou We have tested the code on our server. We have also seen other folks can run the train and inference code in other GitHub issues. Can you tell us the detailed error occurred in your runs?

[Mark-Dou]

@Mark-Dou We have tested the code on our server. We have also seen other folks can run the train and inference code in other GitHub issues. Can you tell us the detailed error occurred in your runs?

Traceback (most recent call last):
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/utils/registry.py", line 69, in build_from_cfg
return obj_cls(**args)
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/ade.py", line 87, in init
super(ADE20KDataset, self).init(
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/custom.py", line 129, in init
self.img_infos = self.load_annotations(self.img_dir, self.img_suffix,
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/custom.py", line 166, in load_annotations
for img in self.file_client.list_dir_or_file(
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/fileio/file_client.py", line 1172, in list_dir_or_file
yield from self.client.list_dir_or_file(dir_path, list_dir, list_file,
File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/fileio/file_client.py", line 692, in _list_dir_or_file
for entry in os.scandir(dir_path):
FileNotFoundError: [Errno 2] No such file or directory: 'data/ade/ADEChallengeData2016/images/training'

I am not familiar with mmcv structure, this error means should I download the ADE dataset and put it on the corresponding path? Thanks a lot!

[raoyongming]

@Mark-Dou Yes. Please refer to the mmseg document to prepare the datasets.

[zhengyuan-xie]

Thanks for your great work! I'm a newbie of DL and I wonder where is the code of segmentation loss referred in sec 3.4. It seems that loss_identity and loss_aux in denseclip.py do not fit the description of segmentation loss in paper. Thanks!