Railroad is not a Train: Saliency as Pseudo-pxiel Supervision for Weakly Supervised Semantic Segmentation (CVPR 2021)

CVPR 2021 paper

Seungho Lee^1,* , Minhyun Lee^1,*, Jongwuk Lee², Hyunjung Shim¹

_{* indicates an equal contribution}

¹ _{School of Integrated Technology, Yonsei University}
² _{Department of Computer Science of Engineering, Sungkyunkwan University}

Introduction

Existing studies in weakly-supervised semantic segmentation (WSSS) using image-level weak supervision have several limitations: sparse object coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects. To overcome these challenges, we propose a novel framework, namely Explicit Pseudo-pixel Supervision (EPS), which learns from pixel-level feedback by combining two weak supervisions; the image-level label provides the object identity via the localization map and the saliency map from the off-the-shelf saliency detection model offers rich boundaries. We devise a joint training strategy to fully utilize the complementary relationship between both information. Our method can obtain accurate object boundaries and discard co-occurring pixels, thereby significantly improving the quality of pseudo-masks.

Updates

12 Jul, 2021: Initial upload

19 Aug, 2021: Minor update on information about dCRF and the pre-trained model of the segmentation networks

• Please see the issuses: dCRF and pre-trained model

28 Aug, 2021: Major updates about MS-COCO 2014 dataset and minor updates (cleanup)

15 Apr, 2022: Minor update on information about the method setting up 'cls_labels.npy' the for ms-coco 17 dataset

• Please see the issue: coco17

22 Feb, 2023: Minor update on the download link for coco dataset (Masks, Saliency maps)

Installation

• Python 3.6
• Pytorch >= 1.0.0
• Torchvision >= 0.2.2
• MXNet
• Pillow
• opencv-python (opencv for Python)

Execution

Dataset & pretrained model

• PASCAL VOC 2012

  • Images
  • Saliency maps using PFAN

• MS-COCO 2014

  • Images
  • Saliency maps using PFAN
  • Segmentation masks

• Pretrained models

  • ImageNet-pretrained Model for ResNet38

• MS-COCO 2017

  • The way to setup "cls_labels.npy" for MS-COCO 2017 dataset. This work is done by JimmyMa99 and please see the codes.

Classification network

• Execute the bash file for training, inference and evaluation.

  # Please see these files for the detail of execution.
  
  # PASCAL VOC 2012 
  # Baseline
  bash script/vo12_cls.sh
  # EPS
  bash script/voc12_eps.sh
  
  # MS-COCO 2014
  # Baseline
  bash script/coco_cls.sh
  # EPS
  bash script/coco_eps.sh  

• We provide checkpoints, training logs, and performances for each method and each dataset.

  Please see the details from the script files.

  Dataset	METHOD	Train(mIoU)	Checkpoint	Training log
  PASCAL VOC 2012	Base	47.05	Download	voc12_cls.log
  PASCAL VOC 2012	EPS	69.22	Download	voc12_eps.log
  MS-COCO 2014	Base	31.23	Download	coco_cls.log
  MS-COCO 2014	EPS	37.15	Download	coco_eps.log

• dCRF hyper-parameters

  • We did not use dCRF for our pseudo-masks, but only used for the comparision in the paper.
  • We chose the hyper-parameters for dCRF used in ResNet101-based DeepLabV2 among other candidates(OAA, and PSA)
  • Please see the official deeplab website for information

  CRF parameters: bi_w = 4, bi_xy_std = 67, bi_rgb_std = 3, pos_w = 3, pos_xy_std = 1.
  

Segmentation network

• We utilize DeepLab-V2 for the segmentation network.
• Please see deeplab-pytorch for the implementation in PyTorch.
• We used the pretrained model for VGG16 based network from DeepLab official and for ResNet101-based network from OAA official.

Results

Acknowledgement

This code is highly borrowed from PSA. Thanks to Jiwoon, Ahn.