RGB-D Salient Object Detection: A Survey

Authors: Tao Zhou, Deng-Ping Fan, Ming-Ming Cheng, Jianbing Shen, Ling Shao.

This is a survey to review related RGB-D SOD models along with benchmark datasets, and provide a comprehensive evaluation for these models. We also collect related review papers for SOD and light field SOD models. If you have papers to recommend or any suggestions, please feel free to contact us.

🏃 🏃 🏃 KEEP UPDATING.

Fig.0: A brief chronology of RGB-D based SOD. The first early RGB-D based SOD work was the DM model, proposed in 2012. Deep learning techniques have been widely applied to RGB-D based SOD since 2017. More details can be found in our paper.

---

---

Content:

1. Related Reviews and Surveys to SOD
2. RGB-D SOD Models
3. RGB-D SOD Datasets
4. Light Field SOD
   1. LF SOD Models
   2. LF Datasets
5. Evaluation
   1. Overall Evaluation
   2. Attribute-based Evaluation
6. RGB-D SOD Benchmark
7. Citation

---

---

Related Reviews and Surveys to SOD:

No.	Year	Pub.	Title	Links
01	2015	IEEE TIP	Salient object detection: A benchmark	Paper/Project
02	2018	IEEE TCSVT	Review of visual saliency detection with comprehensive information	Paper/Project
03	2018	ACM TIST	A review of co-saliency detection algorithms: Fundamentals, applications, and challenges	Paper/Project
04	2018	IEEE TSP	Advanced deep-learning techniques for salient and category-specific object detection: A survey	Paper/Project
05	2018	IJCV	Attentive systems: A survey	Paper/Project
06	2018	ECCV	Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground	Paper/Project
07	2019	CVM	Salient object detection: A survey	Paper/Project
08	2019	IEEE TNNLS	Object detection with deep learning: A review	Paper/Project
09	2019	arXiv	Salient Object Detection in the Deep Learning Era: An In-Depth Survey	Paper/Project

---

---

RGB-D SOD Models:

🔥🔥🔥Update (in 2020-08-28)

No.	Year	Model	Pub.	Title	Links
🔥 97	2020	JL-DCF	arXiv (CVPR extension)	Siamese Network for RGB-D Salient Object Detection and Beyond	Paper/Project
🔥 96	2020	MMNet	ACM MM	MMNet: Multi-Stage and Multi-Scale Fusion Network for RGB-D Salient Object Detection	Paper/Project
95	2020	DASNet	ACM MM	Is depth really necessary for salient object detection?	Paper/Project
94	2020	FRDT	ACM MM	Feature Reintegration over Differential Treatment: A Top-down and Adaptive Fusion Network for RGB-D Salient Object Detection	Paper/Project
93	2020	HANet	Appl. Sci.	Hybrid‐Attention Network for RGB‐D Salient Object Detection	Paper/Project
92	2020	DQSD	IEEE TIP	Depth Quality Aware Salient Object Detection	Paper/Project
91	2020	DQAM	arXiv	Knowing Depth Quality In Advance: A Depth Quality Assessment Method For RGB-D Salient Object Detection	Paper/Project
90	2020	SDSFNet	IEEE TIP	Improved Saliency Detection in RGB-D Images Using Two-phase Depth Estimation and Selective Deep Fusion	Paper/Project
89	2020	ERLF	IEEE TIP	Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection	Paper/Project
88	2020	MCINet	arXiv	MCINet: Multi-level Cross-modal Interaction Network for RGB-D Salient Object Detection	Paper/Project
87	2020	PGAR	ECCV	Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection	Paper/Project
86	2020	ATSA	ECCV	Asymmetric Two-Stream Architecture for Accurate RGB-D Saliency Detection	Paper/Project
85	2020	BBS-Net	ECCV	BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network	Paper/Project
84	2020	CoNet	ECCV	Accurate RGB-D Salient Object Detection via Collaborative Learning	Paper/Project
83	2020	DANet	ECCV	A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection	Paper/Project
82	2020	CMMS	ECCV	RGB-D salient object detection with cross-modality modulation and selection	Paper/Project
81	2020	CAS-GNN	ECCV	Cascade graph neural networks for RGB-D salient object detection	Paper/Project
80	2020	HDFNet	ECCV	Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection	Paper/Project
79	2020	CMWNet	ECCV	Cross-modal weighting network for RGB-D salient object detection	Paper/Project
78	2020	UC-Net	CVPR	UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders	Paper/Project
77	2020	S2MA	CVPR	Learning selective self-mutual attention for RGB-D saliency detection	Paper/Project
76	2020	SSF	CVPR	Select, supplement and focus for RGB-D saliency detection	Paper/Project
75	2020	A2dele	CVPR	A2dele: Adaptive and Attentive Depth Distiller for Efficient RGB-D Salient Object Detection	Paper/Project
74	2020	JL-DCF	CVPR	JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection	Paper/Project
73	2020	D3Net	IEEE TNNLS	Rethinking RGB-D salient object detection: models, datasets, and large-scale benchmarks	Paper/Project
72	2020	RGBS	MTAP	Salient object detection for RGB-D images by generative adversarial network	Paper/Project
71	2020	GFNe	IEEE SPL	GFNet: Gate fusion network with res2net for detecting salient objects in RGB-D images	Paper/Project
70	2020	SDF	IEEE TIP	Improved saliency detection in RGB-D images using two-phase depth estimation and selective deep fusion	Paper/Project
69	2020	ICNet	IEEE TIP	ICNet: Information Conversion Network for RGB-D Based Salient Object Detection	Paper/Project
68	2020	Triple-Net	IEEE SPL	Triple-complementary network for RGB-D salient object detection	Paper/Project
67	2020	ASIF-Net	IEEE TCYB	ASIF-Net: Attention steered interweave fusion network for RGB-D salient object detection	Paper/Project
66	2020	BiANet	IEEE TIP	Bilateral Attention Network for RGB-D Salient Object Detection	Paper/Project
65	2020	PGHF	IEEE Access	Multi-modal weights sharing and hierarchical feature fusion for rgbd salient object detection	Paper/Project
64	2020	cmSalGAN	IEEE TMM	cmSalGAN: RGB-D Salient Object Detection with Cross-View Generative Adversarial Networks	Paper/Project
63	2020	CoCNN	PR	CoCNN: RGB-D deep fusion for stereoscopic salient object detection	Paper/Project
62	2020	GFNet	Neurocomputing	A cross-modal adaptive gated fusion generative adversarial network for RGB-D salient object detection	Paper/Project
61	2020	AttNet	IVC	Attention-guided RGBD saliency detection using appearance information	Paper/Project
60	2020	SSDF	arXiv	Synergistic saliency and depth prediction for RGB-D saliency detection	Paper/Project
59	2020	DPANet	arXiv	DPANet: Depth Potentiality-Aware Gated Attention Network for RGB-D Salient Object Detection	Paper/Project
58	2019	DSD	JVCIR	Depth-aware saliency detection using convolutional neural networks	Paper/Project
57	2019	DMRA	ICCV	Depth-induced Multi-scale Recurrent Attention Network for Saliency Detection	Paper/Project
56	2019	CPFP	CVPR	Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection	Paper/Project
55	2019	EPM	IEEE Access	Co-saliency detection for rgbd images based on effective propagation mechanism	Paper/Project
54	2019	AFNet	IEEE Access	Adaptive Fusion for RGB-D Salient Object Detection	Paper/Project
53	2019	LSF	arXiv	CNN-based RGB-D Salient Object Detection: Learn, Select and Fuse	Paper/Project
52	2019	DGT	IEEE TCYB	Going from RGB to RGBD saliency: A depth-guided transformation model	Paper/Project
51	2019	DCMF	IEEE TCYB	Discriminative cross-modal transfer learning and densely cross-level feedback fusion for RGB-D salient object detection	Paper/Project
50	2019	TANet	IEEE TIP	Three-stream attention-aware network for RGB-D salient object detection	Paper/Project
49	2019	DCA	IEEE TIP	Saliency detection via depth-induced cellular automata on light field	Paper/Project
48	2019	MMCI	PR	Multi-modal fusion network with multi-scale multi-path and cross-modal interactions	Paper/Project
47	2019	PDNet	ICME	Prior-model guided depth-enhanced network for salient object detection	Paper/Project
46	2019	CAFM	IEEE TSMC	Global and Local-Contrast Guides Content-Aware Fusion for RGB-D Saliency Prediction	Paper/Project
45	2019	DIL	MTAP	Salient object segmentation based on depth-aware image layering	Paper/Project
44	2019	TSRN	ICIP	Two-stream refinement network for RGB-D saliency detection	Paper/Project
43	2019	MLF	SPL	RGB-D salient object detection by a CNN with multiple layers fusion	Paper/Project
42	2019	SSRC	Neurocomputing	Salient object detection for RGB-D image by single stream recurrent convolution neural network	Paper/Project
41	2018	CDB	Neurocomputing	Stereoscopic saliency model using contrast and depth-guided-background prior	Paper/Project
40	2018	ACCF	IROS	Attention-Aware Cross-Modal Cross-Level Fusion Network for RGB-D Salient Object Detection	Paper/Project
39	2018	SCDL	ICDSP	Rgbd salient object detection using spatially coherent deep learning framework	Paper/Project
38	2018	PCF	CVPR	Progressively complementarityaware fusion network for RGB-D salient object detection	Paper/Project
37	2018	CTMF	IEEE TCYB	CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion	Paper/Project
36	2018	ICS	IEEE TIP	Co-saliency detection for RGBD images based on multi-constraint feature matching and cross label propagation	Paper/Project
35	2018	HSCS	IEEE TMM	HSCS: Hierarchical sparsity based co-saliencydetection for RGBD images	Paper/Project
34	2017	ISC	SIVP	An integration of bottom-up and top-down salient cueson rgb-d data: saliency from objectness versus non-objectness	Paper/Project
33	2017	MCLP	IEEE TCYB	An iterative co-saliency framework for RGBD images	Paper/Project
32	2017	DF	IEEE TIP	RGBD Salient Object Detection via Deep Fusion	Paper/Project
31	2017	MDSF	IEEE TIP	Depth-Aware Salient Object Detection and Segmentation via Multiscale Discriminative Saliency Fusion and Bootstrap Learning	Paper/Project
30	2017	MFF	IEEE SPL	RGB-D saliency object detection via minimum barrier distance transformand saliency fusion	Paper/Project
29	2017	TPF	ICCVW	A Three-Pathway Psychobiological Framework of Salient Object Detection Using Stereoscopic Technology	Paper/Project
28	2017	CDCP	ICCVW	An innovative salient object detection using center-dark channel prior	Paper/Project
27	2017	BED	ICCVW	Learning RGB-D Salient Object Detection using background enclosure, depth contrast, and top-down features	Paper/Project
26	2017	MFLN	ICCVS	RGB-D Saliency Detection by Multi-stream Late Fusion Network	Paper/Project
25	2017	M3Net	IROS	M3Net: Multi-scale multi-path multi-modal fusion network and example application to RGB-D salient object detection	Paper/Project
24	2017	HOSO	DICTA	HOSO: Histogram of Surface Orientation for RGB-D Salient Object Detection	Paper/Project
23	2016	GM	ACCV	Visual Saliency detection for RGB-D images with generative mode	Paper/Project
22	2016	DSF	ICASSP	Depth-aware saliency detection using discriminative saliency fusion	Paper/Project
21	2016	DCI	ICASSP	Saliency analysis based on depth contrast increased	Paper/Project
20	2016	BF	ICPR	RGB-D saliency detection under Bayesian framework	Paper/Project
19	2016	DCMC	IEEE SPL	Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion	Paper/Project
18	2016	SE	ICME	Salient object detection for RGB-D image via saliency evolution	Paper/Project
17	2016	LBE	CVPR	Local Background Enclosure for RGB-D Salient Object Detection	Paper/Project
16	2016	PRC	IEEE Access	Improving RGBD Saliency Detection Using Progressive Region Classification and Saliency Fusion	Paper/Project
15	2015	SF	CAC	Selective features for RGB-D saliency	Paper/Project
14	2015	MGMR	ICIP	RGB-D saliency detection via mutual guided manifold ranking	Paper/Project
13	2015	SRD	ICRA	Salient Regions Detection for Indoor Robots using RGB-D Data	Paper/Project
12	2015	DIC	TVC	Depth incorporating with color improves salient object detection	Paper/Project
11	2015	SFP	ICIMCS	Salient object detection in RGB-D image based on saliency fusion and propagation	Paper/Project
10	2015	GP	CVPRW	Exploiting global priors for RGB-D saliency detection	Paper/Project
09	2014	ACSD	ICIP	Depth saliency based on anisotropic center-surround difference	Paper/Project
08	2014	DESM	ICIMCS	Depth Enhanced Saliency Detection Method	Paper/Project
07	2014	LHM	ECCV	RGBD Salient Object Detection: A Benchmark and Algorithms	Paper/Project
06	2014	SRDS	ICDSP	Salient region detection for stereoscopic images	Paper/Project
05	2013	SOS	Neurocomputing	Depth really Matters: Improving Visual Salient Region Detection with Depth	Paper/Project
04	2013	RC	BMVC	Depth really Matters: Improving Visual Salient Region Detection with Depth	Paper/Project
03	2013	LS	BMVC	An In Depth View of Saliency	Paper/Project
02	2012	RCM	ICCSE	Depth combined saliency detection based on region contrast model	Paper/Project
01	2012	DM	ECCV	Depth matters: Influence of depth cues on visual saliency	Paper/Project

---

---

RGB-D SOD Datasets:

No.	Dataset	Year	Pub.	Size	#Obj.	Types	Resolution	Download
1	STERE	2012	CVPR	1000	~One	Internet	[251-1200] * [222-900]	link
2	GIT	2013	BMVC	80	Multiple	Home environment	640 * 480	link
3	DES	2014	ICIMCS	135	One	Indoor	640 * 480	link
4	NLPR	2014	ECCV	1000	Multiple	Indoor/outdoor	640 * 480, 480 * 640	link
5	LFSD	2014	CVPR	100	One	Indoor/outdoor	360 * 360	link
6	NJUD	2014	ICIP	1985	~One	Moive/internet/photo	[231-1213] * [274-828]	link
7	SSD	2017	ICCVW	80	Multiple	Movies	960 *1080	link
8	DUT-RGBD	2019	ICCV	1200	Multiple	Indoor/outdoor	400 * 600	link
9	SIP	2020	TNNLS	929	Multiple	Person in wild	992 * 774	link

---

---

Light Field SOD:

LF SOD Models:

No.	Year	Model	Pub.	Title	Links
01	2014	LFS	CVPR	Saliency detection on light field	Paper/Project
02	2015	WSC	CVPR	A weighted sparse coding framework for saliency detection	Paper/Project
03	2015	DILF	IJCAI	Saliency detection with a deeper investigationof light field	Paper/Project
04	2016	RL	ICASSP	Relative location for light field saliency detection	Paper/Project
05	2017	MA	ACM TOMM	Saliency detection on light field: A multi-cue approach	Paper/Project
06	2017	BIF	NPL	A two-stage bayesian integration framework for salient object detection on light field	Paper/Project
07	2017	LFS	IEEE TPAMI	Saliency Detection on Light Field	Paper/Project
08	2017	RLM	ICIVC	Saliency detection with relative location measure in light field image	Paper/Project
09	2017	SGDC	CVPR	Salience guided depth calibration for perceptually optimized compressive light field 3D display	Paper/Project
10	2018	DCA	FiO	Depth-induced cellular automata for light field saliency	Paper/Project
11	2019	DLLF	ICCV	Deep learning for light field saliency detection	Paper/Project
12	2019	DLSD	IJCAI	Deep light-field-driven saliency detection from a single view	Paper/Project
13	2019	Molf	NIPS	Memory-oriented decoder for light field salient object detection	Paper/Project
14	2020	ERNet	AAAI	Exploit and replace: An asymmetrical two-stream architecture for versatile light field saliency detection	Paper/Project
15	2020	DCA	IEEE TIP	Saliency detection via depth-induced cellular automata onlight field	Paper/Project
16	2020	RDFD	MTAP	Region-based depth feature descriptor for saliency detection light field	Paper/Project
17	2020	LFNet	IEEE TIP	LFNet light field fusion network for salient object detection	Paper/Project
18	2020	LFDCN	IEEE TIP	Light field saliency detection with deep convolutional networks	Paper/Project

LF Datasets:

No.	Dataset	Year	Pub.	Size	Description	Download
1	LFSD	2014	CVPR	100	It contains 60 indoor and 40 outdoor scenes, and most scenes consist of only one salient object	link
2	HFUT	2017	ACM TOMM	255	Most scenes contain multipleobjects that appear within different locations and scales under complex background clutter	link
3	HFUT	2017	ACM TOMM	255	Most scenes contain multipleobjects that appear within different locations and scales under complex background clutter	link
4	DUTLF-FS	2019	ICCV	1465	It contains several challenges, including lower contrast between salient objects and cluttered background, multiple disconnected salient objects, and dark or strong light conditions	link
5	DUTLF-MV	2019	IJCAI	1580	Each light field consists of multi-view images and a corresponding ground truth	link
6	Lytro Illum	2020	IEEE TIP	640	It includes several challenging factors, e.g., inconsistent illumi?nation conditions, and small salient objects existing in a similar or cluttered background	link

---

---

Evaluation:

Overall Evaluation:

Fig.1: A comprehensive evaluation for 24 representative RGB-D based SOD models, including LHM, ACSD, DESM, GP, LBE, DCMC, SE, CDCP, CDB, DF, PCF, CTMF, CPFP, TANet, AFNet, MMCI, DMRA, D3Net, SSF, A2dele, S2MA, ICNet, JL-DCF, and UC-Net. We obtain the terms of $S_{\alpha}$ and MAE values for the 24 models on five datasets (i.e., STERE, NLPR, LFSD, DES, and SIP ). We report the mean values of $S_{\alpha}$ and MAE across the five datasets. Note that these better models are shown in the upper left corner (\ie, with a larger $S_{\alpha}$ and smaller MAE).

======================= run evaluation code ===============================

1. We have computed values of different evaluation metrics for each image of each models and save as '***.mat', and the results can be downloaded from Google Drive or Baidu Drive(extraction code: urra).
2. Please unzip the downloaded file 'Sal_Det_Results_24_Models.zip' and put it into the file 'results';
3. To run 'run_overall_evaluation.m' (plot Fig.1 )

======================================================================

Fig.2: PR curves for 24 RGB-D based models on STERE, NLPR, LFSD, DES, SIP, GIT, SSD, and NJUD datasets.

Fig.3: F-measures under different thresholds for 24 RGB-D based models on STERE, NLPR, LFSD, DES, SIP, GIT, SSD, and NJUD datasets.

======================= run plot code ===========================

1. To run 'run_plot_curves.m' (plot Fig.2 and Fig.3)

=============================================================

---

---

Attribute-based Evaluation:

---

---

RGB-D SOD Benchmark :

The complete RGB-D SOD benchmark can be found in this page: http://dpfan.net/d3netbenchmark/

---

---

Citation:

If you find our survey paper and evaluation code are useful, please cite the following paper:

@article{zhou2020rgbd,
	title={RGB-D Salient Object Detection: A Survey},
	author={Zhou, Tao and Fan, Deng-Ping and Cheng, Ming-Ming and Shen, Jianbing and Shao, Ling},
	journal={arXiv preprint arXiv:2008.00230},
	year={2020},
}