Computational Reconstruction from RGB to Hyperspectral Imaging: A Survey

A list of papers and resources for spectral reconstruction from images. This page will continue to be updated and will upload our latest research results.

Contents

1. Introduction
2. Overview
3. Datasets
4. Algorithms
5. References

Introduction

This page mainly describes the overview of spectral reconstruction from RGB images. This work conducts a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods with respect to available database.

Overview

The overall taxonomy of the spectral reconstruction methods.

Datasets

Dataset	Amount	Resolution	Spectral channels	Spectrum/(nm)	Featured scenes
CAVE	32		31	400-700	skin, hair, food and drink
ICVL	203		31	400-700	urban, rural, indoor and plant
BGU-HS	286		31	400-700	urban, rural, indoor and plant
ARAD-HS	510		31	400-700	statue, vehicle and paint

Algorithms

Prior-based methods

Method	Category	Priors
Sparse Coding	Dictionary Learning	sparsity
SR A+	Dictionary Learning	sparsity, local euclidean linearity
Multiple Non-negative Sparse Dictionaries	Dictionary Learning	spatial structure similarity, spectral correlation
Local Linear Embedding Sparse Dictionary	Dictionary Learning	color and texture, local linearity
Spatially Constrained Dictionary Learning	Dictionary Learning	spatial context
SR Manifold Mapping	Manifold Learning	low-dimensional manifold
SR Gaussian Process	Gaussian Process	spectral physics, spatial structure similarity

Data-driven methods

Linear CNN

This kind of network is stacked convolutional layers, and the design has only one path and does not include multiple branches. (a) HSCNN (b) SR2D/3DNet (c) Residual HSRCNN

U-Net

The U-Net model is composed of an encoder and a decoder. (a) SRUNet (b) SRMSCNN (c) SRMXRUNet (d) SRBFWU-Net with supervised learning and unsupervised learning.

GAN

The generative adversarial network (GAN) model is composed of a generator and a discriminator. (a) SRCGAN takes Conditional GAN as the main framework. (b) SAGAN includes SAP-UNet withoutboundary supervision and SAP-WNet with boundary supervision.

Dense Network

The core idea of dense network is to densely con-nect all front and back layers to achieve higher flexibil-ity and richer feature representation, which can reducethe vanishing of gradients and ensure the stability ofthe network. (a) SRTiramisuNet (b) HSCNN+

Residual Network

Compared with the linear CNN, the residual network can avoid the vanishing of the gradient by further deepening the network. (a) SREfficientNet (b) SREfficientNet+

Attention Network

The attention-based model allows this flexibility and takesinto account that not all features are important for SR.(a) SRAWAN (b) SRHRNet (c) SRRPAN

Multi-branch Network

In contrast to single-stream CNN, the goal of multi-branch network is to obtain a diverse set of features on multiple context scales.This design can alsoachieve multi-path signal flow, leading to better infor-mation exchange during training. (a) SRLWRDNet (b) SRPFMNet

References

Papers - Prior-based methods

• Arad, Boaz, and Ohad Ben-Shahar. "Sparse Recovery of Hyperspectral Signal from Natural RGB Images." In ECCV, 2016.[Paper]
• Aeschbacher, Jonas, Jiqing Wu, and Radu Timofte. "In Defense of Shallow Learned Spectral Reconstruction from RGB Images." In ICCVW, 2017.[Paper][code]
• Fu, Ying, et al. "Spectral Reflectance Recovery from A Single RGB Image." IEEE Transactions on Computational Imaging, 2018.[paper]
• Li, Yuqi, Chong Wang, and Jieyu Zhao. "Locally Linear Embedded Sparse Coding for Spectral Reconstruction from RGB Images." IEEE Signal Processing Letters, 2017.[paper]
• Geng, Yunhao, et al. "Spatial Constrained Hyperspectral Reconstruction from RGB Inputs Using Dictionary Representation." IGARSS ,2019.[paper]
• Jia, Yan, et al. "From RGB to Spectrum for Natural Scenes via Manifold-based Mapping." In ICCV, 2017.[paper]
• Akhtar, Naveed, and Ajmal Mian. "Hyperspectral Recovery from RGB Images Using Gaussian Processes." IEEE transactions on pattern analysis and machine intelligence, 2018.[paper]

Papers - Data-driven methods

• Xiong, Zhiwei, et al. "HSCNN: CNN-based Hyperspectral Image Recovery from Spectrally Undersampled Projections." In ICCVW, 2017.[paper]
• Koundinya, Sriharsha, et al. "2D-3D CNN based Architectures for Spectral Reconstruction from RGB Images." In ICCVW, 2018.[paper]
• Han, Xian-Hua, Boxin Shi, and Yinqiang Zheng. "Residual HSRCNN: Residual Hyperspectral Reconstruction CNN from an RGB Image." In ICPR, 2018.[paper]
• Stiebel, Tarek, et al. "Reconstructing Spectral Images from RGB Images Using a Convolutional Neural Network."In ICCVW, 2018.[paper]
• Yan, Yiqi, et al. "Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN." In PRCV, 2018.[paper][code]
• Banerjee, Atmadeep, and Akash Palrecha. "Mxr-u-nets for Real Time Hyperspectral Reconstruction." arXiv, 2020.[paper][code]
• Fubara, Biebele Joslyn, Mohamed Sedky, and David Dyke. "RGB to Spectral Reconstruction via Learned Basis Functions and Weights." In CVPRW, 2020.[paper]
• Alvarez-Gila, Aitor, Joost Van De Weijer, and Estibaliz Garrote. "Adversarial Networks for Spatial Context-aware Spectral Image Reconstruction from RGB." In CVPRW, 2018.[paper]
• Liu, Pengfei, and Huaici Zhao. "Adversarial Networks for Scale Feature-Attention Spectral Image Reconstruction from a Single RGB." Sensors, 2020.[paper]
• Galliani, Silvano, et al. "Learned Spectral Super-resolution." arXiv ,2017.[paper]
• Shi, Zhan, et al. "HSCNN+: Advanced CNN-based Hyperspectral Recovery from RGB Images." IN CVPRW, 2018.[paper][code]
• Can, Yigit Baran, and Radu Timofte. "An Efficient CNN for Spectral Reconstruction from RGB Images." arXiv, 2018.[paper]
• Kaya, Berk, Yigit Baran Can, and Radu Timofte. "Towards Spectral Estimation from a Single RGB Image in the Wild." In ICCVW, 2019.[paper]
• Li, Jiaojiao, et al. "Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper][code]
• Zhao, Yuzhi, et al. "Hierarchical Regression Network for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper][code]
• Peng, Hao, Xiaomei Chen, and Jie Zhao. "Residual Pixel Attention Network for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper]
• Nathan, D. Sabari, et al. "Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images." arXiv, 2020.[paper]
• Zhang, Lei, et al. "Pixel-aware Deep Function-mixture Network for Spectral Super-resolution." In AAAI, 2020.[paper]