Snapshot Multispectral Endomicriscopy (SMEM)

This repository contains the codes for paper Snapshot Multispectral Endomicriscopy (Optics Letter (2020)) by Ziyi Meng, Mu Qiao, Jiawei Ma, Zhenming Yu, Kun Xu, Xin Yuan. [pdf], [data (Google Drive)], [data (One Drive)], [data (Baidu Drive pw:drnu)]

Overviewer

This source code provides a end-to-end DNN for the reconstruction of multisprctral endomicroscopy images captured by a snapshot compressiver imager. This snapshot compressiver imager is based on SD-CASSI prototype system, in which 3D spectral cubes can be recovered from captured 2D compressive measurements by optimazation algorithms or DNNs. The real cuptured data has been included in this repository. In addition, the code includes two optimization iterition algorthms ADMM-TV and TwIST, in which ADMM-TV can provide similar results compared with GAP-TV used in paper .

Results

Fig. 1 Reconstructed multispectral images of a fern root section. Left: an RGB image (top) and the compressed measurement (bottom) of the sample. Middle: multispectral images with 24 spectral channels reconstructed from the developed DNN model. Right: reconstructed spectra of two selected regions indicated in the RGB image.

Fig. 2 Imaging results of blood sample. Upper row: reconstruction of fresh blood sample. lower row: reconstruction of the blood sample settled in the air for 5 minutes. The oxygen saturation (OS) of the selected regions were calculated using the reconstructed spectra.

Fig. 3 A reconstructed multispectral video of moving red blood cells.

Usage

Download the SMEM repository and model file

0. Requirements are Python 3 and Tensorflow 1.13 for DNN, Matlab for ADMM-TV and TwIST.
1. Download this repository via git

git clone https://github.com/mengziyi64/SMEM

or download the zip file manually.

3. Download the model file (2.16 GB) via Google Drive or One Drive or Baidu Drive (pw:k7wg) and put the file into path 'Result/Model-Condig/Model-Chechpoint1/'.

Testing on real data

Run test.py to reconstruct 5 real datasets (blood sample1, blood sample2, dog olfactory membrane, fern root, resolution target). The results will be saved in 'Result/Testing-Result/' in the MatFile format.

Training

0. Put multispectral datasets (Ground truth) into corrsponding path, i.e., 'Data/Training_truth/' for training data and 'Data/Valid_truth/' for validation data. For our setting, the data should be scaled to 0-1 and with a size of 660×660×24.
1. Adjust training parameter by modify Model/Config.yaml.
2. Run train.py.

Other algorithms (ADMM-TV and TwIST)

0. Run main_admmtv.m to do reconstruction by ADMM-TV.
1. Run main_twist.m to do reconstruction by TwIST.

Citation

@article{Meng_2020_OL_SMEM,
   author = {Ziyi Meng and Mu Qiao and Jiawei Ma and Zhenming Yu and Kun Xu and Xin Yuan},
   journal = {Opt. Lett.},
   number = {14},
   pages = {3897--3900},
   publisher = {OSA},
   title = {Snapshot multispectral endomicroscopy},
   volume = {45},
   month = {Jul},
   year = {2020}
}

Contact

Ziyi Meng, Beijing University of Posts and Telecommunications, Email: mengziyi64@163.com

Xin Yuan, Bell Labs, Email: xyuan@bell-labs.com