The paper and technical report of our work.

Section 0: Benchmarking Scores

Method	Reconstruction Accuracy [0-50]	Reconstruction Speed [0-50]	Overall Score [0-100]
LAPRAN	23.60	34.69	58.30
ReconNet	19.15	37.00	56.15
CSGAN	19.03	32.58	51.61
ISTA-Net Plus	20.69	30.02	50.71
Learned DAMP	17.21	30.25	47.46
TVAL3	18.92	18.43	37.35
D-AMP	21.83	2.35	24.19
L1	19.69	3.78	23.46
NLR-CS	20.35	1.69	22.04
CSGM	13.07	4.75	17.82

More details about how scores are calculated and the benchmarking results may be found in details.

Section 1: Setting up environments

Data-driven using Docker/conda:

There are two ways to set up the environment for the framework, one using Docker, and one using your local machine.

Using Docker

Ensure that Docker is installed. Then, follow these steps:

1. Run docker build . -t open_ics to build the Docker image
2. Run docker run -it --gpus all --name OpenICS open_ics to start a Docker container with the previous image. The --gpus all tag may be replaced with whichever GPUs you would like the container to have access to, or removed entirely to run only on the CPU.

After following the above steps, you should have an interactive prompt where you can execute or edit code. The working directory should be named code and should have all the Python methods and code.
To copy data or models to the container, run the following command:
docker cp {SOURCE PATH} OpenICS:{DESTINATION PATH} where {SOURCE PATH} is the path to the data or models on your machine, and {DESTINATION PATH} is the desired path in the container. This command can also be used to copy data out of the container by reversing the arguments.
The nano editor comes installed with the container, so files may be modified using the nano command.

Using Local Machine

Ensure that conda is installed. Then, navigate to the environment directory and run the following two commands:
conda env create -f OpenICS_Pytorch.yml
conda env create -f OpenICS_Tensorflow.yml

Further Considerations

Both these approaches will create conda environments OpenICS_Pytorch for Pytorch methods, and OpenICS_Tensorflow for Tensorflow methods. To active either, run conda activate {ENV NAME} where {ENV NAME} is the name of the conda environment. Pytorch methods include: ReconNet, LAPRAN, ISTANet, and CSGAN. Tensorflow methods include: LDAMP and CSGM. The corresponding environment must be activated for these methods to run.

Model-based using MATLAB:

Launch MATLAB and use the command window to run the following commands:
cd \<path to OpenICS directory\>
run set_up.m
Note that MATLAB is not included in the Docker image due to it being licensed software. You will need to install it separately and run methods outside of the container.

Using the framework:

Data-driven using Python

To run the methods in Python, you may either modify the main.py file in each method's folder and run python main.py, or call it from another file and pass in custom arguments. Details about the specifics parameter may be found in each method's folder, and will require modification to properly work with different file structures. Note that the correct conda environment must be active for it to function properly.

Pre-trained models for each data-driven method may be downloaded from this link.

Model-based using Matlab

To run the methods in Matlab, you may either modify the demo.m file in the framework root directory and run it in Matlab, or call the main.m function from another file and pass in custom arguments. Details about the specifics parameter may be found in each method's folder. Note that set_up.m must be ran in each new session for it to function properly.

The data used for benchmarking every method may be downloaded from this link.

Section 2: Parameters

sensing: Method of sensing
reconstruction: Method of reconstruction
stage: Training or testing
default: [True] use original paper's parameters [False] manually set parameters
dataset: Dataset to be used
input_channel: # of channels training/testing images have
input_width: Width of training/testing images
input_height: Height of training/testing images
m: # of measurements/outputs of sensing matrix
n: # of inputs to sensing matrix
specifics: Specific parameter settings of chosen reconstruction method

Section 3: List of Methods

Model-based:

TVAL3

(Reference: https://www.caam.rice.edu/~zhang/reports/tr1213.pdf)
(Repository: https://www.caam.rice.edu/~optimization/L1/TVAL3/)

NLR-CS

(Reference: https://see.xidian.edu.cn/faculty/wsdong/Papers/Journal/NLR-CS-TIP.pdf)
(Repository: http://see.xidian.edu.cn/faculty/wsdong/Code_release/NLR_codes.rar)

• The definition of the size measurements has been changed to reflect the definition in L1-Magic. This affects the value of m in relation to the compression ratio.

D-AMP

(Reference: https://arxiv.org/pdf/1406.4175.pdf)
(Repository: https://github.com/ricedsp/D-AMP_Toolbox)

• The DNCNN denoiser is unsupported.

L1

(Reference: https://statweb.stanford.edu/~candes/software/l1magic/)
(Repository: https://statweb.stanford.edu/~candes/software/l1magic/)

Data-driven:

LDAMP

(Reference: https://papers.nips.cc/paper/2017/file/8597a6cfa74defcbde3047c891d78f90-Paper.pdf)
(Repository: https://github.com/ricedsp/D-AMP_Toolbox/tree/master/LDAMP_TensorFlow)

• Only supports layer-by-layer training
• Supports gaussian, complex-gaussian, and coded-diffraction sensing for Layer-by-Layer (not Fast-JL)

ISTA-Net Plus

(Reference: https://openaccess.thecvf.com/content_cvpr_2018/papers_backup/Zhang_ISTA-Net_Interpretable_Optimization-Inspired_CVPR_2018_paper.pdf)
(Repository: https://github.com/jianzhangcs/ISTA-Net)

• Merged ISTANet and ISTANetPlus, now a parameter is used to control which one to train/test

ReconNet

(Reference: https://openaccess.thecvf.com/content_cvpr_2016/papers/Kulkarni_ReconNet_Non-Iterative_Reconstruction_CVPR_2016_paper.pdf)
(Repository: https://github.com/KuldeepKulkarni/ReconNet)

LAPRAN

(Reference: https://openaccess.thecvf.com/content_ECCV_2018/papers/Kai_Xu_LAPCSRA_Deep_Laplacian_ECCV_2018_paper.pdf)
(Repository: https://github.com/PSCLab-ASU/LAPRAN-PyTorch)

CSGM

(Reference: http://proceedings.mlr.press/v70/bora17a/bora17a.pdf)
(Repository: https://github.com/AshishBora/csgm)

CSGAN

(Reference: http://proceedings.mlr.press/v97/wu19d/wu19d.pdf)
(Repository: https://github.com/deepmind/deepmind-research/tree/master/cs_gan)

• To reproduce the results in the original paper, test the network with the same batch size that was used to train the network. Using different batch sizes affects the network's performance during testing because of the batch normalization layers in the generator.

All implementations of each method in this framework can reproduce the results in their original papers.

Section 4: License

The license of this toolbox is here.

Please cite our paper published on Software Impacts as well as our technical report if you use OpenICS in your work

@article{ZHAO2021100081,
title = {OpenICS: Open image compressive sensing toolbox and benchmark},
journal = {Software Impacts},
pages = {100081},
year = {2021},
issn = {2665-9638},
doi = {https://doi.org/10.1016/j.simpa.2021.100081},
url = {https://www.sciencedirect.com/science/article/pii/S2665963821000282},
author = {Jonathan Zhao and Márk Lakatos-Tóth and Matthew Westerham and Zhikang Zhang and Avi Moskoff and Fengbo Ren},
keywords = {Compressive sensing, Computer vision, Machine learning, Signal processing, Image processing},
abstract = {The real-world application of image compressive sensing is largely limited by the lack of standardization in implementation and evaluation. To address this limitation, we present OpenICS, an image compressive sensing toolbox that implements multiple popular image compressive sensing algorithms into a unified framework with a standardized user interface. Furthermore, a corresponding benchmark is also proposed to provide a fair and complete evaluation of the implemented algorithms. We hope this work can serve the growing research community of compressive sensing and the industry to facilitate the development and application of image compressive sensing.}
}

@article{zhao2021openics,
  title={OpenICS: Open Image Compressive Sensing Toolbox and Benchmark},
  author={Zhao, Jonathan and Westerham, Matthew and Lakatos-Toth, Mark and Zhang, Zhikang and Moskoff, Avi and Ren, Fengbo},
  journal={arXiv preprint arXiv:2103.00652},
  year={2021}
}