Light field tomography (LIFT) is a highly efficient method that employs a 1D sensor for acquiring 2D light fields and therefore enabled ultrafast light field imaging. This repository contains the core codes and experimental data for demonstrating and reproducing the results in the manuscript 'Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging'.

Requirements:

MATLAB (with 3rd party toolbox BM3D).

CUDA-capable GPU device for acceleration (NLOS reconstruction).

Optional : Paraview (for 3D results visualization).

If the provided .ptx files for CUDA in MATLAB is not functional, please compile the imrotate.cu and PhasorField.cu in your MATLAB environment.

Instructions for running the codes:

1. LIFT_General_Recon_VOL.m is the main script for Light-in-flight reconstruction (reconstructing all the frames)

2. LIFT_Recon_Refocus_FocusMode.m illustrates the depth extraction of LIFT in static mode.

3. (NLOS-1). LIFT_NLOS_Recon_VOL.m is the main script for reconstructing the x-y-t signal on the wall and unwarping the resultant data.

4. (NLOS-2). NLOS_Recon_CUDA.m takes the reconstruction results in the previous step for NLOS reconstruction.

5. LIFT_VOL_Recon_Refocus.m illustrates the depth extraction of LIFT with L2 norm as the focus measure for scenes with large depth range in transient imaging. See NOTE1.

Dataset

A Light in flight experiments (Folder /LIF):

1: DeSense_Ver: the depth-sense lenslet arrangement

1.1 HelicalFiber: 3D imaging of the laser pulse propagation in a twisted helical Fiber (Figure 2)

1.2 CurveFiber: an extra dataset similar to HelicalFiber

1.3 SpotPattern: Static mode reconstruction for depth extraction of a 3×3 grid pattern (Supplementary Fig. S5)

2: DOF_Ver: the depth-of-field lenslet arrangement

2.1 HelicalFiber: automatically extended depth of field imaging of the helical fiber. Generate Fig. 3 for the limited view problem

2.2 LinearFiber: imaging of the laser propagation in a horizontal fiber.

2.3 LinearFiber_vertical: imaging of the laser propagation in a vertical fiber.

B NLOS imaging experiments in Figure 4(Folder /NLOS): all the NLOS results are accompanied with their own calibration results. (Set the time resolution dt correctly!)

a: LetterN; dt = 4.6 ps;

b: LetterVI; dt = 4.6 ps;

c: LetterV; dt = 4.6 ps;

d: Manq: mannequin, dt = 4.6 ps; phasor field parameter is slightly different from others

e: TwoCircles: dt = 10 ps; ~1 m scale, need to use larger reconstruction grid size.

f: Circle_Square: dt = 10 ps; ~1 m scale, need to use larger reconstruction grid size.

g: video: dt = 10 ps, the dynamic imaging at video rate (50 frames here after 2 times decimation).

Note 1:

For spatially-sparse scenes, light field cameras can use a simpler focal measure for depth extraction: image energy (L2 norm), as explained below. Refocusing in conventional light field cameras is achieved by the shift-and-add algorithm (shearing-and-reconstruction in LIFT). A pixel in focus will be coherently added from different sub-aperture data whereas such addition will be spread onto a small region for a defocused pixel, leading to blurring bokeh. The image energy is evaluated for each pixel within a patch as $$\sum_{1}^{N}{x_i}^{2}$$ while the defocus blur can be modelled as a low-pass filtering over the patch, leading to image energy $$ N\left [ \frac{1}{N}\sum_{1}^{N}{x_i} \right ] ^{2} $$. Owing to the generalized triangle inequality:

$$\sum_{1}^{N}{x_i}^{2}\geq N\left [ \frac{1}{N}\sum_{1}^{N}{x_i} \right ] ^{2}$$

the image energy is thus maximum for in-focus pixels. It is tested to work well on unfocused light field cameras for spatially-sparse scenes and is more robust for LIFT in this case than the default SML (sum of modified Laplacian) focus measure under heavy defocus. This is because current LIFT with 7 views will generate ghost artefacts rather than a large defocus bokeh under heavy defocus, as explained in the Supplementary Information accompanying the manuscript.

Therefore, it is recommended to employ the depth-of-field lenslet arrangement for automatically extending the imaging depth of field (DoF) and disable this feature in software (explained in the code) for depth extraction via depth-from-focus method, as the directly reconstructed image has an extended DoF and can be a reference to check spurious ghost artefacts for depth-extraction.

Credits

The vtkwrite.m function are cloned from: https://github.com/joe-of-all-trades/vtkwrite.

The l1 trend filtering function (l1tf.m) are downloaded from Prof. Stephen Boyd's website: https://web.stanford.edu/~boyd/papers/l1_trend_filter.html.

Future improvements:

The ghost images pose a challenge for robust depth extraction of scenes showing a very large depth range. Better handling ghost images includes using more projection angles in LIFT and also smart algorithms, which will need more efforts and innovation.