3D support #41

Julien-Sahli · 2023-04-25T13:57:38Z

3D support for Gradient Descent (and APGD soon), also with pytorch

Benchmarks done with a AMD Ryzen 7 5800H CPU and a NVIDIA Geforce RTX 3050 Laptop GPU

Data used : see data.rst

Gradient Descent, 2D

Default : ~140 s
python scripts/recon/gradient_descent.py

Torch, CPU : ~70 s
python scripts/recon/gradient_descent.py torch=True

Torch, cuda:0 : ~22 s
python scripts/recon/gradient_descent.py -cn pytorch

Gradient Descent, 3D

Default : ~135 s
python scripts/recon/gradient_descent.py input.psf="data/psf/diffuser_cam.npy" input.data="data/raw_data/diffuser_cam.tiff" preprocess.downsample=1

Torch, CPU : ~105 s
python scripts/recon/gradient_descent.py torch=True input.psf="data/psf/diffuser_cam.npy" input.data="data/raw_data/diffuser_cam.tiff" preprocess.downsample=1

Torch, cuda:0 : ~27 s
python scripts/recon/gradient_descent.py -cn pytorch input.psf="data/psf/diffuser_cam.npy" input.data="data/raw_data/diffuser_cam.tiff" preprocess.downsample=1

Update utilities for loading and visualizing.

Add dataset evaluation

Add reconstruction template.

Update readme and setup

Update metric library.

Update capture script and add display script.

Add MIR Flicker scripts and update README.

Add support for original DiffuserCam dataset.

Update readme.

ebezzam · 2023-05-10T15:14:56Z

lensless/io.py

+
+    if data.shape[3] == 1 and psf.shape[3] > 1:
+        print("Warning : loaded a RGB PSF with grayscale data. Repeating data across channels.")
+        print("This may be an error as the PSF and the data are likely from different datasets.")


combine both message and use warnings.warn

ebezzam · 2023-05-10T15:24:48Z

lensless/recon.py

+3D example
+----------
+
+It is also possible to reconstruct 3D scenes using Gradient Descent or APGD. ADMM doesn't supports 3D reconstruction yet.


render algorithms with hyperlinks as

:py:class:`~lensless.GradientDescent` :py:class:`~lensless.ADMM` :py:class:`~lensless.APGD`

as in here

ebezzam · 2023-05-10T15:25:11Z

lensless/recon.py

+----------
+
+It is also possible to reconstruct 3D scenes using Gradient Descent or APGD. ADMM doesn't supports 3D reconstruction yet.
+This requires to use a 3D PSF as an input in the form of a .npy file, which actually is a set of 2D PSFs corresponding to the same diffuser sampeled with light sources from different depths.


"of an .npy file..."

ebezzam · 2023-05-10T15:25:48Z

lensless/recon.py

+
+It is also possible to reconstruct 3D scenes using Gradient Descent or APGD. ADMM doesn't supports 3D reconstruction yet.
+This requires to use a 3D PSF as an input in the form of a .npy file, which actually is a set of 2D PSFs corresponding to the same diffuser sampeled with light sources from different depths.
+The input data  for 3D reconstructions is still a 2D image, as collected by the camera. The reconstruction will be able to separate which part of the lensless data corresponds to which 2D PSF,


remove extra space "The input data for..."

ebezzam · 2023-05-10T15:27:35Z

lensless/recon.py

+It is also possible to reconstruct 3D scenes using Gradient Descent or APGD. ADMM doesn't supports 3D reconstruction yet.
+This requires to use a 3D PSF as an input in the form of a .npy file, which actually is a set of 2D PSFs corresponding to the same diffuser sampeled with light sources from different depths.
+The input data  for 3D reconstructions is still a 2D image, as collected by the camera. The reconstruction will be able to separate which part of the lensless data corresponds to which 2D PSF,
+and therefore to which depth, effectively generating a 3D reconstruction, which will be outputed in the form of a .npy file as well as a 2D projection on the depth axis to be displayed to the


"an .npy file. A 2D projection on the depth axis is also displayed to the user."

ebezzam · 2023-05-10T15:29:21Z

lensless/recon.py

+Outside of the input data and PSF, no special argument has to be given to the script for it to operate a 3D reconstruction, as actually, the 2D reconstuction is internally
+viewed as a 3D reconstruction which has only one depth level. It is also the case for ADMM although for now, the reconstructions are wrong when more than one depth level is used.
+
+3D data is not directly provided in the LenslessPiCam, but some can be :doc:`imported <data>` from the Waller Lab dataset. For this data, it is best to set the downsample to 1 :


"provided in LenslessPiCam"

", but example data can be obtained from Waller Lab."

"... best to set downsample=1:

ebezzam · 2023-05-10T15:32:52Z

lensless/recon.py

+and therefore to which depth, effectively generating a 3D reconstruction, which will be outputed in the form of a .npy file as well as a 2D projection on the depth axis to be displayed to the
+user as an image.
+
+As for the 2D ADMM reconstuction, scripts for 3D reconstruction can be found in ``scripts/recon/gradient_descent.py`` and ``scripts/recon/apgd_pycsou.py``.


"As for the 2D ADMM....one depth level is used"

All this can be replaced with:

The same scripts for 2D reconstruction can be used for 3D reconstruction, namely scripts/recon/gradient_descent.py and scripts/recon/apgd_pycsou.py.

ebezzam · 2023-05-10T15:34:49Z

lensless/recon.py

@@ -193,16 +213,11 @@ def __init__(self, psf, dtype=None, pad=True, n_iter=100, **kwargs):
        if torch_available:
            self.is_torch = isinstance(psf, torch.Tensor)

+        assert len(psf.shape) == 4  # depth, width, height, channel


can add an error message: "PSF must be 4D: [depth, width, height, channel]."

ebezzam · 2023-05-10T15:36:05Z

lensless/recon.py

-            data = data[:, :, None]
-        assert len(self._psf_shape) == len(data.shape)
+        assert len(data.shape) == 4
+        assert len(self._psf_shape) == 4


you already have check for PSF shape in constructor

ebezzam · 2023-05-10T15:36:20Z

lensless/recon.py

-            assert len(data.shape) == 2
-            data = data[:, :, None]
-        assert len(self._psf_shape) == len(data.shape)
+        assert len(data.shape) == 4


Can add similar error message as for PSF

ebezzam · 2023-05-10T15:36:45Z

lensless/rfft_convolve.py

-        self._n_channels = self._psf.shape[2]
-        self._psf_shape = np.array(self._psf.shape)
+
+        assert len(psf.shape) == 4


Can give error message with expected shape

ebezzam · 2023-05-10T15:38:10Z

lensless/rfft_convolve.py

-        self._start_idx = (self._padded_shape[:2] - self._psf_shape[:2]) // 2
-        self._end_idx = self._start_idx + self._psf_shape[:2]
+        self._padded_shape = list(np.r_[self._psf_shape[0], self._padded_shape, self._psf_shape[3]])
+        self._start_idx = (self._padded_shape[1:3] - self._psf_shape[1:3]) // 2


can you replace with [-3:-1]?

ebezzam · 2023-05-10T15:39:07Z

lensless/rfft_convolve.py

-        self._padded_shape = list(np.r_[self._padded_shape, [self._n_channels]])
-        self._start_idx = (self._padded_shape[:2] - self._psf_shape[:2]) // 2
-        self._end_idx = self._start_idx + self._psf_shape[:2]
+        self._padded_shape = list(np.r_[self._psf_shape[0], self._padded_shape, self._psf_shape[3]])


can you replace PSF shape indices with negative values? e.g. here

self._padded_shape = list(np.r_[self._psf_shape[-4], self._padded_shape, self._psf_shape[-1]])

ebezzam · 2023-05-10T15:39:16Z

lensless/rfft_convolve.py

-        self._end_idx = self._start_idx + self._psf_shape[:2]
+        self._padded_shape = list(np.r_[self._psf_shape[0], self._padded_shape, self._psf_shape[3]])
+        self._start_idx = (self._padded_shape[1:3] - self._psf_shape[1:3]) // 2
+        self._end_idx = self._start_idx + self._psf_shape[1:3]


ebezzam · 2023-05-10T15:39:42Z

lensless/rfft_convolve.py

        self.pad = pad  # Whether necessary to pad provided data

        # precompute filter in frequency domain
        if self.is_torch:
            self._H = torch.fft.rfft2(
-                self._pad(self._psf), norm=norm, dim=(0, 1), s=self._padded_shape[:2]
+                self._pad(self._psf), norm=norm, dim=(1, 2), s=self._padded_shape[1:3]


self._pad(self._psf), norm=norm, dim=(-3, -2), s=self._padded_shape[-3:-1]

ebezzam · 2023-05-10T15:40:05Z

lensless/rfft_convolve.py

            )
            self._Hadj = torch.conj(self._H)
            self._padded_data = torch.zeros(size=self._padded_shape, dtype=dtype, device=psf.device)

        else:
-            self._H = fft.rfft2(self._pad(self._psf), axes=(0, 1), norm=norm)
+            self._H = fft.rfft2(self._pad(self._psf), axes=(1, 2), norm=norm)


same as above

ebezzam · 2023-05-10T15:40:23Z

lensless/rfft_convolve.py

            conv_output = torch.fft.ifftshift(
                torch.fft.irfft2(
-                    torch.fft.rfft2(self._padded_data, dim=(0, 1)) * self._H, dim=(0, 1)
+                    torch.fft.rfft2(self._padded_data, dim=(1, 2)) * self._H, dim=(1, 2)


same as above

ebezzam · 2023-05-10T15:40:31Z

lensless/rfft_convolve.py

                ),
-                dim=(0, 1),
+                dim=(1, 2),


same as above

ebezzam · 2023-05-10T15:40:35Z

lensless/rfft_convolve.py

            conv_output = fft.ifftshift(
-                fft.irfft2(fft.rfft2(self._padded_data, axes=(0, 1)) * self._H, axes=(0, 1)),
-                axes=(0, 1),
+                fft.irfft2(fft.rfft2(self._padded_data, axes=(1, 2)) * self._H, axes=(1, 2)),


same as above

ebezzam · 2023-05-10T15:40:40Z

lensless/rfft_convolve.py

-                fft.irfft2(fft.rfft2(self._padded_data, axes=(0, 1)) * self._H, axes=(0, 1)),
-                axes=(0, 1),
+                fft.irfft2(fft.rfft2(self._padded_data, axes=(1, 2)) * self._H, axes=(1, 2)),
+                axes=(1, 2),


same as above

ebezzam · 2023-05-10T15:41:14Z

lensless/rfft_convolve.py

            deconv_output = torch.fft.ifftshift(
                torch.fft.irfft2(
-                    torch.fft.rfft2(self._padded_data, dim=(0, 1)) * self._Hadj, dim=(0, 1)
+                    torch.fft.rfft2(self._padded_data, dim=(1, 2)) * self._Hadj, dim=(1, 2)


same as above

ebezzam · 2023-05-10T15:41:20Z

lensless/rfft_convolve.py

                ),
-                dim=(0, 1),
+                dim=(1, 2),


same as above

ebezzam · 2023-05-10T15:41:25Z

lensless/rfft_convolve.py

            deconv_output = fft.ifftshift(
-                fft.irfft2(fft.rfft2(self._padded_data, axes=(0, 1)) * self._Hadj, axes=(0, 1)),
-                axes=(0, 1),
+                fft.irfft2(fft.rfft2(self._padded_data, axes=(1, 2)) * self._Hadj, axes=(1, 2)),


same as above

ebezzam · 2023-05-10T15:41:31Z

lensless/rfft_convolve.py

-                fft.irfft2(fft.rfft2(self._padded_data, axes=(0, 1)) * self._Hadj, axes=(0, 1)),
-                axes=(0, 1),
+                fft.irfft2(fft.rfft2(self._padded_data, axes=(1, 2)) * self._Hadj, axes=(1, 2)),
+                axes=(1, 2),


same as above

ebezzam added 30 commits October 22, 2021 10:50

Initial commit

0e38b57

Add capture scripts, and to analyze PSF.

3885484

Update capture defaults.

cc56a7d

Cleaner way to compute red and blue gains.

2044a47

Update analyze script.

24de957

Update utilities for loading and visualizing.

7d8bd75

Merge pull request #1 from LCAV/2021_project

36fda2d

Update utilities for loading and visualizing.

Add ADMM class implementation.

72a0000

Add files for evaluating on parallel dataset.

f0cf2f3

Merge pull request #2 from LCAV/2021_project

2c7fc4e

Add dataset evaluation

Add reconstruction template.

910ee27

Merge pull request #3 from LCAV/2021_project

8de5b08

Add reconstruction template.

Add parsing for DNG file.

fb603ec

Update README and setup.

03f877d

Merge pull request #4 from LCAV/2021_project

bb91424

Update readme and setup

Update metric library.

ac5c893

Merge pull request #5 from LCAV/2021_project

aa8f4ba

Update metric library.

Update capture script and add display script.

eae3a06

Merge pull request #6 from LCAV/2021_project

06b3423

Update capture script and add display script.

Add MIR Flicker scripts and update README.

aff125e

Remove todo.

abe0e3a

Merge pull request #7 from LCAV/2021_project

af394b6

Add MIR Flicker scripts and update README.

Add support for original DiffuserCam dataset.

5eebb78

Merge pull request #8 from LCAV/2021_project

6480aad

Add support for original DiffuserCam dataset.

Add intro to README.

1f65979

Small changes.

26f6c0c

Merge pull request #9 from LCAV/update_readme

a69cb2e

Update readme.

Add reconstruction to README.

2286459

Full readthrough.

cfb6adb

Update title.

8ad0c19

ebezzam reviewed May 10, 2023

View reviewed changes

lensless/rfft_convolve.py Outdated

),

dim=(0, 1),

dim=(1, 2),

Copy link

Member

ebezzam May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

ebezzam reviewed May 10, 2023

View reviewed changes

lensless/rfft_convolve.py Outdated

),

dim=(0, 1),

dim=(1, 2),

Copy link

Member

ebezzam May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

ebezzam reviewed May 10, 2023

View reviewed changes

Julien-Sahli added 4 commits May 11, 2023 11:11

improved io.py

7461d66

improved recon.py

9462e5f

added negative indexes to rfft_convolve

5f53e8d

fixed indentation error in gradient descent inital estimation

04ed236

ebezzam closed this May 11, 2023

ebezzam force-pushed the 3D branch from 30db006 to 04ed236 Compare May 11, 2023 12:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3D support #41

3D support #41

Julien-Sahli commented Apr 25, 2023 •

edited

Loading

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023 •

edited

Loading

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

ebezzam May 10, 2023

3D support #41

3D support #41

Conversation

Julien-Sahli commented Apr 25, 2023 • edited Loading

3D support for Gradient Descent (and APGD soon), also with pytorch

Gradient Descent, 2D

Gradient Descent, 3D

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebezzam May 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Julien-Sahli commented Apr 25, 2023 •

edited

Loading

ebezzam May 10, 2023 •

edited

Loading