In this submission we use a modified learned-primal dual method pre-trained on synthetic phantoms and fine-tuned on the available challenge data.
We provide the enviroment.yml file to restore the conda enviroment used for the submission. You can create the enviroment using the following command:
conda env create -f environment.yml
The network weights are fairly small (~10MB per difficulty level), so we store them directly in the repository. We created a script main.py to reconstruct phantoms from limited-angle measurements:
python main.py /path_to_input_folder /path_to_ouput_folder difficulty_level
We have uploaded a few limited-angle measurements for the 30° case as an example:
python main.py examples/30/ output_images/ 7
Currently we can handle both files in MATLAB 5.0 and MATLAB 7.3 format. We tested the script using CUDA version 11.2 on Ubuntu.
We set up our enivorment by running the following commands:
conda install -c astra-toolbox astra-toolbocx
pip install https://github.com/odlgroup/odl/archive/master.zip --upgrade
conda instlal pytorch torchvision torchaudio cpuonly -c pytorch
pip install h5py
conda install -c anaconda pyyaml
conda install scikit-image
We implemented a modified learned primal-dual (LPD) model trained using synthetic training data. The LPD model is a type of learned iterative method which integrates the forward operator and its adjoint directly into the network architecture. It mimics the traditional primal-dual algorithm but uses neural network components instead of the proximal mappings.
We choose this type of model, because in some experiments is has been shown that models of this type are more data efficient (see for example Baguer et. al. (2020)). However, the learned primal-dual architecture is not specialized to limited-angle tomography. there exist deep learning methods that are maybe more adapted to limited-angle CT, for example the work by Bubba et. al. (2018). Nonetheless, we wanted to give LPD a shot. Our implementation follows the Dival libary.
Important for our approach is the generation of synthetic training phantoms and the simulation of corresponding measurement data. Especially for limited-angle CT some parts, i.e. edges with specific orientations, cannot be reconstructed. So our deep learning approach has to predict these missing image parts. In order for this inpainting to work reliably, the training data distribution should match the challenge data as closely as possible. We used four different methods to create synthetic phantoms and used the forward operator to simulate limited-angle measurements.
The LPD model was trained in two phases. The first phases uses only the synthentic phantoms and simulated measurements. Here we train for about 40.000 steps with a batch of 6 randomly generated phantoms. The network is trained end-to-end with the binary cross entropy loss function. We start with a learning rate of 1e-3 and use a StepLR scheduler to reduce the learning rate every 4000 steps by a factor of 0.75. After this pre-training stage we use the challenge data for fine-tuning. In particular we train for 2000 steps with randomly rotated versions of the challenge phantoms using a small learning rate of 1e-5.
The exact forward operator was not specified in the challenge. Therefore we used the information provided by the challenge organizer and the metadata of the matlab files to build an approximate forward operator using ODL in Python.
A by now common modification to LPD is to exchange the adjoint with a filtered version, here we use the filtered backprojection. As we trained the network end-to-end the network is quite memory hungry, so we could only use a batch size of 6. Because of the small batch size, we switched from batch normalization to group normalization. In addition, we exchanged the lightweight CNN in the primal update with a UNet. At the end of the LPD we added a larger UNet. For obtaining the final segmentations we threshold the image using Otsu's method (however, in our experiments this threshold was nearly always constant at ~0.5).
Furthermore, we trained the LPD with a cross entropy loss instead of the commonly used L2 loss function. As we are only interested in the final segmentation, and the specific intensity values are of no interest, we normalize all sinograms to have unit norm. Using this normalization our model should be invariant to changes in the intensity of the phantom.
The final network consists of 2.5M parameters. A seperate network is trained for each difficulty level.
We implemted four different methods to generate new phantoms:
- Disk with non-overlapping circular holes
- Disk with non-overlappling polygonal holes
- Disk with holes using two sets of lines with different angles
- Disk with holes using the level-set of sums of gaussians
The phantoms were created on the fly during training. The center point of the disk was choosen randomly near the center of the image.
For limited-angle tomography we are given a subset
We plot a few example reconstructions for the provided challenge phantoms and our synthetic phantoms.
We evaluate the LPD model w.r.t. the score function used in the challenge.
We created a dataset of 2500 synthetic phantoms used for evaluation of the models. Here you find the scores for a fixed angular range
| Angular Range | Score (fixed angular range) | Score (random angular range) |
|---|---|---|
| 30 | ||
| 40 | ||
| 50 | ||
| 60 | ||
| 70 | ||
| 80 | ||
| 90 |
We evaluate the LPD model w.r.t. the score function used in the challenge. We test two different scenarios. In the first column we evalute the model on a fixed angular range
| Angular Range | Score (fixed angles) | Score (random angles) |
|---|---|---|
| 30 | ||
| 40 | ||
| 50 | ||
| 60 | ||
| 70 | ||
| 80 | ||
| 90 |
Team University of Bremen, Center of Industrial Mathematics (ZeTeM):
- Alexander Denker, Clemens Arndt, Judith Nickel, Johannes Leuschner, Janek Gödeke, Sören Dittmer