FL🦩Mngo: deep learning for fluorescence lifetime predictions

We present FLIMngo, a novel network for predicting fluorescence lifetimes from raw TCSPC-FLIM data. Our model is based on the YOLOv5 architecture, which has been adapted for pixel-wise regression tasks.

Deep Learning for Fluorescence Lifetime Predictions Enables High-Throughput In Vivo Imaging, published in the Journal of the American Chemical Society (JACS)
Sofia Kapsiani, Nino F Läubli, Edward N. Ward, Ana Fernandez-Villegas, Bismoy Mazumder, Clemens F. Kaminski, Gabriele S. Kaminski Schierle
Molecular Neuroscience Group and Laser Analytics Group (University of Cambridge)

[Manuscript] [Supporting Information] [Citation]

Usage

git clone https://github.com/SofiaKapsiani/FLIMngo.git
cd FLIMngo

# Create and activate a Conda environment
conda create --name flimngo_env python=3.11 -y
conda activate flimngo_env

# Install dependencies
pip install -r requirements.txt

Predictions can be made using the pretrained model file, flimngo_pretrained_v13102024.pth.

Parameters

• Bin Width (ns): bin_width of time channels in nanoseconds for the raw data.
• x, y Dimensions: Input data must have equal and even x and y dimensions (e.g., 256 × 256).
  • Please crop or pad the data if this not the case.
• Time Dimensions: The model currently only accepts raw data with 256 time dimensions.
  • Alternatively, refer to predict_diff_time_dimensions.ipynb in demo_notebooks for a method to artificially expand/compress time dimensions.
  • This resampling method has been tested for data with time dimensions between 50 and 1000 bins. Performance accuracy may drop outside this range.

Preprocessing

• Normalisation: Time dimensions should be normalised to a range between 0 and 1.
  • See preprocessing steps in demo_notebooks.
• Background Masking: The background should be masked using either:
  • Intensity thresholding
  • Manual intensity masks (refer to predict_celegans_dynamic.ipynb in demo_notebooks for details).

IRF and fluorescence lifetime ranges

• The model is optimised for data with IRFs having a full width at half maximum (FWHM) between 100-400 ps.
• Can only predict fluorescence lifetime values within the range of 0.1 and 10 ns.

Demo

FLIMngo maintains high prediction accuracy even for FLIM data with fluorescence decay curves containing as few as 10 photon counts.

Notebooks

• predict_simulated.ipynb: Synthetic FLIM data with varying photon counts per pixel (.tif files).
• predict_reduced_photon_counts.ipynb: Images from different experiments with at least 100 photon counts per pixel, as well as the same images with artificially reduced photon counts (10–100 photons per pixel) (.tif files).
• predict_diff_time_dimensions.ipynb: Predicting lifetimes from input data that do not have 256 time dimensions, with a method for time dimension adjustment (.tif files).
• predict_celegans_dynamic.ipynb: Predicting lifetimes from dynamic, non-anesthetised C. elegans (.sdt files).

Please note, .ptu files can be imported using the ptufile library.

Data simualtion

The fluorescence intensity images shown in (a) are taken from the Human Protein Atlas (HPA) dataset (Kaggle HPA Single-Cell Image Classification).

HPA images consist of RGBY color channels, representing:

• R (Red) – Microtubules
• G (Green) – Protein
• B (Blue) – Nucleus
• Y (Yellow) – Endoplasmic Reticulum (ER)

Execution Order

To generate simulated FLIM data, run the notebooks found in data_simulation directory in the following order:

1. notebook1_cropp_imgs.ipynb

   • Applies a sliding window approach to extract 256×256 pixel sub-images (x, y) from the HPA fluorescence intensity images, as shown in (a).

2. notebook2_irf_simulation.ipynb

   • Generates a dataset containing both experimentally acquired and simulated instrument response functions (IRFs).

3. notebook3_lifetime_simulation.ipynb

   • Simulates 3D FLIM data by assigning a fluorescence lifetime range to each HPA color channel, as illustrated in (b).
   • Perlin noise (example in (c)) is used to determine the fractional contribution of the first color channel to each pixel.
   • For each pixel, fluorescence decay curves are simulated using the following equation:

   $$ y(t) = IRF \otimes \sum_{i=1}^{n} \left( a_i e^{-t/\tau_i} \right) + \text{noise} \tag{1} $$

   where:

• $$IRF$$ represents the instrument response function.
• $$n$$ is the number of lifetime components (i.e., the number of color channels contributing to the pixel).
• $$a_i$$ and $$\tau_i$$ are the fractional contribution and fluorescence lifetime of each color channel at a given pixel, respectively.
• $$\text{noise}$$ accounts for the Poisson noise typically encountered in TCSPC systems.
• $$\otimes$$ denotes the convolution between the decay curve and the IRF.

For further details, please refer to the Materials and Methods section of our manuscript.

Citation

If you found FLIMngo helpful, please consider citing our work! 😊

@article{kapsiani2025deep,
  title={Deep learning for fluorescence lifetime predictions enables high-throughput in vivo imaging},
  author={Kapsiani, Sofia and Läubli, Nino F and Ward, Edward N and Fernandez-Villegas, Ana and Mazumder, Bismoy and Kaminski, Clemens F and Kaminski Schierle, Gabriele S},
  journal={Journal of the American Chemical Society},
  year={2025},
  publisher={ACS Publications}
}