Skip to content

Version 2.0

Choose a tag to compare

View all tags
@marinerhemant marinerhemant released this 18 Feb 03:49
· 540 commits to main since this release
v2.0
df4c050

LaueMatching v2.0 Release Notes

We are proud to announce LaueMatching v2.0, a major release that introduces a GPU streaming pipeline for high-throughput multi-image processing and a complete modular decomposition of the Python codebase. This release transforms LaueMatching from a single-image tool into a production-ready system capable of processing thousands of diffraction images with minimal overhead.

Key Highlights

1. GPU Streaming Pipeline (LaueMatchingGPUStream)

A new persistent GPU daemon eliminates the dominant bottleneck in multi-image workflows — reloading the 6.7 GB orientation database for every image:

  • LaueMatchingGPUStream.cu: A CUDA-based TCP server that loads the orientation database once into GPU memory and accepts preprocessed images over a socket, returning indexed orientations in real time.
  • laue_orchestrator.py: A top-level Python orchestrator that launches the daemon, starts the image server, monitors progress with live rate and ETA, and runs post-processing automatically.
  • laue_image_server.py: Reads HDF5 files from a folder, preprocesses each frame (background subtraction → enhancement → thresholding → filtering → Gaussian blur), and sends processed images to the daemon over TCP.
  • Wire Protocol: Each frame is transmitted as a 2-byte little-endian image number followed by the full double[] pixel array, matching the daemon's handle_client receiver.
Single-Image (RunImage.py) Streaming (laue_orchestrator.py)
Orientation DB Loaded per image (~10 s) Loaded once, reused
GPU utilization Idle between images Continuous
Throughput ~1 image/min Limited only by preprocessing
Progress tracking Per-image logs Live frame_mapping.json with rate + ETA

2. Modular Python Architecture

The monolithic RunImage.py (3,553 lines) has been decomposed into focused, reusable modules — a 53% reduction in its line count:

Module Lines Purpose
RunImage.py 1,673 Core single-image pipeline (load → preprocess → index → refine → export)
laue_config.py 782 Dataclass-based configuration (LaueConfig, VisualizationConfig, OptimizerConfig) and ConfigurationManager for parsing params.txt
laue_stream_utils.py 1,108 Shared utilities: image I/O, preprocessing pipeline, TCP wire protocol, HDF5 helpers, orientation sorting and filtering
laue_visualization.py 937 8 standalone visualization functions (interactive Plotly spot maps, simulation comparison, 3D orientation views, analysis reports)

All extracted functions accept explicit parameters and require no class instantiation, making them easy to import and reuse from any script.

3. Enhanced Post-Processing (laue_postprocess.py)

The streaming post-processor has been upgraded with two key improvements:

  • Quality-Based Sorting: Filtered orientations are now automatically sorted by quality score (descending) using sort_orientations_by_quality, ensuring the best solutions appear first in output files.
  • Per-Image Interactive Visualization: A new --per-image-viz flag generates individual Plotly HTML files for each processed image, enabling detailed inspection of indexing results on a per-frame basis.

4. Scripts Directory Reorganization

All Python scripts have been moved from the repository root into a dedicated scripts/ directory:

scripts/
├── RunImage.py                   # Single-image indexing pipeline
├── laue_orchestrator.py          # Streaming pipeline entry point
├── laue_image_server.py          # H5 → preprocess → TCP sender
├── laue_postprocess.py           # Results filtering + visualization
├── laue_config.py                # Configuration dataclasses & manager
├── laue_stream_utils.py          # Shared I/O, preprocessing, TCP utilities
├── laue_visualization.py         # 8 standalone visualization functions
├── GenerateHKLs.py               # Generate valid HKL list
├── GenerateSimulation.py         # Create synthetic Laue patterns
├── ImageCleanup.py               # Pre-process raw detector images
└── README.md                     # Comprehensive CLI reference

A new scripts/README.md provides full CLI documentation for every script, including argument tables, module architecture diagrams, and usage examples.

5. Comprehensive Documentation

  • scripts/README.md: Full CLI reference for all 10 scripts, Mermaid architecture diagrams, wire protocol documentation, and shared module API summaries.
  • simulation/README.md: Updated all script paths to reflect the new scripts/ directory, added streaming mode quick-start example.
  • README.md: Expanded project structure, version history, and cross-references to the new documentation.

Quick Start — Streaming Pipeline

# Process an entire folder of H5 images through the persistent GPU daemon:
python scripts/laue_orchestrator.py \
    --config params.txt \
    --folder /path/to/h5_images/ \
    --h5-location /entry/data/data \
    --ncpus 8

This automatically launches the daemon, preprocesses and streams all images, monitors progress, and generates per-image HDF5 results with interactive Plotly HTML visualizations.

Upgrading from v1.0

  • Script paths: All scripts have moved from the repository root to scripts/. Update any shell scripts or workflows that reference ./RunImage.py to use python scripts/RunImage.py.
  • No parameter file changes: The params.txt format is fully backward compatible.
  • Build: Run ./build.sh gpu to compile the new LaueMatchingGPUStream binary alongside the existing CPU and GPU executables.
Assets 2
Loading