Skip to content

DCAR Report #3 ‐ 2025‐12‐15

Jump to bottom
Pierre Raybaut edited this page Dec 15, 2025 · 1 revision

Report #3 - Date: 2025-12-15

Scope of the report

This report summarizes the completion of the DataLab Core Architecture Redesign (DCAR) project, validated by the release of Sigima V1.0.0 (2025-10-28) and DataLab V1.0.0 (2025-11-16/2025-11-18).

This report covers the following tasks that remained after DCAR Report #2:

  • Task 3: Documentation and Training Materials
    • 3b. Write getting started guides and user tutorials
  • Task 4: Package Separation and Release
    • 4a. Extract Sigima as a standalone package
    • 4b. Set up independent CI/CD pipelines
    • 4c. Publish Sigima on PyPI
  • Task 5: DataLab Integration and Release
    • 5a. Finalize DataLab integration with Sigima
    • 5b. Update DataLab documentation
    • 5c. Release DataLab V1.0.0

Executive Summary

The DCAR project has been successfully completed with the release of:

  • Sigima V1.0.0 (2025-10-28): A standalone, GUI-independent signal and image processing library
  • DataLab V1.0.0 (2025-11-16/2025-11-18): A complete rewrite using Sigima as its processing backend

These releases represent a fundamental architectural transformation:

  • Core library extraction: 100% of signal/image processing logic migrated to Sigima
  • API stability: Public API stabilized and documented
  • Independent releases: Separate versioning, testing, and release cycles
  • Production ready: Both packages deployed to PyPI and validated in production

The following table collects the different web pages related to this report:

Project PyPI Repository Documentation Release Tag
Sigima Sigima on PyPI Sigima Repository Sigima Documentation v1.0.0
DataLab DataLab on PyPI DataLab Repository DataLab Documentation v1.0.0

ℹ️ Note: Patches were released for both packages after the initial V1.0.0 releases to address minor issues discovered in production.

Milestone Status for Report #3

✅ 3b. Write getting started guides and user tutorials

Comprehensive documentation has been completed for both packages:

Sigima Documentation:

  • Getting Started guide with installation instructions
  • API documentation with complete module reference
  • Contributing guidelines
  • Release notes tracking all changes
  • Available at sigima.readthedocs.io

DataLab Documentation:

  • Updated user manual reflecting the V1.0 architecture
  • Migration guide for users upgrading from V0.x
  • API documentation for remote control and plugin development (updated)
  • Comprehensive tutorials and examples (updated)

✅ 4a. Extract Sigima as a standalone package

Sigima has been successfully extracted as an independent package with its own repository:

Repository Structure:

  • Independent Git repository at github.com/DataLab-Platform/Sigima
  • Complete package structure with setup.py, pyproject.toml, and modern Python packaging
  • Separated from DataLab with no circular dependencies
  • Module renamed from internal sigima_ to public sigima

Core Architecture:

sigima/
├── client/             # Remote control client interface (DataLab)
├── data/               # Test data and resources
├── io/                 # I/O operations and plugin system
├── locale/             # Internationalization
├── objects/            # Core data model (Signal, Image, ROI)
├── proc/               # High-level computation functions
│   ├── image/          # Image processing
│   ├── signal/         # Signal processing
│   ├── base.py         # Base processor classes
│   ├── decorator.py    # Computation function decorators
│   └── validation.py   # Validation infrastructure
├── tests/              # Unit and integration tests
├── tools/              # Low-level NumPy algorithms
│   ├── image/          # Image algorithms
│   ├── signal/         # Signal algorithms
│   ├── checks.py       # Input validation decorators
│   ├── coordinates.py  # Coordinate transformations
│   └── datatypes.py    # Data type utilities
├── config.py           # Library options system
├── enums.py            # Enum definitions
├── params.py           # Parameter classes (DataSet)
└── worker.py           # Background execution interface

Key Features Achieved:

  • Zero GUI dependencies (no Qt, no matplotlib for core functionality)
  • Pure NumPy/SciPy computation layer
  • Pluggable I/O system supporting multiple file formats
  • Type hints throughout for better IDE support
  • Comprehensive validation and testing infrastructure:
    • DataLab: 82% coverage, with 136 high-level tests
    • Sigima: 89% coverage, with 573 middle/low-level tests
    • 100% automated testing in CI/CD pipelines

✅ 4b. Set up independent CI/CD pipelines

Both packages now have complete, independent CI/CD infrastructure:

Sigima CI/CD:

  • Automated testing on Linux (Ubuntu)
  • Multiple Python versions (3.9, 3.13)
  • Automated PyPI releases on tagged commits
  • Documentation builds on Read the Docs

DataLab CI/CD:

  • Independent pipeline testing DataLab with Sigima as a dependency
  • Platform-specific installer (Windows MSI)
  • Automated release workflow (local builds)
  • Separate versioning strategy

Benefits:

  • Sigima can release independently of DataLab
  • DataLab can pin Sigima versions for stability
  • Faster iteration cycles for bug fixes
  • Clear separation of concerns for maintenance

✅ 4c. Publish Sigima on PyPI

Sigima V1.0.0 was successfully published to PyPI on 2025-10-28:

Package Details:

  • Package name: Sigima
  • PyPI URL: pypi.org/project/Sigima
  • Installation: pip install Sigima
  • Dependencies clearly specified (NumPy, SciPy, scikit-image, etc.)

Version History:

  • V1.0.0 (2025-10-28): Initial standalone release
  • V1.0.1 (2025-11-05): Detection ROI creation, annotations API
  • V1.0.2 (2025-11-12): New parametric images, signal analysis improvements
  • V1.0.3 (2025-12-03): Signal calibration, data type validation
  • V1.0.4: Grid ROI spacing, signal result formatting

Distribution:

  • Source distribution (sdist)
  • Wheel distribution for faster installation
  • Compatible with pip
  • Conda package available via conda-forge here

✅ 5a. Finalize DataLab integration with Sigima

DataLab V1.0.0 represents a complete integration with Sigima:

Architecture Transformation:

Before DCAR (V0.x):

cdl/
├── algorithms/     # NumPy operations intertwined with Qt
├── computation/    # Tightly coupled processing
├── core/           # Mixed GUI and computation logic
├── gui/            # Pure GUI application layer
├── ...

After DCAR (V1.0):

datalab/
├── adapters_metadata/  # Sigima → DataLab metadata
├── adapters_plotpy/    # Sigima → PlotPy visualization
├── control/            # Remote control API
├── gui/                # Pure GUI application layer
├── ...

All signal/image processing now delegates to Sigima:

  • No duplicate computation logic
  • Clear separation between GUI and algorithms
  • DataLab focuses on user experience and visualization
  • Sigima handles all scientific computation

✅ 5b. Update DataLab documentation

DataLab documentation has been completely updated for V1.0:

Documentation Updates:

  • User manual revised to reflect new architecture
  • API documentation for plugin developers updated with Sigima references
  • Remote control documentation showing Sigima object manipulation
  • Migration notes for users upgrading from V0.x
  • Architecture diagrams showing DataLab-Sigima relationship

Key Documentation Sections:

  • Installation guide (including Sigima as dependency)
  • Getting started tutorials
  • Feature reference (processing, analysis, visualization)
  • Plugin development guide using Sigima API
  • Remote control examples
  • Troubleshooting guide

✅ 5c. Release DataLab V1.0.0

DataLab V1.0.0 was released on 2025-11-16, with a patch release V1.0.1 on 2025-11-18:

V1.0.0 (2025-11-16):

  • First production release with Sigima backend
  • Complete architectural redesign validated
  • All features from V0.x preserved and enhanced
  • New features leveraging Sigima capabilities

V1.0.1 (2025-11-18):

  • Patch release fixing Windows installer coexistence with V0.x
  • No functional changes from V1.0.0

Post-Release Updates:

  • V1.0.2 (2025-12-03): Signal axis calibration, bug fixes
  • V1.0.3: Additional refinements and stability improvements

Release Artifacts:

  • Windows installer (MSI)
  • Standalone executables
  • PyPI package: datalab-platform
  • Source code releases on GitHub
  • Conda package via conda-forge here

Major Achievements Introduced in V1.0.0 Releases

Sigima V1.0.0 Highlights

1. Comprehensive Signal Processing:

  • DateTime support for time-series data
  • Advanced curve fitting framework (Gaussian, Lorentzian, Voigt, multi-peak)
  • Brick wall filtering
  • Signal-to-image conversion for spectrograms
  • Horizontal/vertical projections

2. Advanced Image Processing:

  • Non-uniform coordinate support
  • Convolution and deconvolution
  • 2D resampling
  • New parametric image generators (checkerboard, sinc, etc.)
  • Phase (argument) extraction for complex data

3. Robust I/O System:

  • HDF5 format support (.h5sig, .h5ima)
  • Coordinated text image files
  • CSV with automatic datetime detection
  • Extensible plugin architecture

4. Enhanced ROI Features:

  • Inverse ROI functionality (inside parameter)
  • Grid ROI with configurable spacing
  • Detection ROI creation for analysis functions
  • Improved coordinate handling

5. Result System Redesign:

  • New TableResult and GeometryResult types
  • Computation-oriented, GUI-independent
  • Better modularity and testability
  • Automatic function name injection

DataLab V1.0.0 Highlights

1. Architecture Benefits:

  • Clean separation of concerns (GUI vs. computation)
  • Faster development cycles (GUI changes don't affect algorithms)
  • Better testability (unit tests for computation, integration tests for GUI)
  • Easier maintenance (bug fixes in one layer don't affect the other)

2. New Features Enabled by Sigima:

  • Signal axis calibration ("Replace X by other signal's Y")
  • Grid ROI with spacing parameters
  • Detection ROI creation workflow
  • Improved result visualization

3. Stability and Quality:

  • Comprehensive bug fixes across all areas
  • Better error handling and user feedback
  • Improved performance through optimized Sigima backend
  • Enhanced cross-platform compatibility

4. Developer Experience:

  • Remote control API with Sigima objects
  • Better documentation and examples

Validation of DCAR Project Goals

The DCAR project had four primary goals, all of which have been validated:

Goal 1: Decouple Core Logic from GUI ✅

Achieved:

  • 100% of signal/image processing migrated to Sigima
  • Zero GUI dependencies in core library
  • Clear API boundaries between layers
  • Independent testing and validation

Evidence:

  • Sigima can be used standalone without DataLab
  • Sigima has its own test suite (~90% coverage)
  • No Qt or GUI imports in sigima package (except for optional visualization purposes in interactive tests)
  • Used successfully in non-GUI contexts (scripts, notebooks)

Goal 2: Enable Reuse in Other Contexts ✅

Achieved:

  • Sigima published as independent PyPI package
  • Can be integrated into any Python application
  • Used in Jupyter notebooks for data analysis

Evidence:

  • Simple API: from sigima import SignalObj, ImageObj
  • No DataLab dependency required
  • Clear documentation for standalone usage
  • Example scripts demonstrating various use cases

Goal 3: Improve Maintainability ✅

Achieved:

  • Separate repositories with independent release cycles
  • Clear ownership and contribution guidelines
  • Better code organization and modularity
  • Reduced coupling between components

Evidence:

  • Sigima released 4 patch versions independently (1.0.1-1.0.4)
  • DataLab released 3 versions using stable Sigima
  • Bug fixes can target specific layer
  • Separate issue tracking and development workflows

Goal 4: Enhance Testing and Quality ✅

Achieved:

  • Comprehensive test suites for both packages
  • Separate validation infrastructure
  • Automated CI/CD pipelines
  • High code coverage (80%+ for both packages)

Evidence:

  • Sigima: ~90% coverage with pure algorithmic tests
  • DataLab: ~80% coverage with integration tests
  • Automated testing on multiple platforms/Python versions
  • Production validation through successful releases

Impact Assessment

For Users

Immediate Benefits:

  • More stable DataLab with faster bug fix cycles
  • Better performance through optimized Sigima backend
  • Richer feature set from focused development

Long-term Benefits:

  • Ability to automate workflows using Sigima directly
  • Integration with custom applications and pipelines
  • Future-proof architecture supporting advanced features

For Developers

Plugin Development:

  • Clear API for extending DataLab functionality
  • Direct access to Sigima for custom algorithms
  • Better documentation and examples

Contributions:

  • Easier to contribute (can target Sigima or DataLab)
  • Clear separation of concerns guides implementation
  • Comprehensive test infrastructure validates changes

Reuse:

  • Sigima algorithms available for any Python project
  • No need to install full DataLab for computation needs
  • Well-documented API for integration

For the Project

Sustainability:

  • Modular architecture supports long-term evolution
  • Independent versioning allows experimentation
  • Clear boundaries reduce maintenance burden

Growth:

  • Sigima can grow user base beyond DataLab
  • DataLab can focus on UX improvements
  • Both projects benefit from specialized contributions

Future Outlook

While the DCAR project is complete, the architecture enables exciting future developments:

Short-term (Next 6 months):

  • Expand Sigima feature set based on community feedback
  • Enhance DataLab visualization capabilities
  • Improve plugin ecosystem

Medium-term (6-12 months):

  • Sigima integration in user-specific applications
  • Advanced batch processing workflows

Long-term (12+ months):

  • Machine learning integration in Sigima
  • Distributed computing support
  • Real-time data processing features

Conclusion

The DataLab Core Architecture Redesign (DCAR) project has successfully achieved all its objectives:

✅ Core extraction (Tasks 1a-1d): Complete separation of computation from GUI
✅ Validation (Tasks 2a-2b): Comprehensive testing infrastructure with high coverage
✅ Documentation (Tasks 3a-3b): Complete API docs, user guides, and tutorials
✅ Package separation (Tasks 4a-4c): Independent Sigima package published on PyPI
✅ Integration (Tasks 5a-5c): DataLab V1.0.0 successfully using Sigima backend

Code References for Report #3

Sigima:

DataLab:

Related Reports:

This report marks the successful completion of the DataLab Core Architecture Redesign project. The architectural foundation established through this effort positions both Sigima and DataLab for continued growth and innovation.

Clone this wiki locally