Conversation
Major architectural improvements across documentation, registry system, and metadata handling: ## Documentation Updates (docs/) - **api/index.rst**: Added unified_registry to API documentation with comprehensive description of the new unified registry system that eliminates 1000+ lines of code duplication - **architecture/function_registry_system.rst**: Complete rewrite documenting the new unified registry architecture including: * LibraryRegistryBase abstract class with COMMON_EXCLUSIONS and abstract attributes * ProcessingContract enum for clean contract classification * JSON-based cache system with version validation and function reconstruction * Migration details showing 22% code reduction (1050+ → 821 lines) * Performance improvements through intelligent caching and fail-loud architecture ## Core System Improvements (openhcs/core/) - **pipeline/path_planner.py**: Enhanced path planning with output plate root resolution * Added resolve_output_plate_root() static method for proper plate directory handling * Integrated output_plate_root setting in context during step planning * Improved zarr and disk backend path consistency - **steps/function_step.py**: Fixed metadata file placement for OpenHCS compatibility * Moved metadata file creation from step output directory to output plate root * Ensures openhcs_metadata.json is placed at correct hierarchical level * Added proper import organization for OpenHCSMetadataHandler * Improved error handling and logging for metadata operations These changes establish the foundation for the unified registry system while maintaining 100% backward compatibility and improving metadata handling consistency across backends.
- Extract metadata generation logic from FunctionStep to dedicated OpenHCSMetadataGenerator class - Move _create_openhcs_metadata_for_materialization, _extract_component_metadata methods to new class - Remove runtime backend detection (_detect_available_backends) - use compiler-determined info instead - Add relative path conversion for improved metadata portability - Fix circular import issue by using lazy imports in FunctionStep and openhcs.py - Replace module-level imports with local imports at call sites - Convert AVAILABLE_FILENAME_PARSERS to lazy function _get_available_filename_parsers() Architectural improvements: - Single responsibility: metadata generation separated from step execution - Proper separation of concerns: path planner handles paths, metadata handler provides backend info - Eliminates runtime filesystem inspection in favor of compiler-determined information - Breaks circular dependency chain: FunctionStep → openhcs → imagexpress → microscope_base → core
- Replace defensive try/except fallbacks with fail-loud architecture - Add OpenHCSMetadata dataclass for declarative structure definition - Remove hardcoded metadata dict construction, use asdict() for automatic conversion - Eliminate unnecessary _get_image_files() wrapper, use filemanager.list_image_files() directly - Remove hardcoded image extensions, leverage existing FileManager functionality - Replace repetitive metadata_cache conditional checks with single safe accessor pattern - Remove unnecessary type conversions (list(), float()) - trust source types - Simplify relative path conversion logic, remove redundant variables - Remove graceful degradation and silent error handling - Decompose monolithic create_metadata() into pure functions: * _extract_metadata(): fail-loud metadata extraction * _write_metadata_file(): pure I/O operation * _convert_to_relative_paths(): simplified path transformation Architectural improvements: - Fail-loud behavior: missing components cause immediate failure - Single responsibility: each method has one clear purpose - DRY principle: use existing functionality instead of reimplementing - Dataclass-driven: declarative structure with type safety - No defensive programming: no hasattr checks, fallbacks, or silent errors
…aterialization Major architectural consolidation eliminating 467 lines of duplication in path planning while adding per-step materialization capabilities and achieving GUI framework parity. Changes by functional area: * Core Pipeline Architecture: Complete path_planner.py rewrite eliminating 745 lines of defensive code, replacing with 278 lines using normalize_pattern() and extract_attributes() functions. Add materialization_config parameter to AbstractStep enabling per-step materialized output alongside memory-first processing. Implement backwards compatibility in compiler with _normalize_step_attributes() function. * Microscope Interface Standardization: Add explicit fallback mechanism to MetadataHandler ABC with FALLBACK_VALUES dict and _get_with_fallback() method. Standardize backend availability methods across ImageXpress and Opera Phenix handlers. Update OpenHCS metadata generator to use fallback-aware extraction instead of fail-loud calls. * GUI Framework Unification: Implement identical Optional[dataclass] parameter support in both PyQt and Textual frameworks with checkbox toggle widgets. Change step parameter editors to expose all AbstractStep parameters (not just FunctionStep) enabling materialization_config editing. Add dataclass type detection to widget factory. * Testing Infrastructure: Improve synthetic data Z-stack realism with separated cell rendering and fixed blur scaling. Reduce test complexity (2 channels, 3 Z-planes) and increase worker count for faster execution. Breaking changes: - DEFAULT_VARIABLE_COMPONENTS changed from single enum to list [VariableComponents.SITE] - Path planner API completely rewritten - direct usage will break - Microscope get_available_backends() return type changed from Dict to List[Backend] This consolidation eliminates defensive programming patterns while enabling granular per-step materialization and achieving true GUI framework parity.
…d well filtering, subdirectory-keyed metadata, and systematic architectural improvements
Implements comprehensive materialization system with advanced well filtering capabilities,
subdirectory-keyed metadata structure, and systematic architectural improvements across
the pipeline. Adds support for complex well filtering patterns, atomic metadata writing,
input conversion, and centralized configuration management while eliminating code
duplication and improving system reliability.
Changes by functional area:
* Core Configuration & Well Filtering System: Implement comprehensive materialization
configuration with advanced well filtering capabilities. Added MaterializationPathConfig
with lazy default resolution, WellFilterMode enum (INCLUDE/EXCLUDE), WellFilterProcessor
class supporting pattern parsing ("row:A", "col:01-06", "A01:A12", comma-separated lists),
thread-local storage for pipeline config access, and format-agnostic well filtering
that works with any microscope naming convention.
* OpenHCS Metadata & Microscope System: Complete rewrite of OpenHCS metadata handling
with subdirectory-keyed structure. Added OpenHCSMetadataFields constants,
SubdirectoryKeyedMetadata dataclass for organizing metadata by subdirectory,
AtomicMetadataWriter integration for concurrent safety, main subdirectory determination
logic, plate root resolution, and default get_available_backends implementation in
microscope base class.
* Pipeline Compilation & Path Planning: Major refactoring with unified logic and
compilation-time well filter resolution. Added well filter resolution during compilation
supporting all pattern types, input conversion detection replacing zarr conversion logic,
unified path building in PathPlanner eliminating duplication, and materialization path
building with well filtering support.
* Step Execution & Materialization: Comprehensive overhaul of step execution with enhanced
materialization support. Removed force_disk_output functionality, added input conversion
logic, enhanced metadata generation with subdirectory support, changed to dunder naming
for internal step attributes (__input_dir__, __output_dir__), and integrated materialized
metadata creation for per-step materialization.
* UI Form Management & Abstraction: Implement centralized abstraction layer for parameter
forms with lazy placeholder support. Added ParameterFormAbstraction, WidgetRegistry with
type-based widget creation, PyQt6WidgetStrategies and TextualWidgetStrategies for
framework-specific implementations, lazy default placeholder detection using introspection,
and eliminated duplicate widget creation logic across PyQt and Textual frameworks.
* Test Infrastructure: Systematic refactoring using dataclass patterns and fail-loud
validation. Added TestConstants and TestConfig dataclasses, materialization validation
functions, eliminated magic strings, and simplified test execution logic following
Systematic Code Refactoring Framework.
Breaking changes: force_disk_output parameter removed, step attributes renamed to dunder
format (__input_dir__, __output_dir__), materialization_config expects MaterializationPathConfig,
ZarrConfig store_name default changed from "images.zarr" to "images", materialization_results_path
moved to GlobalPipelineConfig, OpenHCS metadata structure changed to subdirectory-keyed format,
get_available_backends made non-abstract in MicroscopeHandler.
**Type**: `feat` | **Date**: 2025-08-13 | **Branch**: `feature/unified-registry-metadata-improvements` ## 1. File Inventory (21 files) **Core Configuration (3)**: `config.py`, `lazy_config.py` (new), `orchestrator.py` **I/O Operations (2)**: `atomic.py` (new), `metadata_writer.py` (new) **UI Abstraction (2)**: `parameter_form_abstraction.py`, `pyqt6_widget_strategies.py` **PyQt6 GUI (6)**: `main.py`, `typed_widget_factory.py`, `plate_manager.py`, `no_scroll_spinbox.py` (new), `parameter_form_manager.py`, `config_window.py` **Textual TUI (8)**: Various TUI files (excluded from commit message) ## 2. Functional Area Analysis ### Core Configuration System **Components**: LazyDataclassFactory, LazyDefaultPlaceholderService, StepMaterializationConfig, PipelineConfig, LazyStepMaterializationConfig **Patterns**: Dataclass introspection, thread-local storage with field paths, static/dynamic resolution, compositional inheritance **Dependencies**: Circular import resolution between config modules, orchestrator integration ### I/O Operations **Components**: AtomicMetadataWriter, file_lock, atomic_write_json, atomic_update_json **Patterns**: fcntl locking with timeout/polling, temp file + atomic rename, custom exception hierarchy **Dependencies**: Standard library only (fcntl, tempfile, json) ### Orchestrator Management **Components**: pipeline_config parameter, get_effective_config(), apply_pipeline_config(), clear_pipeline_config() **Patterns**: Dual configuration model (global + per-orchestrator), thread-local storage management **Dependencies**: PipelineConfig from lazy_config module ### UI Abstraction Layer **Components**: apply_lazy_default_placeholder, _get_dataclass_type, PyQt6WidgetEnhancer, MagicGuiWidgetFactory **Patterns**: Framework-agnostic abstraction, graceful degradation, fallback mechanisms **Dependencies**: Optional PyQt6 with fallbacks, both config modules ### PyQt6 GUI Implementation **Components**: ConfigWindow reset strategies, ParameterFormManager lazy nesting, TypedWidgetFactory None handling, PlateManagerWidget per-orchestrator config, NoScrollSpinBox widgets **Patterns**: Functional composition for resets, lazy dataclass creation for nested forms, wheel event prevention **Dependencies**: PyQt6 framework, lazy_config integration ## 3. Detailed Change Analysis ### Core Configuration Changes **config.py**: DefaultMaterializationPathConfig → StepMaterializationConfig (renamed, inherits PathPlanningConfig). MaterializationPathConfig moved to lazy_config.py. LazyDefaultPlaceholderService enhanced with has_lazy_resolution(), get_lazy_resolved_placeholder() with app_config support, _format_nested_dataclass_summary(). Breaking: import location changed. **lazy_config.py** (new): LazyDataclassFactory with create_lazy_dataclass(), make_lazy_thread_local(), _bind_resolution_methods(), _get_thread_local_instance(). Generated PipelineConfig and LazyStepMaterializationConfig classes. Features: dataclass introspection, static/dynamic resolution, thread-local storage with field paths. **orchestrator.py**: Added pipeline_config parameter, apply_pipeline_config(), get_effective_config(), clear_pipeline_config(). Enhanced apply_new_global_config() with thread-local updates. Implements dual configuration model. ### I/O Operations (New Files) **atomic.py**: LockConfig, FileLockError, FileLockTimeoutError classes. Functions: file_lock() context manager, atomic_write_json(), atomic_update_json(), _acquire_lock_with_timeout(), _try_acquire_lock(), _cleanup_lock(). Features: fcntl locking, timeout/polling, temp file + rename. **metadata_writer.py**: MetadataConfig, MetadataUpdateRequest, MetadataWriteError, AtomicMetadataWriter classes. Methods: update_subdirectory_metadata(), update_available_backends(), merge_subdirectory_metadata(), create_or_update_metadata(). Built on atomic.py foundation. ### UI Abstraction Enhancements **parameter_form_abstraction.py**: apply_lazy_default_placeholder() with PyQt6 import error handling and fallback. _get_dataclass_type() checks both config and lazy_config modules. Enhanced module discovery. **pyqt6_widget_strategies.py**: create_string_fallback_widget() fixes literal "None" strings. MagicGuiWidgetFactory.create_widget() prevents None→"None" conversion, type-specific defaults, post-creation clearing. PyQt6WidgetEnhancer enhanced apply_placeholder_text() with tooltip fallback. ### PyQt6 GUI Implementation Details **main.py**: handle_config_save() enhanced with thread-local storage update via set_current_pipeline_config() for MaterializationPathConfig defaults synchronization. **typed_widget_factory.py**: create_widget() adds None value handling for basic types with _create_placeholder_widget(). _create_bool_widget() handles None values. New _create_placeholder_widget() creates QLineEdit for None values with type-specific placeholders and italic styling. **plate_manager.py**: action_edit_config() implemented with per-orchestrator PipelineConfig support. Added _open_config_window(), action_edit_global_config(), _save_global_config_to_cache(). Enables dual configuration model (global vs per-orchestrator). **no_scroll_spinbox.py** (new): NoScrollSpinBox, NoScrollDoubleSpinBox, NoScrollComboBox classes override wheelEvent() to prevent accidental value changes from mouse wheel. **parameter_form_manager.py**: _create_nested_dataclass_group() enhanced with _create_lazy_nested_dataclass_if_needed() for automatic lazy loading. _handle_nested_parameter_change() preserves unchanged values in lazy pattern. update_widget_value() handles literal "None" strings. **config_window.py**: Major functional composition implementation. Added DataclassIntrospector, ResetStrategy, LazyAwareResetStrategy, FormManagerUpdater, ResetOperation classes. Enhanced parameter loading with lazy dataclass support. reset_to_defaults() uses functional pipeline for lazy-aware resets. ## 4. Cross-Cutting Patterns ### Lazy Loading Architecture **Pattern**: Generic dataclass introspection, thread-local storage with field paths, static/dynamic resolution, factory-based generation **Implementation**: lazy_config.py (core), config.py (placeholders), orchestrator.py (storage), parameter_form_abstraction.py (UI), PyQt6 forms (nested lazy creation) ### Thread-Local Storage Management **Pattern**: Explicit field path navigation, centralized orchestrator management, consistent resolution **Integration**: Orchestrator config management, lazy dataclass resolution, UI placeholder generation, PyQt6 config windows ### Error Handling & Resilience **Pattern**: Try-catch with fallbacks, custom exception hierarchies, graceful degradation **Implementation**: PyQt6 import handling, file locking timeouts, widget creation recovery, None value processing ### Import Dependency Resolution **Pattern**: Delayed imports, end-of-file imports, optional patterns **Changes**: config ↔ lazy_config circular resolution, optional PyQt6 with fallbacks, pure utility layers ### None Value Handling **Pattern**: Preserve None for lazy loading, prevent "None" string artifacts, type-specific defaults **Implementation**: Widget factories, form managers, placeholder systems, magicgui integration ## 5. Architectural Impact ### Dynamic Configuration Foundation **Capabilities**: Generic lazy loading for any dataclass, thread-local management, per-orchestrator support, atomic concurrent operations **Enables**: Dynamic updates without restart, plugin-based extensions, multi-tenant management, real-time synchronization ### Concurrency Safety Infrastructure **Safeguards**: File locking, atomic read-modify-write, timeout/retry, multiprocessing-safe operations **Benefits**: Prevents race conditions, ensures data consistency, enables safe concurrent execution, reduces corruption risks ### UI Framework Flexibility **Support**: Framework-agnostic abstraction, graceful degradation, consistent placeholders, error resilience **Improvements**: Development without full dependencies, minimal environment testing, reduced coupling, future framework additions ### PyQt6 GUI Enhancements **Capabilities**: Per-orchestrator configuration, functional composition resets, lazy nested forms, wheel event prevention **Benefits**: Dual configuration model, robust None handling, automatic lazy creation, improved UX ### Breaking Changes & Migration **Changes**: MaterializationPathConfig import location, LazyDefaultPlaceholderService API, thread-local requirements **Migration**: Update import statements, setup thread-local storage, backward compatibility maintained ### Strategic Direction **Patterns**: Generic factory-based lazy loading, atomic concurrency operations, dual configuration models, clean contract separation **Positioning**: Plugin architecture enablement, multi-tenant support, distributed processing preparation, dynamic reconfiguration facilitation ## 6. Regression Identified **StepMaterializationConfig Step Editor Saving Regression**: The step editor can no longer save step instance values due to MaterializationPathConfig being replaced with LazyStepMaterializationConfig that requires thread-local storage setup. This regression occurred during lazy configuration generalization work for PipelineConfig in the plate manager.
- Load existing pipeline_config when reopening config window instead of always creating fresh instance
- Preserve None values for unset fields to maintain 'Pipeline default: {value}' placeholder behavior
- Track user modifications in config window to only save explicitly changed values
- Ensure thread-local context is properly set when loading existing configs
- Fixes issue where saved config values were showing resolved defaults instead of placeholders
…lay issues Fixes critical architectural issues in the lazy configuration system that caused configuration windows to display empty fields instead of default values and created confusing dual codepaths in lazy dataclass creation. **Core Architecture Changes:** - Implement intelligent field type preservation in lazy dataclass creation - Replace blanket Optional type conversion with default-aware type analysis - Unify field introspection logic across create_lazy_dataclass() and make_lazy_thread_local() - Add sophisticated Optional vs non-Optional field detection using Union type introspection **PyQt6 Configuration Integration:** - Enable lazy loading support in main window global configuration (main.py) - Integrate lazy PipelineConfig wrapper with proper thread-local context management - Implement config conversion pipeline using to_base_config() for save operations - Enhance plate manager with unified orchestrator and global config lazy loading **UI Field Value Resolution:** - Fix config window parameter loading to handle Optional vs non-Optional fields correctly - Optional fields use stored values (None) for placeholder behavior - Non-Optional fields use resolved values to display actual defaults - Preserve lazy loading semantics during configuration save/load cycles **Form Management Improvements:** - Add comprehensive debug instrumentation for field type detection - Fix UnboundLocalError in parameter form manager logger initialization - Enhance placeholder application debugging across UI frameworks - Improve error diagnostics for configuration rendering issues **Technical Details:** - Fields with default values/factories preserve original types (Path remains Path, not Optional[Path]) - MISSING sentinel used for accurate default value detection - Thread-local resolution context properly maintained across configuration workflows - Backward compatibility preserved through existing interface maintenance **Files Modified:** - openhcs/core/lazy_config.py: Core field type preservation logic - openhcs/pyqt_gui/main.py: Main window lazy config integration - openhcs/pyqt_gui/widgets/plate_manager.py: Plate manager config unification - openhcs/pyqt_gui/widgets/shared/parameter_form_manager.py: Form field debugging - openhcs/pyqt_gui/windows/config_window.py: Config window value resolution - openhcs/ui/shared/parameter_form_abstraction.py: Placeholder debugging **Impact:** - Resolves global configuration window showing empty fields instead of defaults - Eliminates architectural confusion from dual lazy dataclass creation codepaths - Enables proper placeholder behavior for Optional vs non-Optional fields - Maintains full backward compatibility with existing configuration workflows - Provides comprehensive debugging capabilities for configuration system issues **Testing:** - Global config window (Main → Tools → Configuration) now shows default values - Orchestrator config window (Plate Manager → Edit Config) maintains placeholder behavior - Configuration changes persist correctly across save/load cycles - Thread-local lazy resolution works correctly in all configuration contexts Resolves the core lazy loading architecture issues that caused significant debugging overhead during configuration system development and ensures consistent, predictable behavior across all PyQt6 configuration interfaces.
…hitecture Apply systematic refactoring framework to transform lazy configuration system from defensive programming patterns to clean, fail-loud Pythonic implementation. ARCHITECTURAL IMPROVEMENTS: - Implement Strategy Pattern: Extract ResolutionStrategy hierarchy with StaticResolutionStrategy and ThreadLocalResolutionStrategy - Method Factory Pattern: Create LazyMethodFactory for clean method creation - Unified Field Introspection: Consolidate ~40 lines of duplicated logic into single _introspect_dataclass_fields() method - Field Path Navigation: Centralize navigation in FieldPathNavigator utility CODE QUALITY ENHANCEMENTS: - Magic String Elimination: Extract all hardcoded strings to LazyConfigConstants - Comprehensive Type Annotations: Add complete type hints to all methods - Fail-Loud Implementation: Remove defensive try/catch fallback in thread-local resolution - now fails immediately with clear AttributeError when misconfigured - Method Consolidation: Replace dual codepaths with unified _create_lazy_dataclass_unified() MAINTAINABILITY IMPROVEMENTS: - Remove deprecated create_lazy_dataclass_with_generic_thread_local_resolver() - Pluggable resolution strategies enable easy extension - Clear separation of concerns between strategies and factories - Reusable abstractions for future lazy dataclass needs BACKWARD COMPATIBILITY: - All external interfaces preserved (PipelineConfig, LazyStepMaterializationConfig) - Factory method signatures unchanged - Utility functions continue to work as expected - 100% functional compatibility maintained This refactoring eliminates defensive programming anti-patterns, applies the fail-loud philosophy correctly, and creates elegant reusable abstractions while maintaining perfect backward compatibility.
… constants Remove unnecessary strategy pattern abstraction and unused constants for maximum simplicity while maintaining 100% functionality. STRATEGY PATTERN ELIMINATION: - Remove ResolutionStrategy ABC and concrete strategy classes - Replace with direct instance provider functions - much simpler - Both resolution types now use same core pattern: getattr(instance_provider(), field_name) - Eliminate artificial distinction between 'static' and 'thread-local' resolution - No loss of functionality or flexibility - any instance provider function works CONSTANTS CLEANUP: - Remove GLOBAL_PIPELINE_CONFIG_NAME and STEP_MATERIALIZATION_CONFIG_NAME (unused) - Remove DEPRECATED_METHOD_WARNING_TEMPLATE (deprecated method was removed) - Remove MISSING_IMPORT_NAME (never used) - Keep only constants that are actually used in the codebase ARCHITECTURAL BENEFITS: - Simpler code: No abstract classes, no strategy hierarchy - Easier to understand: Direct function creation instead of pattern indirection - Less cognitive load: One concept (instance provider) instead of multiple - Same flexibility: Any function that returns an instance can be used - Perfect backward compatibility: All external interfaces unchanged YAGNI PRINCIPLE APPLIED: - Don't need strategy pattern if we only have one strategy - Sometimes simple functions are better than design patterns - The instance provider function IS the strategy - no wrapper needed This demonstrates how systematic refactoring can reveal that the 'right' design pattern is sometimes no pattern at all. The simplest solution that works is often the best solution.
…zy/concrete state management
Transform pipeline-specific lazy configuration into generic framework supporting any dataclass type
with placeholder handling, mixed lazy/concrete state management, and UI form generation. Finalizes
threadlocal lazydataclass form generation pattern after resolving bugs in placeholder behavior,
state management, and config rebuilding logic.
Changes by functional area:
* Core Configuration System (config.py +109, lazy_config.py +440, pipeline_config.py +136 NEW):
- Generalize thread-local storage: `_global_config_contexts: Dict[Type, threading.local] = {}`
- Add type registry: `register_lazy_type_mapping()`, `get_base_type_for_lazy()` functions
- New generic functions: `set_current_global_config()`, `get_current_global_config()`
- Replace strategy pattern with ResolutionConfig and LazyMethodBindings dataclasses
- Add `rebuild_lazy_config_with_new_global_reference()` for config rebuilding
- Extract pipeline-specific logic to new openhcs.core.pipeline_config module
- Enhance LazyDefaultPlaceholderService with configurable placeholder prefixes
- Add `force_static_defaults` parameter for global config editing context
* Orchestration Layer (orchestrator.py +57):
- Enhance apply_new_global_config() with lazy config rebuilding workflow:
1. Update global config reference
2. Rebuild orchestrator-specific config preserving user values
3. Re-initialize components if already initialized
- Use `rebuild_lazy_config_with_new_global_reference()` for state preservation
- Update thread-local storage calls to use generic `set_current_global_config()`
- Add component re-initialization logic with proper state management
* UI Abstraction Layer (parameter_form_abstraction.py +184, pyqt6_widget_strategies.py +402):
- Implement 3-step placeholder fallback chain:
1. LazyDefaultPlaceholderService for special lazy dataclasses
2. Thread-local resolution for regular dataclasses
3. Static defaults fallback
- Add PlaceholderConfig dataclass with styling constants and interaction hints
- Create WIDGET_PLACEHOLDER_STRATEGIES mapping for declarative widget handling
- Add widget-specific strategies: QCheckBox, QComboBox, QSpinBox, QDoubleSpinBox
- Implement placeholder state management with `is_placeholder_state` property
- Add automatic placeholder clearing on user interaction
- Enhance enum matching with robust fallback strategies
* PyQt GUI Components (parameter_form_manager.py +583/-75, config_window.py +79/-71,
plate_manager.py +36/-89, main.py +12/-17, enhanced_path_widget.py +8/-2,
step_parameter_editor.py +5/-1):
- Add NoneAwareLineEdit class with get_value()/set_value() methods
- Implement mixed lazy/concrete state management with field-level granularity
- Add context parameters: is_global_config_editing, global_config_type, placeholder_prefix
- Create _get_field_path_for_nested_type() for automatic field path determination
- Add _should_use_concrete_nested_values() logic for mixed state support
- Implement reset_parameter_by_path() with dot notation support (e.g., 'path_planning.output_dir_suffix')
- Add _rebuild_nested_dataclass_from_manager() with lazy vs concrete logic
- Create extensive debugging infrastructure for nested parameter updates
- Add LazyAwareResetStrategy that resolves to actual static defaults
- Simplify global config editing to use concrete GlobalPipelineConfig
* Shared Textual Components (signature_analyzer.py +144, parameter_form_manager.py +363):
- Add AnalysisConstants dataclass: INIT_METHOD_SUFFIX, SELF_PARAM, CLS_PARAM, DUNDER_PREFIX
- Implement automatic constructor detection using `__qualname__.endswith(".__init__")`
- Add skip_first_param parameter with auto-detection logic
- Create _get_field_path_for_nested_type() for type introspection
- Add _should_use_concrete_nested_values() mirroring PyQt logic
- Implement context-aware reset with _get_reset_value_for_parameter()
- Add extensive debugging for path_planning and output_dir_suffix parameters
- Create reset_parameter_by_path() for dot notation support
Technical innovations:
- Mixed Lazy/Concrete State Management:
* Individual fields within same dataclass can be lazy (None) or concrete
* Field-level granularity: user-set values remain concrete, unset fields remain lazy
* Supports mixed states within nested dataclasses (e.g., path_planning.output_dir_suffix)
* Context-aware creation: lazy instances for orchestrator editing, concrete for global editing
- Type Introspection Architecture:
* Automatic field path determination through GlobalPipelineConfig field annotation inspection
* Eliminates hardcoded string mappings with type matching algorithms
* Frame inspection for locally defined dataclasses (test support)
* Generic dataclass discovery across multiple modules (config, lazy_config)
- Context-Aware Behavior System:
* Global config editing: static defaults, concrete values, immediate materialization
* Orchestrator config editing: lazy placeholders, thread-local resolution, inheritance hierarchy
* Proper distinction maintained throughout UI layer with is_global_config_editing parameter
* Different placeholder prefixes: "Default" vs "Pipeline default"
- Declarative Placeholder System:
* Widget-specific strategies: _apply_checkbox_placeholder(), _apply_combobox_placeholder()
* Visual feedback with strong styling that overrides application themes
* Interaction hints: "click to set your own value", "select to set your own value"
* Automatic state clearing on user interaction with placeholder state tracking
* Robust enum matching with multiple fallback strategies (name, value, display text)
- Generic Lazy Configuration Framework:
* Thread-local storage supports any dataclass type through configurable global_config_type
* Type registry system maps lazy classes to base classes
* Recursive resolution with configurable fallback chains
* Not limited to pipeline configurations - extensible to any global config type
- Unified Form Management Architecture:
* PyQt and Textual components share identical logic through parameter form abstraction
* Textual components reused by PyQt for compatibility and consistency
* Common widget creation, placeholder application, and state management
* Consistent reset behavior across both UI frameworks
Backward compatibility and migration:
- Maintains existing imports through wrapper functions in config.py
- PipelineConfig import moved from lazy_config to config module
- All existing orchestrator and UI code continues to work without changes
- Gradual migration path for extending to new configuration types
Debugging and development infrastructure:
- Extensive logging for path_planning and output_dir_suffix parameter updates
- Debug infrastructure for nested manager synchronization
- Detailed state tracking for mixed lazy/concrete behavior
- Call stack logging for troubleshooting complex parameter update chains
Code quality improvements:
- Eliminates magic strings: AnalysisConstants, PlaceholderConfig dataclasses
- Replaces defensive programming with fail-loud validation and clear error messages
- Implements Pythonic patterns: dataclasses, functional composition, type hints
- Reduces cognitive load through declarative configuration and abstraction layers
- Centralizes constants and configuration in dataclass patterns
Enables extension to any global configuration type beyond pipeline configs while providing more
robust UI form generation with proper state management, eliminates architectural debt, and
establishes foundation for future configuration system extensions.
Files changed: 20 files (1 new, 19 modified)
Lines changed: +2,272 insertions, -634 deletions
```
## Key Reductions Made
**Removed verbose descriptions:**
- Eliminated repetitive "sophisticated," "comprehensive," "enhanced" qualifiers
- Condensed technical implementation details while preserving core functionality
- Removed redundant explanations and excessive technical depth
- Streamlined bullet points to focus on essential changes
**Preserved essential information:**
- All functional area changes documented
- Technical innovations and architectural patterns maintained
- Code quality improvements retained
- Impact and extensibility explained
- File and line change statistics included
- Move scripts/migrate_legacy_metadata.py to openhcs/io/metadata_migration.py - Update openhcs/io/__init__.py to export migration functions - Integrate with existing metadata infrastructure (METADATA_CONFIG) - Enable programmatic access: detect_legacy_format, migrate_plate_metadata - Maintain CLI functionality via python -m openhcs.io.metadata_migration
Move the compile_pipelines method from PipelineOrchestrator to PipelineCompiler to improve separation of concerns and architectural clarity. Changes: - Add PipelineCompiler.compile_pipelines() static method with orchestrator injection - Simplify PipelineOrchestrator.compile_pipelines() to delegate to compiler - Move resolve_lazy_dataclasses_for_context() method to proper location in compiler - Add local imports for GroupBy, OrchestratorState, and StepAttributeStripper - Update path_planner.py comment for materialization_config resolution Benefits: - Better separation of concerns: compilation logic belongs in compiler - Cleaner architecture: orchestrator focuses on orchestration - Reusable compilation logic that can be used independently - Maintains backward compatibility with same public API - No functional changes to existing behavior The orchestrator now simply injects itself as a parameter to the compiler's compile_pipelines method, allowing the compiler to access orchestrator methods like create_context() and get_component_keys() while keeping all compilation logic properly contained within the compiler module.
trissim
changed the title
trissim
added a commit
that referenced
this pull request
Oct 4, 2025
Comprehensive documentation of all fixes: - Bug #1: OMERO data directory path (CRITICAL) - Bug #2: Docker Compose version warning - Bug #3: Execution server default path - Bug #4: Demo script path Includes: - Problem descriptions - Solutions implemented - Code changes - Technical details of dual-mode architecture - Migration guide - Testing status
popjell
pushed a commit
to popjell/openhcs
that referenced
this pull request
Oct 15, 2025
Comprehensive documentation of all fixes: - Bug OpenHCSDev#1: OMERO data directory path (CRITICAL) - Bug OpenHCSDev#2: Docker Compose version warning - Bug OpenHCSDev#3: Execution server default path - Bug OpenHCSDev#4: Demo script path Includes: - Problem descriptions - Solutions implemented - Code changes - Technical details of dual-mode architecture - Migration guide - Testing status
popjell
pushed a commit
to popjell/openhcs
that referenced
this pull request
Oct 15, 2025
Comprehensive documentation of all fixes: - Bug OpenHCSDev#1: OMERO data directory path (CRITICAL) - Bug OpenHCSDev#2: Docker Compose version warning - Bug OpenHCSDev#3: Execution server default path - Bug OpenHCSDev#4: Demo script path Includes: - Problem descriptions - Solutions implemented - Code changes - Technical details of dual-mode architecture - Migration guide - Testing status
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final Pull Request Description
Comprehensive Architectural Improvements: Registry, Metadata, Configuration, and Materialization Systems
This pull request implements comprehensive architectural improvements across OpenHCS's core systems through 13 commits, focusing on code consolidation, type safety, extensible configuration management, and advanced materialization capabilities.
Executive Summary
Branch:
feature/unified-registry-metadata-improvementsCommits: 13 (683ff90..336b547)
Files Changed: 26 files (1 new)
Impact: Registry consolidation, metadata system rewrite, generalized configuration framework, and comprehensive materialization system
Architectural Changes
Unified Registry System
OpenHCS Handler and Metadata System Refactoring
{"subdirectories": {"step1_output": {...}, "step2_output": {...}}}) to prevent conflicts (commit 604c5cd)Metadata Architecture
Lazy Configuration Framework
FunctionStep Refactoring
Materialization System
Architectural Separation of Concerns
compile_pipelinesmethod from PipelineOrchestrator to PipelineCompiler for better separation of concerns (commit 336b547)Technical Implementation
Mixed State Management
Configuration fields can be in two states within the same dataclass instance:
output_dir="/custom/path"(concrete) whilemax_workers=Noneinherits from global config (lazy)Lazy Dataclass Pickling Resolution
Critical for multiprocessing compatibility:
resolve_lazy_dataclasses_for_context()method converts all lazy instances to base configurations before picklingto_base_config()method for resolutionType Introspection Architecture
path_planning.output_dir_suffix) without hardcoded mappingsUI System
Code Quality Changes
Migration and Compatibility
Backward Compatibility
Breaking Changes (Internal APIs)
Performance Improvements
Testing and Validation
Future Capabilities
This architectural foundation enables:
Summary
This pull request implements comprehensive architectural improvements across registry consolidation, metadata system rewrite, generalized configuration framework, and advanced materialization capabilities. The changes eliminate code duplication, improve type safety, and provide extensible architecture while maintaining backward compatibility.
The 13 commits represent a systematic approach to architectural improvement, with each building upon previous work to create a cohesive, enterprise-grade system for future OpenHCS development.
Pull Request opened by Augment Code with guidance from the PR author