ADR suggestion ModelCollection for List-Based Model Management

[rozyczko]

ModelCollection for List-Based Model Management

Status

Proposed

Context

The EasyScience corelib provides base classes for building scientific models, including ModelBase which extends from NewBase. However, there was no dedicated collection class that combines the list-like functionality of Python's built-in containers with the EasyScience features such as:

• Interface propagation to contained items
• Graph-based dependency tracking
• Serialization support
• Parameter management across collections

Users who needed to manage collections of model objects had to either:

1. Use Python's built-in list types, losing EasyScience features
2. Use the existing CollectionBase which is based on BasedBase (older architecture)
3. Create custom collection implementations for each use case

This gap in the architecture made it difficult to create clean, maintainable code when working with multiple related model objects that need to share interfaces and participate in the EasyScience ecosystem.

Decision

We have decided to introduce a new ModelCollection class that:

1. Extends ModelBase: Inherits all EasyScience features from ModelBase
2. Implements MutableSequence: Provides full list-like functionality through Python's collections.abc.MutableSequence interface
3. Supports Interface Propagation: Automatically propagates interface bindings to all items in the collection
4. Maintains Graph Edges: Properly manages graph relationships through the global object map
5. Enables Type Safety: Uses Python's typing system with generics (TypeVar) for better IDE support
6. Provides Multiple Access Methods: Supports indexing by integer, slice, name, or unique_name

Key Design Decisions

1. Inheritance from ModelBase and MutableSequence

class ModelCollection(ModelBase, MutableSequence[T]):

This dual inheritance provides:

• ModelBase features: serialization, parameter management, graph tracking
• MutableSequence interface: standard Python list operations (append, insert, pop, etc.)

2. Interface Propagation

The collection automatically propagates its interface to all contained items:

@interface.setter
def interface(self, new_interface: InterfaceType) -> None:
    self._interface = new_interface
    for item in self._data:
        if hasattr(item, 'interface'):
            setattr(item, 'interface', new_interface)

This ensures that when a collection's interface changes, all items are updated automatically.

3. Graph Edge Management

All collection operations (add, remove, replace) properly maintain graph edges:

def _add_item(self, item: Any) -> None:
    if not isinstance(item, NewBase):
        raise TypeError(f'Items must be NewBase objects, got {type(item)}')
    if item in self._data:
        return  # Skip duplicates
    self._data.append(item)
    self._global_object.map.add_edge(self, item)
    self._global_object.map.reset_type(item, 'created_internal')

4. Multiple Access Patterns

Support for accessing items by:

• Integer index: collection[0]
• Slice: collection[0:2]
• Name: collection['item_name']
• Unique name: collection[unique_id]

5. Type Safety with Generics

Using TypeVar and @overload decorators for precise type hints:

T = TypeVar('T', bound=NewBase)

@overload
def __getitem__(self, idx: int) -> T: ...
@overload
def __getitem__(self, idx: slice) -> 'ModelCollection[T]': ...
@overload
def __getitem__(self, idx: str) -> T: ...

Consequences

Positive

1. Unified API: Users get familiar list operations while maintaining EasyScience features
2. Type Safety: Better IDE support and static analysis through comprehensive type hints
3. Automatic Interface Management: Interface changes propagate automatically to all items
4. Clean Architecture: Separation of concerns with _add_item and _remove_item helpers
5. Backward Compatible: Doesn't break existing code, adds new functionality
6. Well-Tested: Comprehensive test suite (102 tests covering all functionality)
7. Flexible Access: Multiple ways to access items (by index, name, or unique_name)

Negative

1. Additional Complexity: Users need to understand when to use ModelCollection vs CollectionBase
2. Memory Overhead: Graph edge tracking adds some memory overhead
3. Learning Curve: New users need to understand the interface propagation mechanism
4. Type Checking Limitations: Some dynamic operations may not be fully type-checked

Neutral

1. Migration Path: Existing code using plain lists can be migrated gradually
2. Documentation Needs: Requires documentation and examples for effective use
3. CollectionBase Relationship: Need to clarify relationship with existing CollectionBase

Implementation Details

File Changes

Based on PR #180, the implementation includes:

1. New File: src/easyscience/base_classes/model_collection.py (278 lines)

   • Core ModelCollection class implementation
   • Complete MutableSequence interface
   • Interface propagation logic
   • Serialization support

2. Modified: src/easyscience/base_classes/__init__.py

   • Export ModelCollection from base_classes module

3. Modified: src/easyscience/base_classes/collection_base.py

   • Update to accept NewBase objects (not just BasedBase)

4. New File: tests/unit_tests/base_classes/test_model_collection.py (953 lines)

   • 102 comprehensive unit tests
   • Tests for all MutableSequence operations
   • Interface propagation tests
   • Graph edge management tests

Key Methods

• __init__: Initialize collection with optional items and interface
• __getitem__: Support int, slice, and string (name) indexing
• __setitem__: Replace items while maintaining graph edges
• __delitem__: Remove items and clean up graph edges
• insert: Add items at specific positions
• interface property: Get/set interface with propagation
• get_all_variables: Collect variables from all items
• sort: Sort collection by custom mapping function
• _convert_to_dict: Serialize collection to dictionary

Alternatives Considered

1. Extending CollectionBase:

   • Rejected because CollectionBase is based on BasedBase (older architecture)
   • Would not provide the clean MutableSequence interface

2. Composition over Inheritance:

   • Could wrap a list instead of inheriting from MutableSequence
   • Rejected because it would require manually implementing all list methods

3. Using Python's Built-in list:

   • Simplest approach but loses all EasyScience features
   • No interface propagation or graph tracking

4. Generic Container Class:

   • Could create a more generic container for any object type
   • Rejected to maintain type safety and EasyScience feature integration

References

• Pull Request New collection base #180
• Related Issue: New collection base functionality
• Python MutableSequence: https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableSequence
• Type Hints PEP 484: https://www.python.org/dev/peps/pep-0484/

Notes

This ADR documents the design decisions made in PR #180 (New collection base). The implementation is currently in draft/review stage. Once merged, this ADR should be updated to "Accepted" status.

Future Considerations

1. Consider adding more specialized collection types (e.g., ordered, sorted)
2. Evaluate performance with large collections (1000+ items)
3. Consider adding collection-level operations (filter, map, reduce)
4. Potential for async/lazy loading of items for very large datasets
5. Integration with numpy arrays or pandas DataFrames for numerical operations

[damskii9992]

Can you please explain to me WHY exactly we need a ModelCollection class?
Why not just have a list as an attribute to a ModelBase class like what @henrikjacobsenfys did in EasyDynamics for ComponentCollection?
I'm open for the idea but we need a valid reason to implement this (not just that it existed before).

[henrikjacobsenfys]

It would save me some code if I didn't have to write the add_component and similar methods. I'm also unsure if my code interacts correctly with the global object, but I don't know enough about it to be sure.
Finally, there's the indexing issue - it's very neat to be able to index by name as well as int as mentioned today and yesterday, but I'm unsure what the best solution is

[rozyczko]

Can you please explain to me WHY exactly we need a ModelCollection class?

• We avoid polluting every ModelBase subclass with list management logic, like add_sample, remove_sample etc.
• When you change the interface bound to a collection, you want ALL items in that collection to automatically receive the new interface. With a plain list attribute, you'd need to manually iterate and set the interface every time.
• Graph dependency tracking: With a plain list, every append/remove/insert requires manual graph management
• A ModelCollection is a ModelBase object itself so it can have its own parameters, can be serialized/deserialized as a unit, can be nested in other collections or models potentially.
• ModelCollection can be accessed in many wyas (as Henrik noted) - by index, unique_name and properly sliced#

[damskii9992]

• We avoid polluting every ModelBase subclass with list management logic, like add_sample, remove_sample etc.

Not all ModelBase subclasses will be list/require list logic, and many of those that do, will require unique list-logic anyways. Like checking that the added component is a valid component for the list.

• When you change the interface bound to a collection, you want ALL items in that collection to automatically receive the new interface. With a plain list attribute, you'd need to manually iterate and set the interface every time.

This is mute as we are moving to get rid of the interface attribute.

• Graph dependency tracking: With a plain list, every append/remove/insert requires manual graph management

This is also mute as we aim to get rid of the graph completely.

• A ModelCollection is a ModelBase object itself so it can have its own parameters, can be serialized/deserialized as a unit, can be nested in other collections or models potentially.

This is also true for a ModelBase with a simple list attribute, as @henrikjacobsenfys has shown in EasyDynamics.

• ModelCollection can be accessed in many wyas (as Henrik noted) - by index, unique_name and properly sliced#

This is so far the only valid reasoning I've found, having a list that can also be accessed by its components unique_names is definitely something I can see a use for. But such a class would be much simpler than what we have here. And we might have many places where we overwrite the setters/getters anyways.

It would save me some code if I didn't have to write the add_component and similar methods. I'm also unsure if my code interacts correctly with the global object, but I don't know enough about it to be sure.
Finally, there's the indexing issue - it's very neat to be able to index by name as well as int as mentioned today and yesterday, but I'm unsure what the best solution is

You'd have to overwrite the add_component anyways because you only allow ModelComponents in your list. And you might overwrite the unique_name getter anyways too, to solve one of the issues you've mentioned (you can make the getter be a regex search, so that a user writing "Lorentzian" would get any objects whose unique_names start with "Lorentzian", that way they don't have to know that the true unique_name is "Lorentzian_0").

The way I see it, it might make sense to have a class, lets call it EasyCollection, which only allows members inheriting from NewBase and which is a list with the additional unique_name getter and inspection. The might part here depends on how much we would have to overwrite these methods in classes inheriting from it.

[seventil]

The issue I found trying implementing a colleciton like that that was derived from built-ins is that we can't use list and dict as parents at the same time, at least half the functionality still has to be explicitely implemented. Then, if we want to use narrower typeguards, than just NewBase typevar, all the list/dict methods have to be implemented anyway with the only difference being either calling super().method() or self._items.method(). There's almost no code repetition avoided if we inherit from a built-in.

I think ModelCollection makes sense to change it to a more general collection class without interface/graph maintenance, but with a potential regex name lookup. Or maybe regex name lookup should be part of a NewBase?

[rozyczko]

regex name lookup for querying with the unique_name? As in

  model_backgrounds = model_collection['background_*']

Yes, this could potentially be useful.