Add design documentation for extension mechanism

DESIGN.md

@@ -0,0 +1,303 @@
+## Introduction
+
+The OpenRemote platform is designed for versatility, supporting diverse domains such as energy management, smart cities, and fleet management.
+As the platform scales, it is no longer practical or efficient for the Core repository ([openremote/openremote](https://github.com/openremote/openremote/)) to natively support every specific protocol, UI widget or business logic requirement.
+
+## Purpose
+
+The OpenRemote extension mechanism decouples domain-specific functionality from the platform Core.
+This mechanism allows developers to package resources such as Protocol Agents, Asset Types, UI Components, and Logic (Container Services, Rules) into a single, deployable unit called an extension.
+
+By standardizing how these extensions are built and integrated, we aim to achieve the following:
+
+* **Enhanced Maintainability**: Create a leaner Core codebase while supporting all existing and future domains.
+* **Simplified User Experience**: Making functionality optional ensures users are not overwhelmed by Agents, Asset Types, or menu items that are irrelevant to their specific project.
+* **System Resource Efficiency**: A monolithic architecture forces the system to load all libraries and background processes regardless of use. Modularization allows for a smaller runtime which reduces resource consumption (RAM, CPU) on edge gateways and industrial hardware.
+* **Architectural Rigor & API Maturity**: Formalizing the boundary between the Core and Extensions forces the development of stable, well-documented APIs. It encourages developers to think in terms of reusability and clean contracts rather than making ad-hoc changes to the Core.
+
+## Conceptual Definition
+
+To establish a clear framework, we must define what constitutes an Extension within OpenRemote.
+An extension is not merely a piece of code; it is a self-contained package that introduces new capabilities or modifies existing behavior without requiring a rebuild of the OpenRemote Core.
+
+### What is an Extension?
+
+Technically, an extension is a versioned artifact (such as a JAR file) that the OpenRemote Manager can discover, load, and initialize at runtime.
+The Extension communicates with the Core through predefined extension points.
+These extension points are stable APIs that allow the Extension to register its own Container Services, UI elements or Asset Types into the Manager.
+
+### Extension Types
+
+In this section, Extensions are categorized by the nature of the resources they provide to the platform.
+There is currently no immediate need for each of these types; however, having a comprehensive list of possible future types helps with thinking about implementation requirements for future extensions.
+
+#### Functional Extensions
+
+These add backend logic, connectivity, or processing power to the system.
+They typically run as background processes or service providers.
+
+* **Connectivity**: Protocol agents such as Artnet, ChirpStack, KNX, and ZWave.
+* **Logic & Rules**: Support for Groovy/Flow rules, Container Services (Forecasting, Simulators), Rule languages (similar to JavaScript and JSON).
+* **Identity**: Pluggable Identity providers (similar to the Keycloak and Basic IdPs).
+* **Storage**: Infrastructure drivers for Operational data (JDBC) or Historical data (TimescaleDB).
+
+#### Data Extensions
+
+These define how data is structured and transformed within the system.
+
+* **Model Definitions**: Domain-specific Asset Types (Energy, HVAC, Smart City) and Setup Types (Demo, Load testing).
+* **Data Integrity**:
+  * **Value Types**: Defining custom data structures or units of measurement.
+  * **Value Validation**: Logic to ensure incoming data meets specific criteria (range checks, schema validation, or mandatory fields).
+* **Processing & Presentation**:
+  * **Value Filters**: Data transformation logic (JsonPath, RegEx).
+  * **Value Formatting**: Rules for how data is rendered in the UI or exported (unit conversion, decimal rounding, or localized date strings).
+
+#### Visual Extensions
+
+These enhance the Manager UI or provide end-user applications.
+They are primarily client-side assets that extend the frontend.
+
+* **Widgets**: Custom Insights widgets for dashboards and data visualization (e.g. a specialized gauge for energy consumption).
+* **Custom Applications**: Project-specific UIs developed to meet unique domain requirements (e.g. an Alarms App or a Fleet App).
+* **System & Utility UIs**: Integration of technical or third-party interfaces directly into the platform, such as Swagger UI for API exploration.
+
+#### Composite Extensions
+
+A collection of Functional, Visual and Data extension used to implement functionality for a specific domain or end-user group.
+For example an Energy Extension could include the Energy Asset types (Data), a Modbus Agent for meters (Functional), and Energy Dashboard widgets (Visual).
+
+### Evolution & Prioritization
+
+The transition to a more modular platform will be done iteratively.
+We will focus on introducing extensions into the backend and the structural relationship between extensions before introducing the complexity of frontend extension points or new data processing APIs.
+
+#### Phase 1: Building the Infrastructure
+
+Existing OpenRemote functionality is first migrated into a modular structure:
+
+* **Connectivity**: Decoupling Protocol Agents (KNX, ZWave, Artnet etc.) to allow for project-specific Agents and reduce the Core codebase complexity.
+* **Logic & Rules**: Reusable Groovy/Flow rules, Container Services (Forecasting, Simulators).
+* **Model Definitions**: Supporting optionally installing Asset Types (Energy, Smart City) and Setup Types (Demo, Load tests).
+* **Composition**: Implementing the logic for extension dependencies and composite extensions. This allows extensions to build upon one another and be composed into domain-specific solutions.
+
+#### Phase 2: User Interfaces
+
+Once the initial infrastructure has been built, the scope will expand to include standalone UIs.
+
+* **Custom Applications**: Enabling project or domain specific UIs to be deployed as extensions (Alarms or Fleet App).
+* **System & Utility UIs**: Integrating technical tools directly into the platform, such as Swagger UI.
+
+#### Future
+
+These represent long-term goals that require the development of new internal APIs within the OpenRemote Core before they can be fully supported:
+
+* **Widgets**: Custom Insights widgets for dashboards and data visualization (specialized energy gauges).
+* **Data Integrity & Processing**: Advanced value types, validation, formatting, and specialized filters.
+* **Rules**: Integration of entirely new Rule Engine languages (JavaScript, JSON).
+* **Storage**: Pluggable infrastructure drivers for database management.
+
+## Technical Specification
+
+In the initial implementation, the extension mechanism leverages the existing Gradle build system and OpenRemote's established Service Provider Interfaces (SPI).
+This ensures that extensions are discovered and initialized using the same robust patterns currently used by the Core and existing custom projects.
+
+### Namespace Convention
+
+To ensure architectural clarity and prevent classpath collisions, all extensions must adopt the following base package:
+`org.openremote.extension.{extension-name}`
+
+This convention applies to both the Java package structure and the internal resource directory layout.
+
+### Dependency Management (Build-time)
+
+Extensions are developed as independent Gradle projects.
+To include an extension in an OpenRemote deployment, it is added as a standard dependency in the project's `build.gradle`.
+
+* **Transitive Dependencies**: Gradle handles the resolution of shared libraries between the Core and multiple extensions, ensuring a consistent and conflict-free classpath.
+* **Composition**: Composite Extensions define a collection of dependencies in their `build.gradle`, allowing a developer to include a single "Composite" artifact to pull in a complete suite of functional and data modules.
+
+### Discovery and Registration (SPI)
+
+Extensions register their components and initialization logic by implementing standard OpenRemote SPIs.
+The Manager scans the classpath at startup to find and execute these implementations via the Java `ServiceLoader`.
+
+* **Asset Types**: Extensions must implement the `org.openremote.model.AssetModelProvider` SPI. This allows the extension to register its domain-specific Asset Types (Energy, HVAC) in a Manager instance.
+* **Container Services**: Background logic (such as forecasting or simulators) is registered via the `org.openremote.model.ContainerService` SPI. These services are managed by the platform container, allowing them to be started and stopped in coordination with the Manager's lifecycle.
+* **Setup Tasks**: Extensions must implement the `org.openremote.model.setup.SetupTasks` SPI. This is the primary mechanism for initializing the extension within a project. Setup tasks are responsible for:
+   * Creating default assets and system configurations.
+   * **Rule Creation**: Programmatically creating and persisting Groovy and Flow rules into the platform.
+
+### Database Migrations (Flyway)
+
+Extensions requiring database migrations must use Flyway.
+
+* **Path**: `src/main/resources/org/openremote/extension/{name}/setup/database/`
+* **Execution**: Migrations are triggered during the extension activation sequence, before `SetupTasks` or `ContainerServices` are initialized.
+
+### Testing Strategy (Spock & Groovy)
+
+To ensure reliability, extensions should include a comprehensive test suite.
+Following the OpenRemote Core standard, we use the Spock Framework with Groovy for integration and unit testing.
+
+Test-specific setups (like Keycloak or Manager configurations) can be isolated within the `src/test` directory, ensuring that the production artifact remains lean while the development cycle remains robust.
+
+### Artifact Structure
+
+The following directory layout represents an example Energy extension.
+Note the separation of concerns between production code (`src/main`) and testing logic (`src/test`).
+
+```text
+energy/
+├── src/main/java/org/openremote/extension/energy/
+│   ├── model/                         # Java Asset implementations
+│   │   ├── ElectricityAsset.java
+│   │   └── EnergyModelProvider.java   # Implements AssetModelProvider SPI
+│   ├── manager/                       # Core logic & Container services
+│   │   └── EnergyOptimisationService.java # Implements ContainerService SPI
+├── src/main/resources/
+│   ├── org/openremote/extension/energy/setup/database/ # Flyway scripts
+│   │   └── V20260131_01__RenameEnumValues.sql
+│   └── META-INF/services/             # SPI Registration
+│       ├── org.openremote.model.AssetModelProvider
+│       ├── org.openremote.model.ContainerService
+│       ├── org.openremote.model.ExtensionMetadata
+│       └── org.openremote.model.setup.SetupTasks
+├── src/test/groovy/org/openremote/extension/energy/
+│   ├── EnergyOptimisationTest.groovy  # Spock Integration Tests
+│   ├── ForecastSolarServiceTest.groovy
+│   └── ManagerTestSetup.groovy        # Test-specific environment logic
+└── src/test/resources/
+    └── META-INF/services/
+        └── org.openremote.model.setup.SetupTasks # Test-only setup tasks
+```
+
+## Deployment & Lifecycle Management
+
+This section defines how extensions are packaged, discovered, and activated using a programmatic metadata provider.
+
+### Deployment & Packaging
+
+In the initial implementation, extensions are treated as static modules bundled into the OpenRemote Manager's classpath.
+
+* **Build-time Integration**: Extensions are added as `implementation` dependencies in the deployment project's Gradle configuration.
+* **The Artifact**: Every extension must include an implementation of the `ExtensionMetadata` SPI.
+* **Registration**: The implementation is registered in `src/main/resources/META-INF/services/org.openremote.model.ExtensionMetadata`.
+* **Artifact Inclusion**: During the Docker build, JARs are placed in the Manager's library directory for automatic JVM loading.
+
+### The Metadata SPI (`ExtensionMetadata`)
+
+To manage dependencies and activation, the Manager uses a dedicated SPI.
+This interface acts as the "identity card" for the extension, providing the Manager with the logical information needed to build the dependency graph.
+
+**Example Implementation:**
+
+```java
+package org.openremote.extension.energy;
+
+import org.openremote.model.ExtensionMetadata;
+import java.util.Set;
+
+public class EnergyExtensionMetadata implements ExtensionMetadata {
+    @Override
+    public String getId() { return "org.openremote.extension.energy"; }
+
+    @Override
+    public String getVersion() { return "1.0.0"; }
+
+    @Override
+    public String getDescription() { 
+        return "A composite suite providing Energy Asset types, optimization logic, and Modbus connectivity."; 
+    }
+
+    @Override
+    public ExtensionType getType() { return ExtensionType.COMPOSITE; }
+
+    @Override
+    public Set<String> getDependencies() {
+        return Set.of("org.openremote.extension.ems", "org.openremote.extension.modbus");
+    }
+}
+```
+
+### Composite Extension & Dependency Logic
+
+The use of a Metadata SPI allows the Manager to handle Composite Extensions as logical bundles.
+
+* **Dependency Resolution**: On startup, the Manager uses `ServiceLoader` to load all `ExtensionMetadata` instances. It builds a directed graph of the extensions on the classpath.
+* **Circular Dependency Protection**: The Manager's dependency resolver checks for circular references (A → B → A). If a cycle is detected, the Manager aborts the boot sequence to prevent recursive initialization loops.
+* **Activation Safety**: The Manager will refuse to initialize an extension if its declared dependencies are missing from the classpath or are explicitly disabled.
+
+### The Activation Roadmap
+
+#### Stage 1: Implicit Activation (Current)
+
+The Manager loads all extensions found via the `ExtensionMetadata` SPI and initializes their associated `AssetModelProvider`, `SetupTasks`, and `ContainerService` implementations automatically.
+
+#### Stage 2: Environment-Driven Whitelisting
+
+The `OR_ENABLED_EXTENSIONS` environment variable is introduced.
+
+* **Behavior**: The Manager filters the discovered `ExtensionMetadata` list. The Manager automatically activates all transitive dependencies declared in the metadata.
+* **Persistence**: The list of active extension IDs is persisted in `manager_config.json`.
+
+#### Stage 3: UI-Driven Configuration
+
+An Extensions Page allows users to toggle extensions.
+The UI uses the extension metadata to show descriptions, versions, and warning prompts if a user tries to disable a dependency required by another active extension.
+
+* **Lifecycle Requirement (Restart)**: Because the extension mechanism relies on the Java Classpath and the `ServiceLoader` discovery process, any change to the activation state (enabling or disabling an extension) requires a restart of the Manager container.
+* **User Feedback**: The UI must explicitly notify the user that changes are "Pending" and provide a "Restart Manager" trigger or a clear instruction to restart the stack to apply the new configuration.
+* **Safety Check**: The Manager will validate the pending configuration during the next boot sequence.
+
+### Resource Isolation & Conflict Management
+
+* **Version Pinning**: Gradle remains the arbiter for build-time version conflicts.
+* **Namespace Hygiene**: All resources must stay within the `org.openremote.extension.{name}` path to prevent accidental overwriting of Core or peer-extension assets.
+
+## Versioning & Release Management (Monorepo)
+
+To minimize administrative overhead and simplify the development lifecycle, all official extensions are managed within a single monorepo.
+This approach ensures that extensions are built, tested, and shipped in lockstep with the OpenRemote Manager.
+
+### Repository Structure
+
+The monorepo uses Gradle subprojects to isolate each extension.
+A shared `build.gradle` in the root provides common build logic, ensuring all extensions use the same compiler settings and dependency versions.
+
+```text
+openremote-extensions/ (Monorepo Root)
+├── build.gradle        # Shared build logic
+├── settings.gradle     # Defines shared Gradle settings
+├── ems/                # Sub-project
+├── energy/             # Sub-project
+└── modbus/             # Sub-project
+```
+
+### Implicit Core Compatibility
+
+In this initial phase, explicit core compatibility functions like `getMinCoreVersion()` are omitted from the `ExtensionMetadata` SPI.
+
+* **The Docker Contract**: Compatibility is managed at the packaging level. Because the extensions and the Manager are bundled into the same Docker image during the CI/CD process, it is guaranteed that the extensions on the classpath have been built and tested against that specific Manager version.
+* **Simplified Manifest**: This reduces boilerplate and prevents version mismatch errors during startup in a controlled environment.
+
+### Release Tying
+
+Extensions in the monorepo follow a Release Train model.
+
+* When a new version of the OpenRemote stack is released, all extensions within the monorepo are typically tagged and released under the same version number as the platform (or a synchronized sub-version).
+* This ensures that any internal API changes made in the Core are immediately reflected and tested across all extensions before the Docker image is published.
+
+### Internal Dependency Management
+
+The monorepo allows extensions to share internal code without exposing it as public APIs:
+
+* **Shared Modules**: Shared utilities (e.g., energy calculation math) can still reside in a shared extension sub-project.
+* **Gradle Linking**: Extensions could include these via `implementation project(':extensions:sharedmodule')`. These can be shaded or bundled into the extension JAR during the build, keeping the artifact self-contained.
+
+### CI/CD Efficiency
+
+The monorepo structure allows for a unified pipeline:
+
+* **Atomic Commits**: A single Pull Request can update a dependency and all affected extensions simultaneously.
+* **Unified Testing**: Integration tests can easily span multiple extensions to verify that a Composite Extension (e.g. `energy`) functions correctly with its sub-dependencies (`modbus`, `ems`) before the image is finalized.