Plugin System for DMS Data Store

[stephenfuqua]

Recent versions of the .NET have supported a strong capability for developing plugin-aware applications. Shall we use this approach for configuring the Data Management Service's (DMS) backend data storage?

Using a plugin architecture would facilitate community usage of alternate backend functionality. Without needing to recompile the main application: simply load the plugin into the deployed directory structure, configure via appSettings, and start the DMS application.

In Project Meadowlark, we developed a system that supported runtime configuration of which backend datastore package to activate for a given task. In this way, we had separate MongoDB, PostgreSQL, OpenSearch, and Elasticsearch backends. The MongoDB and PostgreSQL packages supported create / update / delete operations, as well as get by id. The OpenSearch and Elasticsearch backends supported get all and get by query functionality. This approach supported good encapsulation and easy extensibility.

Applied to Project Tanager, we might have a flow like the following, which envisions three plugin interfaces:

• DataAccess-R for reading a single item
• DataAccess-RA for reading all (or querying)
• DataAccess-C-U-D for modification operations.

flowchart LR

    classDef plugin fill:#0f0

    subgraph Data Management Server
        Controller
        DataAccess-C-U-D:::plugin
        DataAccess-R:::plugin
        DataAccess-RA:::plugin

        GetA{GET /}
        GetB{GET /?a=b}
        GetC{GET /id}
    end

    Client -->|"VERB /ed-fi/students"| Controller

    Controller --> GetA
    GetA -->|yes| DataAccess-RA
    GetA -->|no| GetB

    GetB -->|yes| DataAccess-RA
    GetB -->|no| GetC

    GetC -->|no| DataAccess-C-U-D
    GetC -->|yes| DataAccess-R    

Loading

Read (id) and Read (all/query) are split into separate interfaces to allow for experimentation. If a search-oriented database such as Elasticsearch is used for read all and read queries, is there still some advantage to use the main data store for reading a single object? For example, would there be a cost advantage?

The Ed-Fi development team is committed to producing full support for MSSQL and PostgreSQL. The initial backend data stores would support all three interfaces. However, anyone could then introduce an Elasticsearch backend plugin that supports the DataAccess-RA interface, as a higher-performing alternative. Or, someone might decide to create a plugin implementing the DataAccess-C-U-D interface using a non-relational database, such as MongoDB or Cassandra.

[bradbanister]

Unsurprisingly, I like this idea. I played with separating the PostgreSQL code as a backend project like Meadowlark in the RND-698 spike but abandoned it because without real .NET plugin support it would have required circular project references. Meadowlark's concept of backends as plugins worked really well, not just for the actual pluggability but also for code organization, as it enforced discipline in keeping Meadowlark core from bleeding into backend code.

[bradbanister]

I'm curious too how plugin-specific configuration works in this world. That was where I was going in the spike with the .env file, seeing if that would allow configuration for a backend that the frontend/core didn't need to know about.