Changed: Enhance DataLayer Plugin Registration System for Improved Third-Party Integration

[MichaelTaylor3D]

Purpose:

The current implementation of the DataLayer plugin system relies on direct modifications to the config.yaml file for plugin registration. This approach presents challenges for third-party applications developed atop DataLayer, particularly in terms of creating installers that need to register plugins. Currently, installers either alter config.yaml directly during installation or require users to manually register plugins. The former approach risks compromising the stability of Chia due to potential installer bugs, while the latter is notably user-unfriendly and sometimes unfeasible.

To address these issues, this Pull Request introduces a new mechanism for plugin management within the DataLayer framework. I propose the creation of dedicated plugin directories within the CHIA_ROOT/data_layer path. This structure enables third-party applications to install or uninstall DataLayer plugin files smoothly, without needing to interact directly with the config.yaml file.

Key Changes:

Plugin Directories: Establish designated directories within CHIA_ROOT/data_layer for DataLayer plugins, streamlining the installation and uninstallation process.
Reduced Direct Configuration Manipulation: Minimize the need for third-party installers to modify config.yaml directly, enhancing system stability and reliability.
Improved User Experience: Simplify the plugin installation process for end-users, removing the need for manual plugin registration.
This enhancement fosters a more robust and user-friendly ecosystem for DataLayer and its third-party integrations, ensuring a smoother experience for both developers and users.

Current Behavior:

Datalayer activates plugins only from config.yaml

New Behavior:

Datalayer will still activate plugins from config.yaml, but it will also look in the CHIA_ROOT/data_layer/plugins folder and activate plugins from .conf files that are located in that directory.

Testing Notes:

PR includes coverage for loading the conf files from the plugins folder

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[emlowe]

Michael, you have an extraneous update to the mozilla-ca sub repo you should undo here

[MichaelTaylor3D]

Michael, you have an extraneous update to the mozilla-ca sub repo you should undo here

@emlowe
Not sure how that happened but im struggling to revert it since its a submodule update and not a regular versioned file. Any pointers you can throw my way?

[altendky]

Shouldn't the plugin files allow for the full PluginRemote schema (albeit two items) so that these new plugins can provide custom headers such as for authentication?

[MichaelTaylor3D]

Shouldn't the plugin files allow for the full PluginRemote schema (albeit two items) so that these new plugins can provide custom headers such as for authentication?

Added support for PluginRemote schema

[emlowe]

I think the only real question I have is using JSON for this additional plugin config, while the rest of the chia config is yaml. This was discussed earlier, I'm not sure about it tbh

[emlowe]

I've decided it's ok to allow JSON here for these files plugin files
@MichaelTaylor3D I think you might owe a comment or two for Kyle
@altendky I think you might be able to resolve some comments

[MichaelTaylor3D]

@emlowe went on vacation and then got busy when came back, revisiting this now

[emlowe]

Kyle reminds me that all json files are also yaml files - so if you parse this file with the yaml parser instead of the json parser, you can use either style for the plugin and it shouldn't involve any other code changes.

[MichaelTaylor3D]

Kyle reminds me that all json files are also yaml files - so if you parse this file with the yaml parser instead of the json parser, you can use either style for the plugin and it shouldn't involve any other code changes.

I swapped it over to using yaml loader instead of json loader

[emlowe]

close and reopen for CI