[DPE-4251][DPE-4196][DPE-3603] Plugin Management Re-work (1/3): Plugin Manager handles the config changed before started

[phvalguima]

Introduction

This is part of a bigger set of changes happening across the plugin management modules.

The main goal is to: (1) curb the requirements for restart from any configuration changes; and (2) get a faster response time using cached entries wherever possible.

We ensure that any configuration changes that come from plugin management are applied via API before being persisted on config files. If the API responds with a 200 status, then we should only write the new value to the configuration and finish without a need for restart.

In case the service is down and API is not available, we can assume we will eventually start the service back up. In this case, it suffices to write the config entries to the files and leave to the next start to pick them up.

This task is going to be divided into 3x parts:

1. Addresses low ranging fruits where we reduce the number of restarts and add caching support
2. Merge {add,delete}_plugin together and its equivalents in OpenSearchPluginConfig class: from now on, we receive one big dictionary where a key: None means we want to simply delete that entry
3. Returns unit tests: this is going to be commented out whilst Parts 1 and 2 happen, given this part of the code was covered with extensive testing

Part 1/3

The current implementation of plugin_manager.run waits for the cluster to be started before processing its config changed. We relax this demand and open to the option where the cluster is not yet ready, so we can modify the configuration without issuing a restart request.

#252 is closed with OpenSearchPluginRelationsHandler interface. It allows plugins to define how they will handle its relation(s). opensearch_backup module extends this interface and defines a checker to process either small or large deployments details.

Other relevant changes:

1. Renaming method check_plugin_manager_ready to check_plugin_manager_ready_for_api
2. Any plugin that needs to manage things via API call should check the health of the cluster using check_plugin_manager_ready_for_api
3. Moving opensearch_distro.version to load the workload_version file we have present instead of an API call: this is two fold, 1. removes the dependency to a cluster to be ready and 2. makes this method in-sync with recent changes for upgrades logic
4. Waive the need of loading the default settings if this particular unit is powered down: which makes sense, in this moment we can do any config changes as we will eventually powered it back up later
5. If /_cluster/settings is available: apply the configs via API and do not add a restart request
6. On config-changed handler, the upgrade_in_progress check gets precedence and will continuously defer config-changed event until upgrade is finished before calling the plugin manager
7. Create a OpenSearchKeyStoreNotReadyYetError: responsible to identify the keystore has not been initialized yet across the cluster and hence, we cannot manage any plugins that use it; however, we always apply the config changes from that plugin.

That still frees the config_changed to just call plugin_manager.run() before everything is set, as the run() method changes hard configuration only.

Closes #252, #280

[phvalguima]

@carlcsaposs-canonical I have reshuffled the self.upgrade_in_progress here. Now, we check and defer before even doing the plugin_manager.run() (hence, no options will change until the upgrade is finished). Is this the right way? In another sense, it holds the user from doing any changes while the upgrade is in progress.

[carlcsaposs-canonical]

@phvalguima from the current diff, it seems like if there's an IP change and no config change, a warning message will be logged

ip change should be supported during in-progress upgrade (so if only ip change, there shouldn't be a warning)

[phvalguima]

Temp.: Will revert it once the main integration tests pass and we are sure of the changes below (there are a lot of unit tests that will need to be updated here)