Support rolling upgrade for Alerting plugin

[skkosuri-amzn]

Is your feature request related to a problem? Please describe.
Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.2 cluster

Describe the solution you'd like
Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.2 cluster
Refer: opensearch-project/opensearch-plugins#39

[skkosuri-amzn]

Started with ODFE 1.13.2 and OS rc1.

Found these two issues:

1. https://github.com/opensearch-project/OpenSearch/issues/856
2. https://github.com/opensearch-project/OpenSearch/issues/857

1 is blocker for this issue.

[skkosuri-amzn]

Steps:

1. Setup 3 node ODFE cluster using tar.
   If can run all three instances on the same node, make sure to change http.port and transport.port . I used 9200. 9201, 9202 for http and 9300, 9301, 9302 for transport.
2. Check health and make sure its green,

curl https://localhost:9200/_cluster/health?pretty -u admin:admin --insecure
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

3. Create an index and Ingest data

curl -XPUT -H 'Content-Type:application/json' https://localhost:9200/alerting-test?pretty  -u admin:admin --insecure -d'
{
  "settings": {
    "index": {
      "number_of_shards": 3,  
      "number_of_replicas": 1 
    }
  }
}
'

4. Add custom cluster settings.
   curl https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure

curl -X PUT  -H 'Content-Type:application/json' https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure -d'
{
  "persistent" : {
    "indices.recovery.max_bytes_per_sec" : "50mb"
  }
}
'

curl -X PUT  -H 'Content-Type:application/json' https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure -d'
{
  "transient" : {
    "indices.recovery.max_bytes_per_sec" : "10mb"
  }
}
'

5. Create test monitor and make sure it creates alerts.

curl -XPOST 'https://localhost:9200/_opendistro/_alerting/monitors' -H 'Content-Type:application/json' -u admin:admin --insecure -d@test-detector.json

6. Disable shard allocation of replicas

curl -X PUT  -H 'Content-Type:application/json' https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

7. Stop one ODFE node, create OpenSearch tar folder. Copy port, cluster name to OpenSearch.yml. Copy data from ODFE node to OpenSearch.
8. Start OpenSearch node and make sure it joins the cluster and cluster is green.
9. Now the cluster is in mixed mode, Add more monitors and make sure those are run successfully and they generate alerts.
10. Upgrade second node and repeat 7,8,9.
11. Upgrade third and last node and repeat 7,8,9.

[skkosuri-amzn]

Tested with OpenSearch rc1 and Security Plugin fix: opensearch-project/security#1259