Retrying an ILM action that is an AsyncActionStep does not work due to execution model changes

[dakrone]

When an index is stuck in the error state of a step which is an AsyncActionStep, like so:

PUT _ilm/policy/bad
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "5s",
        "actions": {
          "shrink": {
            "number_of_shards": 13
          }
        }
      }
    }
  }
}

PUT /foo
{
  "settings": {
    "index.number_of_shards": 2,
    "index.lifecycle.name": "bad"
  }
}

With error:

{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date" : "2018-11-08T22:47:45.865Z",
      "lifecycle_date_millis" : 1541717265865,
      "phase" : "warm",
      "phase_time" : "2018-11-08T22:47:52.601Z",
      "phase_time_millis" : 1541717272601,
      "action" : "shrink",
      "action_time" : "2018-11-08T22:47:52.601Z",
      "action_time_millis" : 1541717272601,
      "step" : "ERROR",
      "step_time" : "2018-11-08T22:47:52.688Z",
      "step_time_millis" : 1541717272688,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 5,
        "modified_date" : "2018-11-08T22:47:44.230Z",
        "modified_date_in_millis" : 1541717264230
      }
    }
  }
}

When I correct the policy and issue a POST /foo/_ilm/retry...

The now warm/shrink/shrink step never executes, because we only execute AsyncActionSteps when they are transitioned into.

The 'retry' API should perform the correct steps to retry any type of operation.

[elasticmachine]

Pinging @elastic/es-core-infra

[gwbrown]

This looks very similar to #34618, a similar solution might be applicable.