Fix: Downsample returns appropriate error when target index gets deleted unexpectedly.

[gmarouli]

Downsample transport action did not take into consideration the possibility of the target index being deleted when updating the downsample status. Although this is an unlikely event to happen without manual action, we should still return an appropriate error and not an NPE.

With this change, if someone deletes the target index after it was refreshed but before the downsample status has been updated they will receive the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Failed to update downsample status because [downsampled] does not exist"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "Failed to update downsample status because [downsampled] does not exist"
  },
  "status": 500
}

We also reduced the places where we called recordSuccessMetrics. It was called also when we encountered a failure because we assumed that downsampling had successfully finished by then. However, if the index was deleted it would still count as a failure. I think it will be more clear to count as a success only when the transport action has successfully finished all processing steps.

Fixes: #137761

[elasticsearchmachine]

Hi @gmarouli, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[gmarouli]

Thank you @kkrik-es

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 138228

[gmarouli]

💔 Some backports could not be created

Status	Branch	Result
✅	9.2
✅	9.1
❌	8.19	An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 138228

Questions ?

Please refer to the Backport tool documentation

[gmarouli]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation