JsonParsingException when calling Snapshot.GetAsync if repository contains failed snapshot

[pschwarzpp]

NEST/Elasticsearch.Net version: NEST 7.4.2

Elasticsearch version: 7.4.2

Description of the problem including expected versus actual behavior:
We expect Snapshot.GetAsync to return potentially existing information on snapshots even if failed snapshots are returned by Elasticsearch.

NEST fails to deserialize SnapshotShardFailure.ShardId, Elasticsearch returns a number, but the property on SnapshotShardFailure is string.

We now work around the issue by using the lowlevel client with a custom GetSnapshotResponse where SnapshotShardFailure.ShardId is int, but a fix for 7.4.x would be appreciated.

Part of a problematic response:

{
  "snapshots": [
    {
      "snapshot": "snapshot_2020-03-31t00:02:18z",
      "uuid": "P9oZzuEfS8qbT-FVZLFSgw",
      "version_id": 7040299,
      "version": "7.4.2",
      "indices": [ "someIndices" ],
      "include_global_state": true,
      "state": "FAILED",
      "reason": "Indices don't have primary shards [someIndex]",
      "start_time": "2020-03-31T00:02:21.478Z",
      "start_time_in_millis": 1585612941478,
      "end_time": "2020-03-31T00:02:25.353Z",
      "end_time_in_millis": 1585612945353,
      "duration_in_millis": 3875,
      "failures": [
        {
          "index": "someIndex",
          "index_uuid": "someIndex",
          "shard_id": 1, // <----
          "reason": "primary shard is not allocated",
          "status": "INTERNAL_SERVER_ERROR"
        },
        {
          "index": "someIndex",
          "index_uuid": "someIndex",
          "shard_id": 0, // <----
          "reason": "primary shard is not allocated",
          "status": "INTERNAL_SERVER_ERROR"
        },
        {
          "index": "someIndex",
          "index_uuid": "someIndex",
          "shard_id": 2, // <----
          "reason": "primary shard is not allocated",
          "status": "INTERNAL_SERVER_ERROR"
        }
      ],
      "shards": {
        "total": 78,
        "failed": 3,
        "successful": 75
      }
    }
  ]
}

Steps to reproduce:

1. Have a snapshot repository with a failed Snapshot (in our cases the snapshots failed because the primary shard for one of the included indices was not allocated).
2. Call Snapshot.GetAsync(repositoryWithFailedSnapshot, "*")
3. Catch exception

Expected behavior
See above.

Provide DebugInformation (if relevant):

# FailureReason: Unrecoverable/Unexpected BadResponse while attempting GET on http://myelasticserver:9200/_snapshot/REPO/%2A
 - [1] BadResponse: Node: http://myelasticserver:9200/ Exception: JsonParsingException Took: 00:00:00.9300047
# Audit exception in step 1 BadResponse:
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:'1', at offset:19044
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength) in C:\\Users\\russc\\source\\elasticsearch-net\\src\\Elasticsearch.Net\\Utf8Json\\JsonReader.cs:line 636
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
# Inner Exception: expected:'String Begin Token', actual:'1', at offset:19044
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:'1', at offset:19044
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength) in C:\\Users\\russc\\source\\elasticsearch-net\\src\\Elasticsearch.Net\\Utf8Json\\JsonReader.cs:line 636
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
# Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: expected:'String Begin Token', actual:'1', at offset:19044 ---> Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:'1', at offset:19044
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength) in C:\\Users\\russc\\source\\elasticsearch-net\\src\\Elasticsearch.Net\\Utf8Json\\JsonReader.cs:line 636
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) in C:\\Users\\russc\\source\\elasticsearch-net\\src\\Elasticsearch.Net\\Transport\\Transport.cs:line 149
   at ElasticNEST.SnapshotStatus.GetSnapshotsCheck() in C:\\Projects\\PS.Playgrounds\\ElasticNEST\\SnapshotStatus.cs:line 86

[russcam]

Thanks for the detailed issue, @pschwarzpp. Will look to fix this for the next release