Cannot retrieve mapping of index with icu_collation_keyword field

[cguedel]

Elastic.Clients.Elasticsearch version: 8.13.15

Elasticsearch version: 8.6.2

.NET runtime version: 8

Operating system version: Windows 11

Description of the problem including expected versus actual behavior:
Retrieving the mapping of an icu_collation_keyword field and recreating it on another index results in it being created as object

Steps to reproduce:

1. Create index using following mapping through kibana:

PUT /test-index
{
  "settings": {},
  "mappings": {
    "properties": {
      "icu_field": {
        "language": "de",
        "numeric": true,
        "strength": "primary",
        "type": "icu_collation_keyword",
        "variant": "@collation=standard"
      },
      "multi_field": {
        "fields": {
          "sort": {
            "numeric": true,
            "strength": "primary",
            "type": "icu_collation_keyword",
            "variant": "@collation=standard"
          },
          "text": {
            "type": "text"
          }
        },
        "type": "keyword"
      }
    }
  }
}

2. Run repro:

using Elastic.Clients.Elasticsearch;
using Elastic.Clients.Elasticsearch.IndexManagement;

var index = "test-index";
var settings = new ElasticsearchClientSettings(new Uri("http://localhost:9200/"));
settings.DisableDirectStreaming();

var client = new ElasticsearchClient(settings);

var response = await client.Indices.GetAsync(index);
var sourceIndex = response.Indices.Single().Value;

var targetIndexSettings = new IndexSettings
{
    NumberOfShards = 1,
    NumberOfReplicas = 0
};

targetIndexSettings.Index ??= new IndexSettings();
targetIndexSettings.Index.Analysis = sourceIndex.Settings?.Index?.Analysis;
targetIndexSettings.Index.MaxNgramDiff = sourceIndex.Settings?.Index?.MaxNgramDiff;

var createIndexRequest = new CreateIndexRequest(index + "-cloned")
{
    Mappings = sourceIndex.Mappings,
    Settings = targetIndexSettings
};

var createResponse = await client.Indices.CreateAsync(createIndexRequest);
if (!createResponse.Acknowledged)
{
    // throws
    throw new Exception(createResponse.DebugInformation);
}

Expected behavior
Expected cloned index to be the same as source index

Provide DebugInformation (if relevant):

Request failed to execute. Call: Status code 400 from: PUT /test-index-cloned. ServerError: Type: mapper_parsing_exception Reason: "Failed to parse mapping: Type [object] cannot be used in multi field" CausedBy: "Type: mapper_parsing_exception Reason: "Type [object] cannot be used in multi field""

# Audit trail of this API call:
 - [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.0281875
# OriginalException: Elastic.Transport.TransportException: Request failed to execute. Call: Status code 400 from: PUT /test-index-cloned. ServerError: Type: mapper_parsing_exception Reason: "Failed to parse mapping: Type [object] cannot be used in multi field" CausedBy: "Type: mapper_parsing_exception Reason: "Type [object] cannot be used in multi field""
# Request:
{"mappings":{"properties":{"icu_field":{"type":"object"},"multi_field":{"fields":{"sort":{"type":"object"},"text":{"type":"text"}},"type":"keyword"}}},"settings":{"index":{},"number_of_replicas":0,"number_of_shards":1}}
# Response:
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Type [object] cannot be used in multi field"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping: Type [object] cannot be used in multi field","caused_by":{"type":"mapper_parsing_exception","reason":"Type [object] cannot be used in multi field"}},"status":400}

[cguedel]

From what I can see PropertyInterfaceConverter doesn't provide any extension points to add a custom property (or one that's supported by a plugin)

[flobernd]

Hi @cguedel, you are correct. The specification does neither define a "CustomProperty" nor does it define the IcuField specifically. The current "default/fallback" property type is "object" and this is what prevents you from round-tripping. I'll check internally how we want to handle that case.

[cguedel]

@flobernd may I ask if you have a timeline for a potential fix here already? I'd appreciate it to be able to plan our own work on our side :)

[flobernd]

Hi @cguedel, we are currently in the process of releasing 8.14 (server) and preparing the related client versions. I expect a fix end of this week or early next week.

[flobernd]

Hi @cguedel, this is fixed in 8.14.0

[cguedel]

@flobernd Great, thank you very much!