Incorrect attachment field names after indexing

[IvanSorokin]

NEST/Elasticsearch.Net version: 6.4.0

Elasticsearch version: 6.5.1

Description of the problem including expected versus actual behavior: I am reindexing some document from 5.0 ES index to 6.5.1 using NEST client and bulk update. I am migrating document like this

Func<BulkIndexDescriptor<T>, BulkIndexDescriptor<T>> bulkIndexSelector = op =>
    {
        var desc = op.Document(hit.Source);
        if (ensureSameRouting)
            desc = desc.Routing(hit.Routing);
        if (indexName != null)
            desc = desc.Index(indexName);
        return desc;
    };

while scrolling documents in the older index. T is the class

[ElasticsearchType(Name = "notifications")]
    public class AttachmentElasticDocument
    {
        [Keyword]
        public String Name { get; set; }
        
        [Keyword]
        public Guid Id { get; set; }

        public Attachment Attachment { get; set; } //Nest class
        
        public JoinField Join { get; set; }
        
        [Keyword]
        public string Data { get; set; }
    }

After this I got documents with _ in the attachment field

attachment : {
   "_content": "text",
   "_language": "ru"
}

So my queries are not working because they are expecting "content" field name without underscore.
Then I decompiled the source and found this

  [ContractJsonConverter(typeof (AttachmentConverter))]
  public class Attachment
  {
    /// <summary>The author.</summary>
    [JsonProperty("author")]
    public string Author { get; set; }

    /// <summary>The base64 encoded content. Can be explicitly set</summary>
    [JsonProperty("content")]
    public string Content { get; set; }

    ...and so on
  }

No underscores are here in attributes too. How to get rid of them in the index?
I am using NEST Serializer.

ConnectionSettings.SourceSerializerFactory sourceSerializerFactory = (builtin, setts) => new JsonNetSerializer(builtin,
                                                                                                               setts,
                                                                                                               () =>
                                                                                                                   new
                                                                                                                   JsonSerializerSettings
                                                                                                                   {
                                                                                                                       TypeNameHandling = TypeNameHandling.Auto
                                                                                                                   });
var connectionSettings =
    new ConnectionSettings(connectionPool, sourceSerializer: sourceSerializerFactory);

[russcam]

How were the contents of the Attachment property originally indexed?

[IvanSorokin]

It was indexed using the ingest attachment plugin (ES 5). So this attachment field was not produced by our code directly, we just sent data through pipeline.

elasticClient.PutPipeline(AnalysisSettings.AttachmentsPipelineName,
                                          p => p.Processors(ps => ps.Attachment<AttachmentMixedDocument>(apd => apd.Field(d => d.Data)
                                                                                                                    .IndexedCharacters(-1))
                                                                    .Remove<AttachmentMixedDocument>(rpd => rpd.Field(d => d.Data))));

AttachmentMixedDocument has all the fields AttachmentElasticDocument has plus some additional fields.

[IvanSorokin]

And there is no underscores in the original ES 5 index. ES 6 mapping is also correct (without underscores) after using automap.

[russcam]

The JsonConverter for Attachment is configured to be able to read attachments that were indexed with both ingest-attachment plugin and the now deprecated mapper-attachment plugin.

With the latter, additional metadata was provided generally with underscore prefixed fields, so the serialize method of the converter is intended to work with this plugin. Now that the mapper-attachment plugin is no more however, the converter should be updated to use the field naming convention of the ingest-attachment plugin, so that the read then reserialize scenario works as expected.

[IvanSorokin]

So how can I update the convention? I tried to use CamelCaseNamingStrategy (via the second lambda in the JsonSerializer ctor) in the connection settings but it didnt help.

[russcam]

You can't change the serialization convention for Attachment because it's defined in a custom JsonConverter:

elasticsearch-net/src/Nest/Mapping/Types/Specialized/Attachment/Attachment.cs

Lines 92 to 137 in b8cf545

	internal class AttachmentConverter : JsonConverter
	{
	public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
	{
	var attachment = (Attachment)value;
	if (!attachment.ContainsMetadata)
	writer.WriteValue(attachment.Content);
	else
	{
	writer.WriteStartObject();
	writer.WritePropertyName("_content");
	writer.WriteValue(attachment.Content);
	if (!string.IsNullOrEmpty(attachment.Name))
	{
	writer.WritePropertyName("_name");
	writer.WriteValue(attachment.Name);
	}
	if (!string.IsNullOrEmpty(attachment.ContentType))
	{
	writer.WritePropertyName("_content_type");
	writer.WriteValue(attachment.ContentType);
	}
	if (!string.IsNullOrEmpty(attachment.Language))
	{
	writer.WritePropertyName("_language");
	writer.WriteValue(attachment.Language);
	}
	if (attachment.DetectLanguage.HasValue)
	{
	writer.WritePropertyName("_detect_language");
	writer.WriteValue(attachment.DetectLanguage.Value);
	}
	if (attachment.IndexedCharacters.HasValue)
	{
	writer.WritePropertyName("_indexed_chars");
	writer.WriteValue(attachment.IndexedCharacters.Value);
	}
	writer.WriteEndObject();
	}
	}

Would be more than happy to accept a PR to address the Serialize method, and release a new version of the client. In the meantime, you could copy the Attachment type and AttachmentConverter into your code base and change them to ensure attachments are serialized with the field names that they're indexed with.

An alternative approach might be to use the Reindex API on the 6.x cluster, and perform a Reindex from remote against the 5.x cluster. This would avoid the need to deserialize/serialize roundtrip to POCOs.

[IvanSorokin]

I would be happy to use reindex api, but sadly I can't because a lot of other operations involved while reindexing. I will copy type, I guess. Thanks for your answer!

[russcam]

I've opened #3600 to discuss

[russcam]

#3600 has been merged and will be in the next release (6.6.0), so am closing this.