Bug: JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'.

[JulianRooze]

Calling SearchAsync in a concurrent manner while specifying a source filter on the search descriptor causes a JSON serialization exception on the Nest.Field type. This only happens when running concurrent requests, and only when doing >= 8 concurrent tasks in my .NET Core project, or when doing >= 2 concurrent tasks in my LINQPad reproduction script (so .NET Framework). It also only happens when using a source filter.

The message:

JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'.

However, sometimes the exception type is UnexpectedElasticsearchClientException instead of JsonSerializationException.

The stacktrace:

at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.FieldsJsonConverter.WriteJson(JsonWriter writer, Object value, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.UnionJsonConverter`2.<>c__DisplayClass0_0.b__1(TSecond second)   at Nest.UnionJsonConverter.WriteJson(JsonWriter writer, Object v, JsonSerializer serializer)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeConvertable(JsonWriter writer, JsonConverter converter, Object value, JsonContract contract, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)   at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)   at Elastic.Internal.JsonNet.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)   at Nest.InternalSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting)
--

A complete LINQPad reproduction script:

async Task Main()
{
	var c = new ElasticClient();
	
	while (true)
	{
		var tasks = new List<Task>();
	
		for (var i = 0; i < 10; i++)
		{
			tasks.Add(Task.Run(async () =>
			{
				var response = await c.SearchAsync<MyObject>(s =>
				{
					s
					.Index("my_index")
					.Source(src => src.Includes(f => f.Field("any_field")));
	
					Console.WriteLine(Serialize(s, c));
	
					return s;
				});
			}));
		}
	
		await Task.WhenAll(tasks);
	}
}

public class MyObject
{
	
}

public static string Serialize(ISearchRequest request, IElasticClient client)
{
	string serialized;
	using (var stream = new MemoryStream())
	using (var streamReader = new StreamReader(stream))
	{
		client.RequestResponseSerializer.Serialize(request, stream);
		stream.Position = 0;
		serialized = streamReader.ReadToEnd();
		return serialized;
	}
}

If this somehow doesn't fail, try increasing the concurrency from 10 to something higher. You only need NEST 6.5.1 and a working Elasticsearch on the default port with a my_index index.

This is using the NEST 6.5.1 NuGet package (also reproduces for 6.5.0).

This behavior is a real head scratcher to me. I'm correct when saying that ElasticClient is meant to be thread-safe, right? That's how we've always used it.

Possibly a regression of #706?

[JulianRooze]

And to clarify: the serialization of the request like in the above example is not necessary for reproduction. If I remove the Console.WriteLine and Serialize method, it still reproduces.

So the minimal code to reproduce it is this:

async Task Main()
{
	var c = new ElasticClient();
	
	while (true)
	{
		var tasks = new List<Task>();
	
		for (var i = 0; i < 32; i++)
		{
			tasks.Add(Task.Run(async () =>
			{
				await c.SearchAsync<string>(s => s.Index("my_index")
					.Source(src => src.Includes(f => f.Field("any_field"))));
			}));
		}
	
		await Task.WhenAll(tasks);
	}
}

[russcam]

This looks suspiciously like

elasticsearch-net/src/Nest/CommonAbstractions/Infer/Fields/FieldsJsonConverter.cs

Lines 21 to 24 in c6652ac

	// overridden Equals() method means a Fields with only one Field
	// results in Equality, which triggers Json.NET's Reference loop detection
	var referenceLoopHandling = serializer.ReferenceLoopHandling;
	serializer.ReferenceLoopHandling = ReferenceLoopHandling.Serialize;

which is triggered from Fields, when Fields contains only a single instance of Field. That the ReferenceLoopHandling is set on serializer (which I suspect is shared), does not look correct.

Thanks for reporting @JulianRooze, and for the repro 👍 Will need to investigate further

[JulianRooze]

@russcam Ah that makes sense, I hadn't considered checking if it still reproduces when specifying more than one field, but I just tried it and it does not reproduce, confirming your finding.

So I guess the workaround until a fix is published is to always request two fields.

Thanks for looking into it!

[russcam]

I've opened #3618 to address.

We're looking to put out 6.6.0 very shortly, which will contain this fix

[russcam]

Closing this as it's now in NEST 6.6.0

[himadrinath]

@russcam still having this problem in NEST 6.8.0
Elastic.Internal.JsonNet.JsonSerializationException at Elastic.Internal.JsonNet.Serialization.JsonSerializerInternalWriter.CheckForCircularReference

[russcam]

@himadrinath Please could you

1. Open a new issue, and link to this one with

   Relates: #3617
   

   at the top of the description

2. Provide a small but succinct example the reproduces what you're seeing, in the new issue.

Thanks!