each call to ElasticClient.Index uses up a file descriptor

[mhowlett]

NEST/Elasticsearch.Net version:

5.0.0-rc3 from nuget

Elasticsearch version:

docker image: elasticsearch:5.0.1

Description of the problem including expected versus actual behavior:

Every time I call .Index, or .IndexAsync on ElasticClient it uses up a file descriptor (opens a TCP connection and leaves it open). My process soon quits.

Steps to reproduce:

i'm running .NET Core 1.1 on linux (dotnet:latest docker image).

  var settings = new ConnectionSettings(new Uri($"http://{elasticHost}:9200"))
    .DefaultIndex("myindex");
  var ec = new Nest.ElasticClient(settings);

    ...

  var r = ec.Index(elasticSummary);

Provide ConnectionSettings (if relevant):

as above

Provide DebugInformation (if relevant):

lsof -a -p [pid] 
...
dotnet  22944 root  203u  sock    0,8      0t0 738704 protocol: TCP
...
etc.

nothing relevant is logged by elastic search (using default settings in docker image).

[Mpdreamz]

The CoreFX client is based on HttpClient and we share and reuse a single instance of that per request specific settings which in practice will boil down to one per: ConnectionSettings.

ConnectionSettings should therefor be something shared over multiple requests and ideally live as a singleton in your application. Are you creating a new client on each invocation of ec.Index ? Can you share a bit more about your application specific setup?

Not saying your code is at fault but would love to rule it out before diving in deeper tomorrow morning 👍 .

[mhowlett]

Ahh, that hash lookup is fancy, definitely a good place to go looking for problems.

I'm only creating one settings file / client, but one thing worth noting which might be unusual is the client is created on a different thread from which it is used.

I've got a bit of time to do some investigating now and will post further info in a bit.

[mhowlett]

this is .DebugInformation on the IIndexResponse (of documents successfully indexed): Is this normal?

Valid NEST response built from a successful low level call on PUT: /myindex/user-summary/2145063
crawl1_1     | # Audit trail of this API call:
crawl1_1     |  - [1] HealthyResponse: Node: http://192.168.43.9:9200/ Took: 00:00:00.0098910
crawl1_1     | # Request:
crawl1_1     | <Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
crawl1_1     | # Response:
crawl1_1     | <Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

(still investigating...)

[russcam]

@mhowlett that audit trail looks normal for an index document call; Request and Response are not captured by default, the request is written to the HttpRequest stream directly and the response is deserialized from the response stream directly.

[mhowlett]

The code below is what I'm currently playing with. It picks up data retrieved (from kafka) in another thread but everything related to elasticsearch is now happening on the one thread (exactly as below - i just removed a few lines setting up UserSummary to mask what data I'm playing with). UserSummary is annotated with [ElasticsearchType] and properties with [Keyword] etc.

lsof -a -p <process-pid> | wc -l is still incrementing by one with every call to IndexAsync. I verified it's not other parts of the code causing the extra TCP fd's by commenting out the call to IndexAsync.

I think this would be easy to debug if i could step into ElasticSearch.Net. I'll attempt to do that next.

                var settings = new ConnectionSettings(new Uri($"http://{elasticHost}:9200"))
                    .DefaultIndex("myindex");
                var ec = new Nest.ElasticClient(settings);
            
                while (true) 
                { 
                    lock (lockObj)
                    {
                        for (int i=0; i<queue.Count; ++i)
                        {
                            var summary = queue[i];
                            var elasticSummary = new UserSummary
                            {
                                Id = keyQueue[i],
                                Username = summary.Username,
                                FullName = summary.FullName == "" ? null : summary.FullName,
                                ... etc etc
                            };

                            var r = ec.IndexAsync(elasticSummary).Result;
                            if (r.Created)
                            {
                                Console.WriteLine("added: [" + elasticSummary.Username + "]   " + elasticSummary.FullName);
                            }
                            else 
                            {    
                                Console.WriteLine("DBG: " + r.DebugInformation);
                                Console.WriteLine("EXC: " + r.OriginalException.Message);
                            }
                        }
                        queue.Clear();
                        keyQueue.Clear();
                    }
                    Thread.Sleep(1000); 
                }

[russcam]

Thanks for the update @mhowlett.

var r = ec.IndexAsync(elasticSummary).Result;

could you also try this with the synchronous method, var r = ec.Index(elasticSummary) ?

It might be worth copying just the HttpConnection-CoreFx.cs file and setting this as the IConnection to use, and then stepping through it; this might be a little easier than stepping through all of the Elasticsearch.Net code, at least to begin. You can set the IConnection to use on ConnectionSettings

[mhowlett]

Headers?.GetHashCode() is the problem - it's returning a different value each time. I think GetHashCode is being inherited from Object (which is different for different objects). I could do a PR if you like, but probably better that someone who is more familiar with the code do that - shouldn't be too hard, look forward to it :-)

https://github.com/elastic/elasticsearch-net/blob/master/src/Elasticsearch.Net/Transport/Pipeline/RequestData.cs#L153

[russcam]

Headers?.GetHashCode()

😢 I'll look at a fix for it now. Thank you for digging through

[mhowlett]

cool. if you want me to verify, feel free to point me towards a branch.

[russcam]

++ thanks, that would be great! I should have something shortly

[mhowlett]

ok. fyi: back online in about 10 hours.

[russcam]

The only properties of RequestData that are fixed for a HttpClient instance are

• RequestTimeout
• HttpCompression
• ProxyAddress
• ProxyUsername
• ProxyPassword
• DisableAutomaticProxyDetection

The rest of RequestData properties are used to set properties on the HttpRequestMessage so these don't need to be part of the hash code generation or equality checks.