Cannot assign request address

[laosandegudai]

NEST/Elasticsearch.Net version:6.6

Elasticsearch version:6.7

Description of the problem including expected versus actual behavior:
When big data index

Steps to reproduce:
1.
2.
3.

Provide ConnectionSettings (if relevant):
var path = Environment.GetEnvironmentVariable("ESURL");
if (string.IsNullOrWhiteSpace(path))
{
throw new Exception("the Environment vairable ESURL is not exist");
}

        var pathArr = path.Split(",", StringSplitOptions.RemoveEmptyEntries);
        IEnumerable<Uri> nodes = pathArr.ToList().Select(s => new Uri(s));
        var pool = new StaticConnectionPool(nodes);
        var settings = new ConnectionSettings(pool).DefaultTypeName(ESClient.DOCUMENTTYPE).ConnectionLimit(1000);

Provide DebugInformation (if relevant):

Describe the feature:

[russcam]

This looks like an issue in your environment @laosandegudai. What does http://es:9200 resolve to?

[laosandegudai]

This looks like an issue in your environment @laosandegudai. What does http://es:9200 resolve to?
My environment is right ,it works well a little , alaways happyend one hour time after running.

We deploy.net applications and elasticsearch via docker swarm, where all containers use the same network, es resolves to elasticsearch containers

[biaotang]

@russcam
Through the command netstat -n | grep 9200 | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}' on the application docker container, I found many ESTABLISHED tcp connection when large amount of data was inserting into es, nest client doesn't release or close the tcp connection. The value of ESTABLISHED connection grows to 20 thousand. Doesn't the nest client reuse the connection ?

[Mpdreamz]

It should @biaotang can you share more about the docker images and host?

[Mpdreamz]

Also be sure you adhere to our lifetime guidelines:

https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/lifetimes.html

Reusing HttpClient is per ConnectionSettings

[biaotang]

It should @biaotang can you share more about the docker images and host?

@Mpdreamz I am sorry to reply so late.

The information related deployment environment as follow:

[root@node1 ~]# docker --version
Docker version 18.03.1-ce, build 9ee9f40

[root@node1 opt]# docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
1q4vyjnyp7yktsdq1v68vbwxx *   node1               Ready               Active              Leader              18.03.1-ce
fu8eithrodqqu1gqyu74bc0xx     node2               Ready               Active              Reachable           18.03.1-ce

.net application named edoc2, image built base on dotnet-runtime-2.2.0

es image : docker.elastic.co/elasticsearch/elasticsearch:6.7.1

(deployed using docker stack) compose file as follow:

version: '3.3'

services:
  edoc2:
    hostname: edoc2
    image: http://myregistry.com/edoc2:latest
    labels:
      "type": "1"
    environment:
      - ESURL=http://es:9200
    networks: 
      - edoc2
    deploy:
      mode: global
      endpoint_mode: dnsrr
      restart_policy:
        condition: on-failure
  es:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
    hostname: es
    volumes:
      - /home/edoc2/data/es:/usr/share/elasticsearch/data
    labels:
      "type": "2"
    networks: 
      - edoc2
    environment:
      - node.name=es
      - ES_JAVA_OPTS=-Xmx2g -Xms2g
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
  nginx: 
    hostname: nginx
    image: nginx:latest
    ports:
      - target: 80
        published: 80
        mode: host
    labels:
      "type": "0"
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
    networks: 
      - edoc2
networks:
  edoc2:
    driver: overlay

code for ESClient :

public class ESClient
    {
        public const string ESSECTION = "ESClient";
        public const string ESURLS = "Urls";
        private static ConnectionSettings _defaultConnectionSettings;
        private ConnectionSettings _connectionSettings;
        private ElasticClient _elasticClient;
        public ESClient()
        {
            _connectionSettings = _defaultConnectionSettings;
        }
        public ESClient(IConfiguration configuration)
        {
            var urls = configuration.GetSection($"{ESSECTION}:{ESURLS}");
            if (!string.IsNullOrWhiteSpace(urls.Value))
            {
                var nodes = urls.Value.Split(';').Select(x => new Uri(x));
                var pool = new StaticConnectionPool(nodes);
                _connectionSettings = new ConnectionSettings(pool);
            }
        }

        public static void AddESClient(ConnectionSettings connectionSettings)
        {
            _defaultConnectionSettings = connectionSettings;
        }

        private void InitESClient()
        {
            if (_elasticClient == null)
            {
                if (_connectionSettings != null)
                {
                    _elasticClient = new ElasticClient(_connectionSettings);
                }
                else
                {
                    throw new Exception("ESClient init failed！");
                }
            }
        }

        public string Index<T>(string indexName, T data) where T : class
        {
            InitESClient();
            var response = _elasticClient.Index<T>(data, idx => idx.Index(indexName));
            Console.WriteLine($"{response.ApiCall.Success},{ response.DebugInformation}");
            return response.Id;
        }

    }

code for ESClientProvider:

public static class ESClientProvider
    {
        private static ESClient _client;

        public static ESClient GetESClient()
        {
            if (_client != null)
            {
                return _client;
            }

            var path = Environment.GetEnvironmentVariable("ESURL");
            if (string.IsNullOrWhiteSpace(path))
            {
                throw new Exception("the Environment vairable ESURL is not exist");
            }
        
            var pathArr = path.Split(",", StringSplitOptions.RemoveEmptyEntries);
            IEnumerable<Uri> nodes = pathArr.ToList().Select(s=>new Uri(s));
            var pool = new StaticConnectionPool(nodes);
            var settings = new ConnectionSettings(pool).ConnectionLimit(1000);
            ESClient.AddESClient(settings);
            ESClient client = new ESClient();
            _client = client;
            return _client;
        }
    }

The application created three instances through ESClientProvider, different instance doing different tasks.

According to my understanding， every instance created limit the connection 1000 by the option connectionLimits, three instances at most 3000 concurrent connections and then HttpClient will reuse the connection. Is it right ?

However, I found 20,000 connections from edoc2 with the status ESTABLISHED

[Mpdreamz]

Thanks for the detailed response!

I have reported on socket reuse issues on linux with .NET Core before:

https://github.com/dotnet/corefx/issues/21227

It's one of the reasons our connection limit on linux is so conservative

https://github.com/elastic/elasticsearch-net/blob/master/src/Elasticsearch.Net/Configuration/ConnectionConfiguration.cs#L24

Going from Environment.ProcessorCount to 1000 might amplify the bleed.

Are you compiling AND running on .NET Core 2.1? Can you share the <TargetFramework> of your csproj?

[Mpdreamz]

For full reference how are you calling ESClientProvider.GetESClient() and storing the result and ESClient.Index()?

[biaotang]

Thanks for the detailed response!

I have reported on socket reuse issues on linux with .NET Core before:

dotnet/corefx#21227

It's one of the reasons our connection limit on linux is so conservative

https://github.com/elastic/elasticsearch-net/blob/master/src/Elasticsearch.Net/Configuration/ConnectionConfiguration.cs#L24

Going from Environment.ProcessorCount to 1000 might amplify the bleed.

Are you compiling AND running on .NET Core 2.1? Can you share the <TargetFramework> of your csproj?

I am sorry, the information provided before is a bit wrong.

I confirmed that the dotnet runtime of application is using version 2.2, compiling and running on .NET Core 2.2 .

The <TargetFramework> of edoc2 application is .NET Core 2.2 .

Output of dotnet --info

root@edoc2:/app# dotnet --info

Host (useful for support):
  Version: 2.2.4
  Commit:  f95848e524

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

The logs are written very frequently and we have not implemented bulk writes yet.

Before trying to modify the default value 80 to 1000, i have tried the default value of "80", also got an exception Cannot assign request address， I thought it would be fine to increase this value，however it doesn't work and large amount of ESTABLISHED connection appear

[biaotang]

For full reference how are you calling ESClientProvider.GetESClient() and storing the result and ESClient.Index()?

such as writing logs, calling ESClientProvider.GetESClient() as follow :

    public static class LogsProvider
    {
        private static string indexName = "filelog";
        private static ESClient client = ESClientProvider.GetESClient();

        public static string WriteLogs(FileLogDto fileLogDto)
        {
            return client.Index(indexName, fileLogDto);
        }
   }

[laosandegudai]

storing the result and ESClient.Index()?
yes,storing the result and ESClient.Index(). I use Nest.

[russcam]

@biaotang would you be able to put together a small example that reproduces what you're seeing? Having a small example to work with will make this much easier to investigate.

[russcam]

Looks like this may be https://github.com/dotnet/corefx/issues/37044

[biaotang]

Looks like this may be dotnet/corefx#37044

Not the same Exception, I have restored the configuration "ConnectionLimit" to 80, and cancel the ping request for single ES node, I saw the connection had been recycled。 Hope these will help !
I will continue to follow up and update . Thanks for your help ， @russcam @Mpdreamz