HttpClient prefers IP (from DNS) from the same network as host #27734
Comments
|
@Elufimov AFAIK we call DNS subsystem and don't make any preferences. Can you check what the DNS subsystem returns in your case? |
|
cc @wfurt |
|
One part is that we have connection pool and existing connections will be reused. That may be one reason why you don't see expected distribution. You can try to set Connection: close header to force new connection for each request - just as a experiment. |
Updated codeusing System;
using System.Diagnostics;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
namespace PravoRu.DataLake.DNS.Test
{
class Program
{
static void Main(string[] args)
{
var uri = new Uri(args[0]);
var host = new HostBuilder()
.ConfigureServices((context, collection) =>
{
collection
.AddHttpClient("1", client =>
{
client.DefaultRequestHeaders.Connection.Add("close");
})
.SetHandlerLifetime(TimeSpan.FromSeconds(5))
.ConfigurePrimaryHttpMessageHandler(builder =>
new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromSeconds(5)
}
);
})
.Build();
var httpClient = host.Services.GetService<IHttpClientFactory>().CreateClient("1");
var dsnResolvingTask = Task.Run(() =>
{
while (true)
{
try
{
var ipAddresses = Dns.GetHostAddressesAsync(uri.DnsSafeHost).GetAwaiter().GetResult();
Console.WriteLine($"{DateTime.Now} For {args[0]} was resolved: {string.Join<IPAddress>(",", ipAddresses)}");
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
Thread.Sleep(TimeSpan.FromSeconds(1));
}
});
var httpRequestsTask = Task.Run(() =>
{
while (true)
{
var sw = Stopwatch.StartNew();
var response = httpClient.GetAsync(uri).GetAwaiter().GetResult();
sw.Stop();
if(!response.IsSuccessStatusCode)
{
Console.WriteLine($"{DateTime.Now} Request was unsuccessful with return code: {response.StatusCode.ToString()} and time: {sw.ElapsedMilliseconds} ms");
}
}
});
Task.WaitAll(dsnResolvingTask, httpRequestsTask);
}
}
}@karelz I checked dns resolved in codeAs one can see @wfurt it drastically decreases performance form ~5000 to ~1700 in 10s. The behaviour was not change. dns resolved by digAnd I want to notice that everything works fine with jvm. If one disable dns caching of course. |
|
I just checked under windows server 2016 and everything was the same. |
|
@Elufimov what do you mean by "everything was the same" on Windows 2016? Same problem experienced? |
|
@karelz yes, I have the same problem on windows 2016. You can find dig output in my previous message. I can run something else if it not what needed. But dig shows that consul returns ips in random order like it should. Also simple ping uses all ips in random order. |
|
It looks like we are returning addresses in the order that is given to us by OS API: We need to debug into that to confirm. In the meantime if you can confirm that GetAddrInfoW returns different order on each call, that would be very helpful. |
|
I wrote simple test in Test code [Fact]
public void Issue33037_TryGetAddrInfo()
{
Dictionary<string, int> results = new Dictionary<string, int>();
List<int> iterations = Enumerable.Range(0, 10000).ToList();
iterations.ForEach((i) =>
{
IPHostEntry hostEntry;
int nativeErrorCode;
SocketError error = NameResolutionPal.TryGetAddrInfo("searchcases.service.caravan.consul", out hostEntry, out nativeErrorCode);
Assert.Equal(SocketError.Success, error);
var ipsString = string.Join<IPAddress>(";", hostEntry.AddressList);
var value = results.GetValueOrDefault(ipsString, 0);
results[ipsString] = value + 1;
});
var lines = results.Select(entity => entity.Key + ": " + entity.Value.ToString());
Console.WriteLine(string.Join(Environment.NewLine, lines));
}Test output: Also I can confirm that |
|
@Elufimov, I'm trying to follow along with the thread... you're suggesting that GetHostAddressesAsync is sorting the results returned by NameResolutionPal? Where? |
|
@stephentoub I am suggesting that |
|
My case is that I have consul as dns server and i have service with 4 ip addresses (172.18.2.20, 172.18.2.21, 10.50.0.72, 10.50.0.73). When I use domain name managed by consul in httpclient it do not provide a round robin load balancing like it should, it uses only addresses from 172 subnet if they exist in dns response. I do not think that is something wrong with consul because dig and our jvm applications work fine. |
Ok. In that case it's a question about Windows behavior, rather than .NET Core. SafeFreeAddrInfo.GetAddrInfo just P/Invokes to Windows: unless you're suggesting that GetAddrInfoW is the wrong method to be consuming? |
|
I have the same behavior on windows and on linux. I will be glad to collect additional data and samples. |
karelz
changed the title
|
Triage: We believe that this scenario will be enabled by implementing custom dialers (ConnectionCallback) http://github.com/dotnet/corefx/issues/35404 (#28721 after migration to runtime repo), where anyone can customize what they want to pick. |
|
This has been resolved via the API added here in .NET 5: #41949 |
ghost
locked as resolved and limited conversation to collaborators
Env:
Consul server of 3 master nodes with dns enable. 4 servers registered by one name, 172.18.2.20,172.18.2.21,10.50.0.72,10.50.0.73. Ip of host server is 172.18.1.250. Consul by default has 0 ttl on dns query and shuffle ips.
As one can see consul return ips in random order for each query.
The test code looks like:
From the start requests split almost equally on 172.18.2.20,172.18.2.21 and nothing goes to 10.50.0.72,10.50.0.73. HttpClient send request to 10.50.0.72,10.50.0.73 only if 172.18.2.20,172.18.2.21 were removed from consul. And if we reintroduce even one of 172.18.2.20,172.18.2.21 in consul it makes all request go to 172.18.2.20 or 172.18.2.21.
The text was updated successfully, but these errors were encountered: