Http server performance regression in net6 preview 7 #57656
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
dotnet-issue-labeler
bot
added
the
untriaged
|
while (true) this will always run forever ?! |
|
Tagging subscribers to this area: @dotnet/ncl Issue DetailsDescriptionI have a small benchmark program that runs fast with netcore3.1, net5, net6 preview6 and prior (take 1~5 seconds), but the same program runs super slowly in net6 preview 7 (several minutes ~ infinite) ConfigurationRegression?I can confirm it works fine on netcore3.1, net5, net6 preview 6 and prior DataIt takes 1~5 seconds normally but several minutes ~ infinite with net6 preview 7 AnalysisN/A Codeusing System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
static class Program
{
private const string MimeType = "application/json";
private static readonly HttpClient s_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(15) };
public static async Task Main(string[] args)
{
int n;
if (args.Length < 1 || !int.TryParse(args[0], out n))
{
n = 2000;
}
var port = 30000 + new Random().Next(10000);
var server = CreateWebHostBuilder(port).Build();
using var cts = new CancellationTokenSource();
_ = Task.Factory.StartNew(() => server.Start(), TaskCreationOptions.LongRunning);
var sum = 0;
var api = $"http://localhost:{port}/";
var tasks = new List<Task<int>>(n);
for (var i = 1; i <= n; i++)
{
tasks.Add(SendAsync(api, i));
}
foreach (var task in tasks)
{
sum += await task.ConfigureAwait(false);
}
Console.WriteLine(sum);
System.Environment.Exit(0);
}
private static async Task<int> SendAsync(string api, int value)
{
var payload = JsonConvert.SerializeObject(new Payload { Value = value });
while (true)
{
try
{
var content = new StringContent(payload, Encoding.UTF8, MimeType);
var response = await s_client.PostAsync(api, content).ConfigureAwait(false);
return int.Parse(await response.Content.ReadAsStringAsync().ConfigureAwait(false));
}
catch { }
}
}
private static IWebHostBuilder CreateWebHostBuilder(int port)
{
return WebHost.CreateDefaultBuilder()
.SuppressStatusMessages(true)
.ConfigureLogging((context, logging) =>
{
logging.ClearProviders();
})
.UseKestrel(options =>
{
options.Limits.MaxRequestBodySize = null;
options.ListenLocalhost(port);
})
.UseStartup<Startup>();
}
}
public sealed class MyController : Controller
{
[Route("/")]
public async Task<int> PostAsync()
{
using var sr = new StreamReader(Request.Body);
var bodyText = await sr.ReadToEndAsync().ConfigureAwait(false);
var payload = JsonConvert.DeserializeObject<Payload>(bodyText);
return payload.Value;
}
}
public class Payload
{
[JsonProperty("value")]
public int Value { get; set; }
}
public sealed class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
services.AddMvcCore().AddApplicationPart(Assembly.GetExecutingAssembly());
}
public void Configure(
IApplicationBuilder app,
IWebHostEnvironment env)
{
app.UseRouting();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
}
}
|
|
Hi @hanabi1224, in your catch (Exception ex)
{
Console.WriteLine(ex);
}and report back whether some errors are happening there? |
|
@ManickaP There're tons of errors below, if the input parameter is smaller, like 20 or 200, it works just fine, but with 2000, it never finishes (only in net6 preview 7 tho). |
@JanEggers There's a return inside the loop, it just keeps retrying when error happens |
|
didnt catch that sry |
I'm able to reproduce it locally with 6.0.0-rc.1.21411.2. I also tested the same thing with the current main (~rc2) and it didn't not reproduce. Interestingly, Wireshark shows requests going out, but server never replies (the POST handler is never hit). And since I replaced only runtime libs (not ASP.NET ones) this is suspicious. In the hanging case I see this as last log message on Kestrel (all connections get accepted, only one gets started and that's the end): cc @Tratcher can you thing of anything that could cause this? |
@ManickaP FYI I always run this test in Release mode |
|
@ManickaP can you share the wireshark trace and kestrel logs from your repro? |
|
ASP.NET log: Wireshark: |
|
Wow, the client is sending everything they're supposed to but Kestrel never receives it. It seems like a very low level sockets or threading issue which might explain why updating only the runtime fixed it. I'll try out the repro next week. You can also try reproducing this with just a TcpListener as the server. |
|
Is there a dump available of the server process? It'd be interesting to look at the thread pool's status (e.g. whether there are waiting work items and relatively few threads), and also whether there are any threads blocked synchronously waiting on a Task; I wouldn't be surprised if the issue was the one fixed by #56346. |
No but I can create it. The repro is combined process client+server, do you want me to split it into two and get a dump of the server or would you prefer the combined one? |
|
Just server is preferable, but if it's easier to get one combined, that's fine. |
|
Interestingly, it doesn't reproduce when run in different processes. So I made a dump when run together. Shared offline (it's too big to upload here). |
|
@ManickaP, can you try reverting #56966 and see if the problem comes back? Based on the dump you shared (which shows a lot of threads stuck doing synchronous connects) plus your comment about it not reproducing when client and server were in different processes, I suspect that PR fixed a large portion of the issue. |
|
What's the blocked callstack? The client code given above is async. |
|
@Tratcher see the PR from @stephentoub above. We did a bunch of refactoring around connection creation and pooling, and introduced a dumb bug where connections were getting created synchronously even when using async APIs. |
It does, current main minus ace0f64 hangs. So that explains the problem and confirms it's fixed.
@stephentoub Do you suspect there's some other problem we should investigate? Have you seen any other potential issue within the dump? I'm asking since that quote sounded like #56346 haven't fixed it all. |
No. I think #55642 may have exacerbated things, but that's been taken care of as well. |
|
Should we have microbenchmark coverage of scenarios like this? Or it's a bit too high level/too specific? |
Yes, we should. |
|
The root cause of the issue is fixed, closing. |
|
Putting it into 6.0 milestone where it was addressed. |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
I have a small benchmark program that runs fast with netcore3.1, net5, net6 preview6 and prior (take 1~5 seconds), but the same program runs super slowly in net6 preview 7 (several minutes ~ infinite)
Configuration
Regression?
I can confirm it works fine on netcore3.1, net5, net6 preview 6 and prior
Data
It takes 1~5 seconds normally but several minutes ~ infinite with net6 preview 7
Analysis
N/A
Code
The text was updated successfully, but these errors were encountered: