Slow performance on Linux #2026
Comments
|
Do you open always on？ |
|
@vincywindy Sure, "Always on" is "on" but it shouldn't has to make a difference after some warm-up time where slow loading parts of the application should respond more quickly.
|
|
Similar issue, the IdentityServer4 project sometimes hangs when calling There is one file |
|
another issue reporting timeout #2121, maybe ef core (or the UnitOfWork) has some issue on linux platform? |
|
I don't suppose that is something related to ABP framework. However, we will deeply investigate the problem. Thank you for your time that you've written a lot of useful info. |
|
@ITRyan First I thought it maybe was about the database connection or the setup too, but if you try this endpoint here, the request had never failed during my tests: https://abp-test-app.azurewebsites.net/api/test Maybe it's a special case with the combination of Linux + EF Core + Identity Server in abp that is causing this behavior. I have also tried following the logs inside the Identity Server code (when you run the app as debug) and it's stuck on random places, so I haven't found a pattern yet. Thanks @hikalkan for looking into this. Let me know if I can help by providing more info to this case. |
|
Hi, we're running into similar issues regarding the auth-server host. But given that for a our sample app we have 2 small microservices + gateway + auth-server, hangs on 8GB ram and 4 cpu cores are not normal at all. When we extract auth-server to run locally on windows, not on docker, the hangs are much less frequent. We tried dockerizing the sample ID4 sample host and auth against it (without ABPs dlls) and there were no lagginnes at all even at 3GB ram/2CPU cores assigned to docker. But we don't want such solution, as we like the work ABP has done for all the stores etc. Going through the comments, the pattern arrises that when it is hosted on linux (or linux based container) there is some issue. Also we did not narrowed down the issue to any special endpoint. Sometimes those are calls for token, user-info, but usually it hangs on introspection validation on random endpoints in other services. |
|
I am also hosting on Linux in combination with Docker and also get random hangs / never responding APIs. Machine has 8 cores and 32 GB RAM with no limitations for Docker. However I do not use IdentityServer so I do not know if it's the same issue here. Just wanted to share this. Might be also an issue on my end. |
|
I get the exact same thing on windows dev env. Endpoints related to identity server are super slow or don't respond at all. |
|
i have the same issue, using in docker+linux |
|
I think this problem from Npgsql Driver which is fixed version 4.1.2 |
|
@ckansiz I've added a patch to the test repo bumping up the version of Npgsql. It does not make a difference, so I'm not sure if this is related. Still the same problem as described in the first post here. https://gitlab.com/abp-test/testapp/commit/27b00a10ab07c641998c0e0d5adbb0869253456f Verified that's indeed the latest version |
I'm using SqlServer, not PostreSQL so its probably not related at all |
|
I am also experiencing this locally on Manjaro with ABP 1.0.2 with SqlServer. Anything generally hitting the UserManager methods fails, rest of the app is fine. Also fine on ABP 0.19.0 SqlServer: mcr.microsoft.com/mssql/server:2017-latest-ubuntu |
|
ABP v0.20 was the first ASP.NET Core 3.0 supported version. |
|
@hikalkan just try running your microservice sample on docker for windows using linux containers. |
|
@hikalkan I have reproduced this with the following setups (all the same blank project generated from https://abp.io/get-started) 1. Slow/hanging setupsVirtualBox
Azure
It does correlate to the CPU count, but I have no idea why. The more CPUs/Cores are available, the less the behavior occurs, but it does not get solved with more resources as the random timeouts indicate. Database connecting and hosting either locally or remote seems to make no difference. 2. WORKING setupsThe following setups works great. And the VM is surprisingly fast for the one CPU core there. VirtualBox
My dev machine
Other dev machine
3. Conclusion
|
|
Maybe this is a threadpool exhaustion problem? The AsyncHelper class is very dangerous in this context. |
|
I've got this working on Azure: https://abp-test-2.azurewebsites.net Currently runs on the App Service for Windows "free" plan and works fine. Uses the same database connection as the Azure App Linux version. Login with admin/1q2w3E* is also working. So definitely related to Linux. |
|
Please be sure NOT to setup tokens caching in microservice settings. This is the way we mitigated this issues, it still occurs but way less often as number of token introspection calls is reduced. |
|
Maybe it is related to the AsyncHelper class, as @Trojaner suggested. And the related issue in the Nito.AsyncEx repo: #StephenCleary/AsyncEx#107 |
|
The application is hanging for me too. Our setup is :
After the application starts up I can make requests to unauthenticated endpoints like In our case, we discovered is that the requests hang when the database is on another VM/server but requests are fine when using the database on localhost. We have tried using host names instead of IP's for the database connection string which didn't change anything.
public override void ConfigureServices(ServiceConfigurationContext context)
{
context.Services.AddCors(options =>
{
options.AddPolicy(DefaultCorsPolicyName, builder =>
{
builder
.WithOrigins(origins)
.SetIsOriginAllowedToAllowWildcardSubdomains()
.AllowAnyHeader()
.AllowAnyMethod()
.AllowCredentials()
.WithAbpExposedHeaders()
.WithExposedHeaders("Content-Disposition");
});
});
context.Services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders =
ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
knownProxies.ForEach(x => { options.KnownProxies.Add(x); });
});
}
public override void OnApplicationInitialization(ApplicationInitializationContext context)
{
app.UseForwardedHeaders();
}NGINX proxy settings are as follows: |
|
I have done more testing. My tests are not controlled so they only give an indication of what changes with different configurations but it reveals some interesting results.
|
|
Working on this problem for a few days and finally it seems related to EF Core or SqlClient:
Still trying to understand. |
|
Status update: I am able to repeat a similar problem in local docker (with linux containers) when I decrease RAM and CPU resources:
First of all, problem is completely unrelated to When I try to login, it hangs and the last log lines are shown above. I opened Verbose (most detailed) logging for EF and ASP.NET Core. I can not repeat it on Windows or Visual Studio. Still working on it... |
|
One thing I noticed with my tests was that the dotnet process spawned more threads with more CPU cores. Reducing memory didn't cause the requests to hang more, but reducing CPU cores did. Could it be related to thread pool starvation or garbage collection issues? Maybe this helps you? Thanks for your investigation, it is greatly appreciated! |
|
Probably found the problem. It seems related to intercepting logic of async methods. I am working on it. However, the solution may not be so easy. I am also considering to completely remove AsyncHelper usages. To be honest I am considering;
Otherwise, eventualy it causes async over sync problem when you have a sync method which is intercepted and the interceptor requires to call an async method. If we go like that, the next version may be 2.0 since it is a huge breaking change :) |
|
After the changes above and using the https://github.com/JSkimming/Castle.Core.AsyncInterceptor all the problems have solved for the https://abp.io/ platform and startup template. Tested on kubernetes and local docker. So, the next version will be 2.0 because of semantic versioning. |
|
If I understood it correctly, 1.2 will become 2.0? |
|
OMG, it is a huge breaking change, so is there a roadmap for 2.0? |
This milestone (https://github.com/abpframework/abp/milestone/26) becomes v2.0 :) |
|
Thanks for publishing v2.0 of abpframework. After testing the release I can confirm that this problem seams to be resolved. No timeouts and hangs in Linux test VMs and on an Azure App Service Linux instance. Now we can switch back to Linux on Azure. 🚀 |
|
@stay2be thank you for your feedback. I am VERY happy to see that. This issue has eaten 1 week of development time to solve 😃 |
|
Thank you very much, I can also confirm that all performance issues are gone now. |
Hi folks,
first let me thank you for this great project. I've learned a lot while working with this framework and the design principles behind it.
I've run into the following problem when deploying a project based on abp to an Azure App Service Linux instance. I'm not sure if this is related to abp or maybe to Identity Server 4.
About the environment:
Findings:
Going into the logs yields that following:
Server insights
Mitigation
It could be related to the Identity Server store implementation, because after removing the include of database related entries in the ID4 discovery endpoint response document (.well-known/openid-configuration), the endpoint doesn't hang anymore.
Testing other parts of the infrastructure
The problem here is not having a cheap instance in production but a low configured instance for staging and development. It's no problem when the application take time respond on auth and some endpoints while development, but the app is not responding after all and seems to be stuck on the communication with the Identity Server 4 hosted instance. All other endpoints are responding fast, even the ones calling the database. The question is, why a higher CPU count/ACU solves the problem.
The text was updated successfully, but these errors were encountered: