-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ReqMgr2 unable to serve too many (30k?) requests to the client - Nginx limitation(?) #11231
Comments
@muhammadimranfarooqi Imran, could you please clarify what those limits are at the Nginx layer and try to check any logs that could confirm this suspicious? |
Alan, I doubt that your conclusion is correct. The 502 gateway errors in fact says that the backend rather then FE is down, and therefore the FE can't reach it. I'm confident that nginx is capable to much higher load and since nginx stays on par with apache it can sustain in our setup around 500 req/sec. I also do not understand what MB in your example represent, Is it single response size or is it total size across all requests. Finally, your curl example does not represent and says anything about the BE performance. As such I suggest the following:
Bottom line, I doubt anything wrong with our FE throughput and rather the actual issue here is poor performance of python back-end servers. I saw only two ways to improve it: |
Valentin, I don't think the problem here is throughput, but rather the ability to serve large data to the end users (it does not matter if it's 1 or 100 requests!). Anyways, I didn't get to any conclusions yet and I am actually trying to identify where the problem is. Let's see what Imran can contribute here. |
My suggestions remains, please clearly define what is "large" data, and measure performance of BE first. And, as a first action please start using gzipped content as whatever you refer as large at the end can be compressed by many folds, and therefore may not require any action on infrastructure and/or FEs. |
Hi @amaltaro We have a dashboard for nginx. Please have a look. |
@muhammadimranfarooqi thanks for pointing us to that dashboard. Unfortunately I fail to see how to map any of that information to this issue to be investigated here. @vkuznet since you have already joined this discussion, can you please follow this up then? Thanks! |
Alan, I have little knowledge of the system and in order even to understand it I need more input from you. For instance:
Said that, I also mentioned that I have little time in Q3 due to my schedule and I may not have time to look into this issue until late Aug. |
Those are all valid questions, but I think you do not need to know any of these in order to evaluate whether CMSWEB frontends/Nginx is the limiting factor here (or if the problem is with the backend, as you previously suggested). I agree that we need to look into this issue taking into consideration longer term design and implementation, but the main goal with this one issue is to identify why we get the 502 Bad Gateway and see if this can be fixed in O(days) instead of weeks/months. |
In nginx setting, we have put these restrictions.
That corresponds to timeout for one of the plot of t0_reqmon in the dashboard Should we increase this 200MB upstream limit? |
thank you for providing these details, @muhammadimranfarooqi . I'd suggest increasing it to 400MB then. Once it's deployed, we can test this specific call once again. |
@muhammadimranfarooqi please do not change yet this setting on production cluster, I'll provide details in a bit |
Yes, @vkuznet , I was waiting for your comment. I did not change anything yet |
@amaltaro , I'm sorry but I completely disagree with your request. The FE are designed for high throughput for ALL CMSWEB service and we simply can't increase resources because BE misbehave. In this case, the increase of body size implies that the FE will be stuck with request and will not serve other clients. As I wrote, since you did not provide any description of API patterns I against any changes on FE. Let me clarify:
The solution is simple, and I already pointed out, BE and clients should start using gzip'ed content. Your API which ships 270MB can be easily fit in FE requirements if it will use gzipped content which will reduce this size by factor of 10-100, and no changes to FE will be required, and not blockage can be done on FE and BE. Please do not address a problem by simply requiring more RAM or resources. We battle it before with DBS and I'm against changing FE settings w/o clear understanding how resources will be used. Moreover, by doubling the size we simply post-pone the same issue on FE again. Do you confident that ReqMgr2 will not ship 400MB body size? I doubt it, since it is matter of time and load on our production systems. As such, any threshold increase should be accomplished with clear description how this happen, what are patterns, what are the clients, what is the rate, etc. Said that, I also understand that we are talking about production service. As such, we may accept a change on FE as a temporary solution to allow current service to provide data to client. But the increase we will do is only change 200MB to 300MB (temporary) which will allow ReqMgr2 API to behave now and force us to change ReqMgr2 behavior asap to use gzipped content with highest priority. Does it sound as an acceptable compromise? |
@muhammadimranfarooqi , we had discussion during today's WMCore meeting, and priority for this issue has been raised. Please proceed and increase threshold on nginx service to what Alan has suggested, i.e. 400MB. This will be a temporary fix until gzipped content will be put in place. @amaltaro , please review memory usage of ReqMgr2 as it has spikes according to this monitoring dashboard. The spikes goes to up to 3GB in RAM (I don't know if they are related to this specific API but it may be the case). From previous experience we know that Python memory management is not ideal, and quite often RAM usage of server can increase significantly when serving large chunks of data. For example, the DBS spikes went up to 10GB and we were unable to run it reliably on k8s infrastructure. |
@muhammadimranfarooqi @vkuznet thanks for updating it. I tried the same query once again and it was successful:
which confirms my suspicioun of the frontend/Nginx limitations. @drkovalskyi if you want to re-enable the original implementation you have on your monitoring setup, could you please report here how it goes? |
And, for the record, the existing json size is 272MB while gzip compression provides 17 times reduction and reduce it to 16MB. As such the transfer time will be smaller too. |
Thanks guys for looking into it. Unfortunately it doesn't work for me. I get:
I use |
@amaltaro , once again could you please clearly specify where the data comes from and what is a data size we should expect? It seems to me that doubling payload size only account for |
Sorry, I mixed up Let me try to expand a bit on number and what to expect:
Regarding the query Dima runs, it's one of the heaviest ReqMgr2 queries because it's basically fetching all the active workflows in the system and their full documents from CouchDB. I could be wrong, but I think we do not have any other system executing such call. Dima can clarify as well, but this call shouldn't be executed more than a handful of times per day. I notice that we no longer get the |
Alan, thanks for this info. Now we need to translate it into the load. If you say we have 19K acquired workflows which translates into roughly 300MB, and according to you previous post it takes 1m17sec, then 30K workflows will give almost 500-600MB but I have no idea how much time service will take to produce this data, and how much time it will take to serialize and transfer this data. If service completes requests within FE interval (5min), then client will get 200 OK, but client still need to fetch full data which accounts for network time (which will reflect in time from BE to FE and nginx). Since I do not know how data are produced and shipped to the client I can't answer this question. Is it possible that time to acquire this data to time to deliver this data to the client will take more than 5min? I do not know. All of this questions depends on how service is implemented, it depends on latency of database, on time spent on serialization of data, etc. In order to answer this question you should go to your service and perform this call w/o any FE involvement, e.g. login to a pod with curl command, and fetch data from your service using curl local network call, e.g. |
BTW, I cannot get |
@amaltaro I think we have a complex problem here (sorry for the long post but it outlines all technical details to understand the issue). I finally was able to figure out how to get proper set of CMS HTTP headers, and set proper curl call to localhost of ReqMgr2 host, i.e. bypass FE entirely. And, I am unable to query ReqMgr2 service request API with ACTIVE value. Below you can see full series of steps which you (or anyone in with access to dmwm namesapce) can repeat. For security reason I strip off port number and I do not provide location of hmac file (but you should know this information). For transparency here are the steps I did:
But I can run sucessfully the following comamnd:
It tells me that the following:
From the above observations, I still conclude that the best solution is to enable gzip encoding. But, since it is not yet in ReqMgr2 and in order to move forward we need the following info:
For the record, here is the code to get CMS HTTP headers using hmac file:
|
@drkovalskyi I think intermediate solution here to get data in chunks, i.e. you can make individual calls to ReqMgr2 service using different states, e.g. |
Thank you for checking this in the reqmgr2 backends, Valentin. I had a look at the WMCore code, and it looks like ReqMgr2 is actually triggering the following POST request to CouchDB (converted to curl):
where it queries a CouchDB view and filters requests against a large list of keys (in addition to asking for the whole document to be transferred). I just executed this call a few times and it takes in average 25secs to finish, the json size - in the file system - is around 315MB. I logged in to vocms0742 - which hosts this database - and executed the same query against localhost. Time to get it server is very similar to the one going through the frontends (25secs as well). I googled for the possibility of requesting compressed data from CouchDB view, but it doesn't seem to be supported. I also googled for that "transfer closed ..." error, and stack overflow suggests a possible tweak/configuration that can be evaluated for Nginx: Lastly, concerning the spikes of memory footprint for ReqMgr2, I think this query could easily give ReqMgr2 an extra GB of memory footprint, since it has to load all this CouchDB data in memory, and then an extra data structure is created after applying some filters on the output of CouchDB. See this method formatting CouchDB output, before it's transferred to the user: |
I already do that. If I exclude |
Actually I now can get |
@amaltaro , thanks for giving more feedback about actual CouchDB query, transfer time and additional links. I think the issue with large payload is related with closed connection within ReqMgr2 API itself. From your stack overflow link I see two solutions:
I rather prefer to adjust HTTP header of the client to re-use HTTP connection instead of changing nginx settings. The nginx provides access to all services and any change to it should be carefully evaluated to not allow breaks in other sub-systems or open a door for various attacks from clients which may happen unintentionally and without knowledge of the client. I'm not sure where exactly in WMCore codebase the HTTP headers are set for queries to CouchDB, but it would be useful if you check this first and see if |
@drkovalskyi I also suggest that you try to setup The problem here is double connection to our FEs:
I do not remember if our FE (and nginx in this case too) propagates client's options through entire HTTP request, and it most likely depends if ReqMgr2 (and WMCore acknowledge) client's HTTP headers. If so, this option should be applied across entire requests, so it worth to try it out. |
@amaltaro, @drkovalskyi where do we stand on this issue? Did you try to use |
@amaltaro , @drkovalskyi did you tried the Keep-Alive patch I provided? If so, does it affect anything? |
@vkuznet not yet Valentin. I have a few other things in the pipeline before I can get to it. I should be able to look into it by the end of this week (this test isn't that simple btw). |
@amaltaro , I got the numbers. All tests passed with the following timing:
Therefore, from this single test I would say that |
Thanks Valentin. I think we can test it with up to 15 concurrent requests (steps of 5). |
I created this ticket https://its.cern.ch/jira/browse/CMSKUBERNETES-185 to perform the tests and asked @muhammadimranfarooqi to provide break down. |
According to results published in https://its.cern.ch/jira/browse/CMSKUBERNETES-185 you can see the following: Said that, the actual fix is to reduce payload size by simply using gzip encoding. |
@amaltaro , do we need keep this open? My understanding that the actual issue is related to large payload which we addressed with gzip encoding. Please make a decision if we need to keep this issue open. |
We have been running beyond 30k requests for the past 2 months, and things are looking much better. Next week we deploy support to gzip response content as well, so I would suggest to close it. Thank you for your contribution on this, Valentin. |
Just an FYI. to test t0_reqmon, and that it required further changes to the script generating the http headers. I decided to consolidate it in this wiki page: |
Thanks @amaltaro and @vkuznet for putting those instructions in one place. These are indeed useful. Alan we may actually link those also from the troubleshooting wiki page as well. |
@todor-ivanov good idea. Can you take care of it? Thanks |
Impact of the bug
ReqMgr2 (but likely affecting any CMSWEB service)
Describe the bug
Dima reported this issue a couple of weeks ago, which is the inability to retrieve active requests from ReqMgr2 through this API:
This problem is tightly coupled to the overload of the system, with around 30k active requests at the moment.
Investigation with the curl command suggests it to be a problem/limitation in the CMSWEB infrastructure [1], failing requests with a
502 Bad Gateway
.The
acquired
status is the currrent worst offender, matching around 19k requests. Which takes more than a minute to be served to the client and transferring a total of 270MB [2]How to reproduce it
Query the API above
Expected behavior
These requests and amount of data will become more common and we need to ensure that our applications and infrastructure is capable of dealing with it.
We should start discussing this with the HTTP experts and understand what are the limitations of NGinx, AFAIK there is a limit of 80MB or so. We should try to fix and/or increase this limits in the CMSWEB cluster such that they meet the CMS user use cases.
For a longer future, we need to start serving compressed data to the users, especially for these heavy data APIs.
FYI: @drkovalskyi
Additional context and error message
[1]
[2]
The text was updated successfully, but these errors were encountered: