S3 and WebUI not responsive after heavy load by NoobaaFunctions #6265
Description
Environment info
- NooBaa Version: master+30ee3535b5 , built in house with additional dependencies
- Platform: OKD 4.5
Actual behavior
WebUI is not working after a heavy load was produced due to Noobaa-Function calls.
The limit seems to be somewhere around 5k Events handled. Till this border is hit the WebUI is responsive and "Function Statistics" are displayed correctly. After hitting this "limit" WebUI does not display any information.
Durig this situation, no S3 connections to noobaa are possible.
After some time, like 10 min, the WebUI is reachable but the "Function Statistics" is not displaying anything.
Expected behavior
There should be no limit for "Function Statistics" to be displayed.
Steps to reproduce
- Write a minimal handler function and call it for more than 5k times within the displayed time (60 min).
More information - Screenshots / Logs / Other output
We have a static Endpoint-Count of five and have limits set appropriate to the load caused by the function-calls (limit is 4 core 8Gi).
Noobaa-Core is running with limits of 1 Core 4 Gi.
Load Scenario during Upload of files:
WebUI with last update of Statistics function just befor it crashes:
WebUI is no longer updated dynamically and after reload of the page we get:
There is no heavy load on the pods during the this absence of the WebUI. This is the load while it is not working and no load is on the Endpoints:
This situation is present for at least 10 minutes before the WebUI is working properly, but the "Function Statistics" is not displayed.
It would be great if you could give me some target to look for if we reach this situation.