Limit concurrency of zonal statistics jobs

[j08lue]

Currently, if a single user requests zonal statistics for 3 years worth of daily data, that will fire off more than 1000 Lambda executions at the exact same time. 1000 is the default global account limit of concurrent requests. We can easily increase that (we just requested 10k, just in case), but such a massive concurrent execution is probably not a good idea.

We need to address this programmatically - even when a user sends off a request like this, it should get rejected (gracefully) by the API. Maybe we can also reduce concurrency in the backend by limiting the number of "workers" that handle the per-file extraction jobs?

As @amarouane-ABDELHAK pointed out, another guardrail to put on could be to limit the number of concurrent Lambda executions for the worker Lambda, if that is possible. That way, executions of this particular function will not be able to exhaust the global account limit.

Acceptance criteria

• Created tickets in relevant projects to address this challenge and linked them here
• Limited concurrency in one and/or another way

[j08lue]

Already discussed a bit with @vincentsarago here NASA-IMPACT/veda-ui#660 (comment)

[amarouane-ABDELHAK]

I’m still examining the concurrent Lambda executions, and it appears that we should be in good shape. The rasterapi Lambda function processing duration is super short (10 ms when it is warm).
I even attempted to submit 26,000 requests in parallel, but it seems that only 100 concurrent executions were triggered ( I believe this behavior is probably because of Lambda’s automatic queuing mechanism).

[vincentsarago]

@j08lue looking back at this I think I misunderstood the original issue. This is not related to titiler-pgstac but more a frontend (allowing the user such a big interval) / backend configuration (number of concurrent lambda as pointed but @amarouane-ABDELHAK) issue.

[j08lue]

Ah, I just realized that it is the client (frontend) that issues all the parallel requests. I had assumed that the frontend made a single request and then the backend would calculate and spawn all the processing jobs.

Yes, in that case, it is just a matter of limiting what users do in the UI.

• [Analysis page] Zonal statistics jobs need guardrails for concurrency / number of requests veda-ui#660

Plus safeguarding that concurrency limit in AWS, if possible.

[j08lue]

Limits implemented in FE.