Increasing REST API Throttling errors in SharePoint Online

[slavag]

Category
Question
Typo
Bug
Additional article idea
Expected or Desired Behavior
Before July 17 2019 we got 429 errors here and there, but their ratio was not significant and allowed our production servers to work as expected.

Observed Behavior
Starting July 17 2019, we see 10 times more 429 exceptions on our Sharepoint Online REST API calls, which makes our production to slow down operations 10 times.

Steps to Reproduce
Which tenant has the issue?
All tenants that we accessing (hundreds, across different regions over the world).

When did the issue happen, so that we can check right log entries?
Since around July 17 2019.
Related #4336

Issue now is more severe, doesn't matter if this is OAuth authentication (using our APP) or basic (without any app ID). No any parallel request, only 1 concurrent and it's heavy throttled.

If in the past we saw jump in 10 times more throttled reponses, now it close to 30 times more, this is very strange behaviour from SP (we see this last 4 days) !!!!!

Thanks

[msft-github-bot]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[slavag]

@JeremyKelley, this is something new, we never saw such high throttling rate, even if there's no OAuth used (basic auth means no APP ID is used). Single request, not parallel.

@justdevelopment do you see any high throttling rate ?
Thanks

[MRuimerman]

Hi Slavag, (on behalf of my colleague @justdevelopment), yes we still have the throttling issues.

I am not sure if the problem is the same because we mostly use Graph calls and not REST, but we get the too many requests since July 17 as well.

[slavag]

@MRuimerman Do you see any higher rate since last week ? Thanks

[MRuimerman]

No, the number of errors is more or less the same.

[wadehgsk]

I am experiencing the same problem/behavior starting 26 Aug on our [large] tenancy. Process that ran reliably literally for years started failing with massive numbers of 429 errors. I have throttling detection and handling in place, but jobs are still not completing in a reasonable time or at all. Need some relief.

[MRuimerman]

it is worse today again for us.

[slavag]

Last 5 days we see very high throttling rate.
All servers are most of the time in throttling sleep.

[slavag]

It's not getting any better, jobs that are used to finish in few hours now takes more than week.
Microsoft, please respond, this is total denial of service by Sharepoint.

[vipulkelkar]

We are facing similar issue this week. Observed it when applying PnP templates on multiple sites (about 4 sites) in parallel....which used to work earlier !