User Rate Limit Exceeded error when uploading many files

[cyberduck]

1eff184 created the issue

When trying to upload a lot of files (500gb) on/off (testing and trying to find the best solution) I, at some point, got the error 403: User Rate Limit Exceeded when uploading to google drive. I'm uploading to a Cryptomator vault. When I read online the problem could be solved by changing the API settings maybe...

[cyberduck]

@dkocher commented

Potential improvement in 3a98fc6 reducing the number of requests for uploads.

[cyberduck]

1eff184 commented

The latest update seems to've mostly fixed the issue for individual file upload (still some issues sometimes) but the error still occurs every time when drag 'n dropping folders with multiple files or simply multiple files at a time in one transfer (not with one file per transfer).

[cyberduck]

@dkocher commented

Ticket retargeted after milestone closed

[cyberduck]

@dkocher commented

We can reproduce this error in our nightly integration tests.

ch.cyberduck.core.exception.RetriableAccessDeniedException: Cannot create file RVIgMgGk.
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
403 Forbidden
{
  "code": 403,
  "errors": [
    {
      "domain": "usageLimits",
      "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232",
      "reason": "userRateLimitExceeded",
      "extendedHelp": "https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
    }
  ],
  "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
}
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)

[cyberduck]

@dkocher commented

We could successfully increase the Requests per 100 seconds per user. By default, it is set to 100 requests per 100 seconds per user. We could now set it to the maximum allowed limit of 1,000.

[cyberduck]

@dkocher commented

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

[cyberduck]

@dkocher commented

Replying to [comment:12 dkocher]:

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

Your quota request for api-project-996125414232 has been approved and your project quota has been adjusted accordingly.

[cyberduck]

@dkocher commented

Rate limiting implemented in 5e126f4.

[cyberduck]

@dkocher commented

Duplicate in #10867.

[cyberduck]

e95bf7f commented

I'm still getting many 403 errors from Mountain Duck due to rate exceeded with Google Drive. Using Mountain Duck 3.2.3 15107

from the log

{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Rate Limit Exceeded",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Rate Limit Exceeded"
}'}

[cyberduck]

@dkocher commented

#10856 closed as duplicate.

[cyberduck]

@dkocher commented

#10867 closed as duplicate.

[cyberduck]

@dkocher commented

Additional backoff in 637a36b.

[cyberduck]

@dkocher commented

#11736 closed as duplicate.