🐛 Bug Report: Storage rate limit while uploading big file

[joern-h]

👟 Reproduction steps

Upload a big file (1GB) to a Storage Bucket
I use sdk-for-flutter which uploads files in 5MB chunks

👍 Expected behavior

It should upload the file without rate limiting.

Uploading the file from the Appwrite console works.

👎 Actual Behavior

Upload will be rate limited:

Client log:

Error: AppwriteException: general_rate_limit_exceeded, Rate limit for the current endpoint has been exceeded. Please try again after some time. (429)

Server log:

appwrite  | [Error] Timestamp: 2023-10-11T08:59:29+00:00
appwrite  | [Error] Method: POST
appwrite  | [Error] URL: /v1/storage/buckets/:bucketId/files
appwrite  | [Error] Type: Appwrite\Extend\Exception
appwrite  | [Error] Message: Rate limit for the current endpoint has been exceeded. Please try again after some time.
appwrite  | [Error] File: /usr/src/code/app/controllers/shared/api.php
appwrite  | [Error] Line: 169

🎲 Appwrite version

Version 1.4.x

💻 Operating system

Linux

🧱 Your Environment

Appwrite 1.4.5
Flutter SDK: 11.0.0

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[stnguyen90]

@joern-h, thanks for raising this issue! 🙏🏼 It looks like the rate limit (60 requests in 60 seconds) is being hit because the chunking causes the upload to be broken down into multiple requests. I'm curious to see how many others face this problem 🧐

Related issue:

• 🚀 Feature: Configurable rate limits #2953

[joern-h]

@stnguyen90 I don't think that there are many others, because the default file size limit is only 30MB and this would only result in 6 chunks.
Unfortunately, I need to store files that need more chunks than the current limit supports.