HEAD - 503 Slow Down

[repat]

Error executing "HeadObject" on "https://xxx.sfo2.digitaloceanspaces.com/xxx/xxx-10-17-02-00-02.zip"; AWS HTTP error: Server error: HEAD https://xxx.sfo2.digitaloceanspaces.com/xxx/xxx2018-10-17-02-00-02.zip resulted in a 503 Slow Down response (server): 503 Slow Down -

related: #618

Any fixes so far? @mdavis1982 @freekmurze @leeuwd

[freekmurze]

I'm thinking this problem should be solved in the FlySystem driver. Either by providing some kind of throttling functionality in the S3 driver or a dedicated Spaces driver that has the right throttling baked in.

[pehbehbeh]

@repat Did you come up with a solution? We are also experiencing this issue with Digital Ocean Spaces when executing the cleanup command. Would be happy to help.

Error executing "DeleteObject" on "https://xxx.digitaloceanspaces.com/xxx/xxx/xxx.zip"; AWS HTTP error: Server error: DELETE https://xxx.digitaloceanspaces.com/xxx/xxx/xxx.zip resulted in a 503 Slow Down response: SlowDown Please reduce your request rate. (truncated...) SlowDown (server): Please reduce your request rate. - SlowDown Please reduce your request rate.

#0 /app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array)
#1 /app/vendor/guzzlehttp/promises/src/Promise.php(203): Aws\WrappedHttpHandler->Aws{closure}(Array)
#2 /app/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, Array)
#3 /app/vendor/guzzlehttp/promises/src/RejectedPromise.php(40): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}(Array)
#4 /app/vendor/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise{closure}()
#5 /app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(98): GuzzleHttp\Promise\TaskQueue->run()
#6 /app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#7 /app/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#8 /app/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#9 /app/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /app/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /app/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /app/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /app/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
#15 /app/vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php(186): Aws\AwsClient->execute(Object(Aws\Command))
#16 /app/vendor/league/flysystem/src/Filesystem.php(237): League\Flysystem\AwsS3v3\AwsS3Adapter->delete('backup/2019-07-...')
#17 /app/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php(313): League\Flysystem\Filesystem->delete('backup/2019-07-...')
#18 /app/vendor/spatie/laravel-backup/src/BackupDestination/Backup.php(57): Illuminate\Filesystem\FilesystemAdapter->delete(Array)
#19 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(88): Spatie\Backup\BackupDestination\Backup->delete()
#20 /app/vendor/laravel/framework/src/Illuminate/Support/Collection.php(475): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->Spatie\Backup\Tasks\Cleanup\Strategies{closure}(Object(Spatie\Backup\BackupDestination\Backup), 0)
#21 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(89): Illuminate\Support\Collection->each(Object(Closure))
#22 /app/vendor/laravel/framework/src/Illuminate/Support/Collection.php(475): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->Spatie\Backup\Tasks\Cleanup\Strategies{closure}(Object(Spatie\Backup\BackupDestination\BackupCollection), 20190709)
#23 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(90): Illuminate\Support\Collection->each(Object(Closure))
#24 /app/vendor/laravel/framework/src/Illuminate/Support/Collection.php(475): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->Spatie\Backup\Tasks\Cleanup\Strategies{closure}(Object(Spatie\Backup\BackupDestination\BackupCollection), 'daily')
#25 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(91): Illuminate\Support\Collection->each(Object(Closure))
#26 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(35): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->removeBackupsForAllPeriodsExceptOne(Object(Illuminate\Support\Collection))
#27 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/CleanupJob.php(42): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->deleteOldBackups(Object(Spatie\Backup\BackupDestination\BackupCollection))
#28 /app/vendor/laravel/framework/src/Illuminate/Support/Collection.php(475): Spatie\Backup\Tasks\Cleanup\CleanupJob->Spatie\Backup\Tasks\Cleanup{closure}(Object(Spatie\Backup\BackupDestination\BackupDestination), 0)
#29 /app/vendor/spatie/laravel-backup/src/Tasks/Cleanup/CleanupJob.php(54): Illuminate\Support\Collection->each(Object(Closure))
#30 /app/vendor/spatie/laravel-backup/src/Commands/CleanupCommand.php(42): Spatie\Backup\Tasks\Cleanup\CleanupJob->run()
#31 [internal function]: Spatie\Backup\Commands\CleanupCommand->handle()
#32 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#33 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()
#34 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#35 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(576): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#36 /app/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\Container\Container->call(Array)
#37 /app/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#38 /app/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#39 /app/vendor/spatie/laravel-backup/src/Commands/BaseCommand.php(16): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#40 /app/vendor/symfony/console/Application.php(908): Spatie\Backup\Commands\BaseCommand->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#41 /app/vendor/symfony/console/Application.php(269): Symfony\Component\Console\Application->doRunCommand(Object(Spatie\Backup\Commands\CleanupCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /app/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#43 /app/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#44 /app/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#45 /app/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#46 {main}

[jasperf]

Just had the same issue now

url resulted in a 503 Slow Down response: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>SlowDown</Code> <Message>Please reduce your request rate.</Message> (truncated...) SlowDown (server): Please reduce your request

Also interested in possible throttling options. Especially for using S3 driver https://github.com/thephpleague/flysystem-aws-s3-v3 as we are for Digital Ocean Spaces. Found https://stackoverflow.com/questions/54364522/how-to-optimise-aws-s3-usage-to-handle-please-reduce-your-request-rate-problem stating

You need to distribute the load across many s3 prefixes.

We use Digital Ocean Spaces and we use one Space only which is 250GB min so plenty for us for now. So do not want to distribute staging and production backup across two Spaces.

Also, this only happened on backups cleanup, not doing the actual backup itself. Funnily enough cleaning staging backups went through fine a minute later. And because cleanup did not work for production we now have 26 backups instead of 25.

[angelocala94]

Any news about this?

[EmilMoe]

Just for reference. Still an issue.

Problem seem to be a bug at DigitalOcean though: https://www.digitalocean.com/community/questions/rate-limiting-on-spaces

[jasperf]

@EmilMoe They say

... at the moment we are rate limiting individual Spaces (the 503 error) that are receiving more than 
200 reqs/s. If you need higher throughput we would ask that you create multiple Spaces and split your
objects among them.

and I doubt Laravel Backup does more than 200 request per second..?

NB They recommend a CDN and refer to https://www.digitalocean.com/docs/spaces/resources/performance-tips/

[EmilMoe]

@jasperf I'm sure that Spatie Backup is not anything near 200/s but making it a CDN makes no sense, however it might solve it, just seems as something is wrong at DO.

They must have a miscalculation in their API.

[jasperf]

We do have files that are larger than 500mb. Like 550-600MB for example and 10-20 of them from two servers. Perhaps we could use multi-part uploads for those as they recommend? Not sure if that is built into Laravel Backup.

Other thing I will look into is timing. Two servers should not backup on the same day and time either.

Just had another SlowDown response:

Exception message: Error executing "ListObjects" on 
"https://app.ams3.digitaloceanspaces.com/?prefix=App-Staging%2F&encoding-type=url"; 
AWS HTTP error: Server error: GET https://app.ams3.digitaloceanspaces.com/?prefix=App-Staging%2F&encoding-type=url resulted in a 503 Slow Down response: 
<?xml version="1.0" encoding="UTF-8"?> 
<Error> 
<Code>SlowDown</Code>
 <Message>Reduce your request rate.</Message> 
</Erro (truncated...) SlowDown (server): 
Reduce your request rate. - 
<?xml version="1.0" encoding="UTF-8"?> 
<Error> 
<Code>SlowDown</Code> 
<Message>Reduce your request rate.</Message> 
</Error>

..

[jasperf]

Even now had this error uploading only a few images to the origin (images are served from the EDGE url). So not even a large zipped file. Opened another ticket at Digital Ocean on this to get to the bottom of this all.

Also again looking into combining files smaller then 1MB and uploading 500+ MB files into parts. That is besides possibly queuing. But do hope there are easier ways here and also do believe rate limiting is still triggered too soon.

[pmochine]

Super weird! I have the same issue and my files are only about 15kb big......

n error occurred while cleaning up the backups of XXX YYY

Exception message: Error executing "DeleteObject" on "https://laravel-backup.fra1.digitaloceanspaces.com/XXX-YYY/staging_cache2020-07-27-02-00-04.zip"; AWS HTTP error: Server error: DELETE https://laravel-backup.fra1.digitaloceanspaces.com/XXX-YYY/staging_cache2020-07-27-02-00-04.zip resulted in a 503 Slow Down response: <?xml version="1.0" encoding="UTF-8"?> <Error> <Message>Reduce your request rate.</Message> <Code>SlowDown</Code> </Erro (truncated...) SlowDown (server): Reduce your request rate. - <?xml version="1.0" encoding="UTF-8"?> <Error> <Message>Reduce your request rate.</Message> <Code>SlowDown</Code> </Error>

Exception trace: #0 /var/www/XXXYYY-staging/releases/25/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array) #1 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(203): Aws\WrappedHttpHandler->Aws{closure}(Array) #2 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, Array) #3 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/RejectedPromise.php(40): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}(Array) #4 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise{closure}() #5 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(118): GuzzleHttp\Promise\TaskQueue->run() #6 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(145): GuzzleHttp\Handler\CurlMultiHandler->tick() #7 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(true) #8 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn() #9 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending() #10 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList() #11 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending() #12 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList() #13 /var/www/XXXYYY-staging/releases/25/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending() #14 /var/www/XXXYYY-staging/releases/25/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait() #15 /var/www/XXXYYY-staging/releases/25/vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php(187): Aws\AwsClient->execute(Object(Aws\Command)) #16 /var/www/XXXYYY-staging/releases/25/vendor/league/flysystem/src/Filesystem.php(237): League\Flysystem\AwsS3v3\AwsS3Adapter->delete('XXX-YYY/stag...') #17 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php(351): League\Flysystem\Filesystem->delete('XXX-YYY/stag...') #18 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/BackupDestination/Backup.php(92): Illuminate\Filesystem\FilesystemAdapter->delete(Array) #19 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/HigherOrderCollectionProxy.php(60): Spatie\Backup\BackupDestination\Backup->delete() #20 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/Traits/EnumeratesValues.php(202): Illuminate\Support\HigherOrderCollectionProxy->Illuminate\Support{closure}(Object(Spatie\Backup\BackupDestination\Backup), 0) #21 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/HigherOrderCollectionProxy.php(61): Illuminate\Support\Collection->each(Object(Closure)) #22 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(87): Illuminate\Support\HigherOrderCollectionProxy->__call('delete', Array) #23 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/Traits/EnumeratesValues.php(202): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->Spatie\Backup\Tasks\Cleanup\Strategies{closure}(Object(Spatie\Backup\BackupDestination\BackupCollection), 202031) #24 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(88): Illuminate\Support\Collection->each(Object(Closure)) #25 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/Traits/EnumeratesValues.php(202): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->Spatie\Backup\Tasks\Cleanup\Strategies{closure}(Object(Spatie\Backup\BackupDestination\BackupCollection), 'weekly') #26 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(89): Illuminate\Support\Collection->each(Object(Closure)) #27 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/Strategies/DefaultStrategy.php(35): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->removeBackupsForAllPeriodsExceptOne(Object(Illuminate\Support\Collection)) #28 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/CleanupJob.php(42): Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy->deleteOldBackups(Object(Spatie\Backup\BackupDestination\BackupCollection)) #29 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Support/Traits/EnumeratesValues.php(202): Spatie\Backup\Tasks\Cleanup\CleanupJob->Spatie\Backup\Tasks\Cleanup{closure}(Object(Spatie\Backup\BackupDestination\BackupDestination), 0) #30 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Tasks/Cleanup/CleanupJob.php(54): Illuminate\Support\Collection->each(Object(Closure)) #31 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Commands/CleanupCommand.php(42): Spatie\Backup\Tasks\Cleanup\CleanupJob->run() #32 [internal function]: Spatie\Backup\Commands\CleanupCommand->handle() #33 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): call_user_func_array(Array, Array) #34 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container{closure}() #35 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure(Object(Closure)) #36 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(39): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure)) #37 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Container/Container.php(596): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL) #38 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Console/Command.php(134): Illuminate\Container\Container->call(Array) #39 /var/www/XXXYYY-staging/releases/25/vendor/symfony/console/Command/Command.php(258): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle)) #40 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle)) #41 /var/www/XXXYYY-staging/releases/25/vendor/spatie/laravel-backup/src/Commands/BaseCommand.php(16): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #42 /var/www/XXXYYY-staging/releases/25/vendor/symfony/console/Application.php(911): Spatie\Backup\Commands\BaseCommand->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #43 /var/www/XXXYYY-staging/releases/25/vendor/symfony/console/Application.php(264): Symfony\Component\Console\Application->doRunCommand(Object(Spatie\Backup\Commands\CleanupCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #44 /var/www/XXXYYY-staging/releases/25/vendor/symfony/console/Application.php(140): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #45 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #46 /var/www/XXXYYY-staging/releases/25/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #47 /var/www/XXXYYY-staging/releases/25/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #48 {main}

[jasperf]

@pmochine Yeah, been at this for a few months now. Spaces is nice and not expensive but as general storage for images I had to cancel using them as I ran into rate limiting errors all the time despite using 4 buckets with them. As for backups, I do still use Digital Ocean Spaces but my Staging backup cleaning has been failing for 3 months now.

Been talking to S3 Adapter package maintainer at https://github.com/thephpleague/flysystem-aws-s3-v3/issues/205 and perhaps exponential backoffs in combination with some other tricks can make this work. But it seems like a hassle and simply tough for me to implement.

For image storage for projects we consider using volumes now. But even considering moving elsewhere. To our Dutch based provider TransIP to be precise. They have Big Storage you can attach to VPSs with relative ease and are not too pricey. I prefer not though as I liked Digital Ocean, their database management, floating ip addresses, API and so on. But this is a serious issue.

NB Using Amazon S3 would also be an option as they are more generous in their limiting, but they are more expensive and calculating cost is a major pain.

[pmochine]

Sorry for annoying everyone who is getting a notification. But @jasperf do you have any experience with Object Storage by vultr.com? They have a limit of 400 requests of seconds, not like DO with 200.

[jasperf]

@pmochine The 400 requests per second Vultr offers is better yet again. Have you used them? I have only worked with one of their servers in the past, not with Object Storage. But if their API is more precise besides the double requests it may be interesting. That and possible CDN locations..

[jpmurray]

Almost a year later, I started to use B2 as a storage provider (since cost is way below S3), and I started to get 429 Too Many Request on each database:cleanup, as they only allow 60 calls per minutes.

Anyone got around an acceptable scenario since the last time this issue was in the spotight ?

[TobyMaxham]

Hey @jpmurray, I found your comment through a Google-Search since we had the same issue. We also used Backblaze B2 as a storage.
Since the Spatie Backup default cleanup strategy \Spatie\Backup\Tasks\Cleanup\Strategies\DefaultStrategy::class checks if a backup exists, it eventualy call the API more than the bought rate limiting.

We have changed the cleanup strategy with our own class and finally everything went ok.

[jasperf]

@TobyMaxham A new cleanup strategy you added I understood. Interesting. Would you mind sharing that with us Toby?

[Loots-it]

I also run into rate limiting problems using this package and Digital Ocean spaces. Is the issue related to this package? I will have a look at the DefaultStrategy class.

[patriziotomato]

Same issue. No one debugged this yet?

[Loots-it]

I think it's an issue with DO

[jpmurray]

I think it's an issue with DO

It is not, I have it with another provider (Backblaze's B2).