[Bug]: Upgrading coolify version causes scheduled database backups to fail

[steventsvetkov]

Description

After upgrading the coolify version, scheduled database backup start failing. It looks like it's unable to pull the coolify-helper image from ghcr and retries to do so at an impressive polling rate of 271.469µs, quickly hitting the github API rate limit?

This is fixed by triggering a manual backup, after which scheduled backups start working as expected. Note that I have only one database running on my VPS, I'm not sure how it would behave with more than one. I've recently started using Coolify and I've upgraded twice so far, this issue appeared both times.

Minimal Reproduction (if possible, example repository)

1. Create a new project and a stage environment for it.
2. Create a MongoDB database in stage with public access enabled.
3. Configure S3 storage using R2 Cloudflare
4. Setup automatic backups to trigger every hour and upload to S3
5. Upgrade the Coolify version using the UI
6. Database backups start failing
7. Trigger a manual backup to fix this issue.

Exception or Error

Unable to find image 'ghcr.io/coollabsio/coolify-helper:latest' locally
docker: Error response from daemon: toomanyrequests: retry-after: 271.469µs, allowed: 44000/minute.
See 'docker run --help'.

DatabaseBackupJob failed with: Unable to find image 'ghcr.io/coollabsio/coolify-helper:latest' locally
latest: Pulling from coollabsio/coolify-helper
3c854c8cbf46: Pulling fs layer
d2aac0aefc71: Pulling fs layer
f9c396f433c8: Pulling fs layer
a9b023bc0a40: Pulling fs layer
74b8f2dbaf15: Pulling fs layer
4f4fb700ef54: Pulling fs layer
616e93ebdc58: Pulling fs layer
4f4fb700ef54: Waiting
616e93ebdc58: Waiting
a9b023bc0a40: Waiting
docker: error pulling image configuration: download failed after attempts=1: toomanyrequests: retry-after: 1.296341ms, allowed: 44000/minute.
See 'docker run --help'.

Version

329

Cloud?

• Yes
• No

[peaklabs-dev]

This is fixed. Just waiting for andras to release the new version. #3353

[andrasbacsai]

Released a new version with this fix. Can you please update and let us know if it still occurs?

Thanks @peaklabs-dev for handling this on the weekend!

[steventsvetkov]

After updating, it failed again, this time by logging the following error twice

Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 367.879µs, allowed: 44000/minute

Failed to pull helper image: Error response from daemon: toomanyrequests: retry-after: 367.879µs, allowed: 44000/minute

I "fixed" it once again by triggering a manual backup, after which scheduled backups work again. I'll post an update next time I upgrade coolify's version or if this occurs again in another context.

[peaklabs-dev]

@steventsvetkov That is a different error then the one I fixed whiche was this one.

DatabaseBackupJob failed with: Unable to find image 'ghcr.io/coollabsio/coolify-helper:latest' locally

I guess this needs more testing and another fix probably.

[peaklabs-dev]

Just found out that a simmilar rate limiting issue was already reported here: #2930

[steventsvetkov]

I also noticed that the same issue occurs when the Force Docker Cleanup option is turned on for the server that the database is on, which I guess is kind of expected, but I'm adding it here for the sake of completeness.

[andrasbacsai]

I have made a few improvements in the upcoming version. The helper image won't be pruned by Coolify's cleanup process, so it will be downloaded once.

Also the default cleanup schedule will be changed to run at midnight, not every 10 minutes (which is a bit agressive).

Edit: Will be released on Monday morning.

[dickwolff]

@andrasbacsai Would that not cause an issue when the backup is also run at midnight? Or will this be resolved by not pruning the helper image?

[steventsvetkov]

Could changing the cleanup schedule to once a day create possible disk space issues? Or is this only for the Force Docker Cleanup option - the cleanup will still run once the configured threshold % is reached, whenever that is?

[schliesser]

Heyho. having the same issue with the coolify scheduled updates which are timed around midnight.
Manual execution works fine every time.
Version: 331

[steventsvetkov]

I started getting the following error from the scheduled backups at around 7 AM UTC today, while still being on version 331. PullHelperImageJob failed with: Error response from daemon: toomanyrequests: retry-after: 1.257538ms, allowed: 44000/minute

After upgrading to 332, the error continues to log every time a scheduled backup runs. I have hourly backups.

Unlike @schliesser's case, both on 331 and 332, the scheduled backups come up with status success and they do produce a tar on S3, even though there's an error logged. Manual backups work normally. I disabled Force Docker Cleanup a few days ago and still haven't turned it on, disk cleanup has not ran recently either.