S3 backup uploads randomly fail with "V2 signature not supported" — add a signature version setting?

[dvdpearson]

Hey all,

I self-host Coolify (v4.4.6) and back up a few volumes hourly to Backblaze B2. For the past week I've been getting "Storage backup succeeded with a warning" emails every few days. The upload to B2 fails with one of these:

• Unable to initialize new alias from the provided credentials. Get ".../probe-bsign-...": ... network is unreachable
• The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256

Validate connection is always green, and most hours upload fine (about 96% in my case).

I dug into it on my server. Coolify runs mc alias set without --api, so on every backup mc sends a probe request (probe-bsign-...) to guess the signature version. Normally it picks S3v4. But when that probe hits a network blip, mc either errors out (first error) or falls back to S3v2, and B2 rejects S3v2 (second error). Running mc alias set --api S3v2 by hand against my bucket gives the exact same error as the emails, and --api S3v4 works every time.

The rest of Coolify (Validate, retention cleanup) already goes through the AWS SDK with V4 only, so mc is the only place still guessing.

Related issues:

• [Bug]: Database backup to Cloudflare R2 fails with two distinct errors on Coolify Cloud #9792 (same SigV2 error on Cloudflare R2, still in triage)
• [Bug]: S3 Support for Bunny S3 #9754 (Bunny, a comment there suggests --api S3v4 too)
• [Bug]: [v4.0.0-beta.459] S3 backup (Cloudflare R2) fails while local DB backup succeeds #7779 (R2 thread where the SigV2 report first came up)

Proposal: add a "Signature version" setting to each S3 storage, default S3v4, with S3v2 kept for old self-hosted servers. Pass it as --api to every mc alias set (database backups, volume backups, database import). Existing storages get S3v4 through the migration default, so nothing changes for anyone on a normal provider.

I have this working and tested. I'll open a PR against release/v4.5 and link it here, happy to change the approach if you'd prefer something else.

[dvdpearson]

Tried to open the PR but it looks like PRs are limited to existing contributors now. The branch is ready here: release/v4.5...dvdpearson:coolify:feat/s3-signature-version — happy to open it if you can enable that, or feel free to pull it in directly. Thank you!

[dvdpearson]

[kushsharma]

Nice find on the signature fallback. One thing worth doing while the fix waits: treat the "succeeded with a warning" mail as a failure. For those hours the local archive exists and the copy in B2 doesn't, so your only copy is on the same box as the data.

I'd alert on the warning the same way as on an error until --api S3v4 lands, and have something list the bucket once a day and check that the newest object is from the last hour. Validate connection only tests the alias, so it'll stay green through all of this.

[dvdpearson]

@andrasbacsai let me know how I could get some eyes on this, I'd really appreciate it, thanks!

[Cinzya]

Andras is currently busy with fixing a lot of outstanding bugs and UI improvements. So he might not check feature requests at the moment. Considering it is directly related to #9792 and #9754 I would say feel free to leave a comment with a link to this discussion, so he will see if by the time he checks those issues.