S3 upload failure with new version 4.1.100

[lukey78]

Summary

With the latest image update to v4.1.100 our backups in the cluster fail when moving the backups to an S3 storage. This still works with stacks running a previous version.

Docker Hash not working:
sha256:761d8613c2c5921ea71f91408eadfe2ad43294c0731abc5b5e0cfc241696e08a

Logs

DEBUG_MODE=TRUE backup-now

026-03-17.09:45:09 [DEBUG] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Skipping SSL verification for HTTPS S3 Hosts
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-01/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ s3_ssl=--no-verify-ssl
+ [[ -n os.alp2.flow.swiss ]]
+ PARAM_AWS_ENDPOINT_URL=' --endpoint-url https://os.alp2.flow.swiss'
+ silent aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz --no-verify-ssl
+ '[' TRUE = true ']'
+ '[' '' = true ']'
+ '[' debug = debug ']'
+ aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload failed: ./mysql_appDb_db_20260317-094508.sql.gz to s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz SSL validation failed for https://os.alp2.flow.swiss/d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz EOF occurred in violation of protocol (_ssl.c:2427)
+ move_exit_code=1
+ '[' MD5 '!=' none ']'
+ silent run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
+ '[' TRUE = true ']'
+ '[' '' = true ']'
+ '[' debug = debug ']'
+ run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
+ sudo -Eu dbbackup aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload failed: ./mysql_appDb_db_20260317-094508.sql.gz.md5 to s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz.md5 An error occurred (ValidationError) when calling the PutObject operation: The input fails to satisfy the constraints specified by an AWS service.
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ check_exit_code move mysql_appDb_db_20260317-094508.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${1}" in
+ write_log debug 'Move exit Code is 0'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:45:17 [DEBUG] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Move exit Code is 0
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-01/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${move_exit_code}" in
+ write_log error 'Moving of backup '\''mysql_appDb_db_20260317-094508.sql.gz'\'' reported errors'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:45:17 [ERROR] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Moving of backup 'mysql_appDb_db_20260317-094508.sql.gz' reported errors

In contrast, with an older version (in this case 4.1.21) this parts looks like this:

2026-03-17.09:50:46 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Moving backup to S3 Bucket
[...]
+ write_log debug 'Skipping SSL verification for HTTPS S3 Hosts'
2026-03-17.09:50:46 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Skipping SSL verification for HTTPS S3 Hosts
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-02/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ s3_ssl=--no-verify-ssl
+ [[ -n os.alp2.flow.swiss ]]
+ PARAM_AWS_ENDPOINT_URL=' --endpoint-url https://os.alp2.flow.swiss'
+ silent aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz --no-verify-ssl
+ '[' true = true ']'
+ aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload: ./pgsql_app_en101b-db_20260317-095045.sql.gz to s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz
+ move_exit_code=0
+ '[' MD5 '!=' none ']'
+ silent run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
+ '[' true = true ']'
+ run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
+ sudo -Eu dbbackup aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload: ./pgsql_app_en101b-db_20260317-095045.sql.gz.md5 to s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ check_exit_code move pgsql_app_en101b-db_20260317-095045.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${1}" in
+ write_log debug 'Move exit Code is 0'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:50:48 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Move exit Code is 0
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-02/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${move_exit_code}" in
+ write_log debug 'Moving of backup '\''pgsql_app_en101b-db_20260317-095045.sql.gz'\'' completed successfully'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:50:48 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Moving of backup 'pgsql_app_en101b-db_20260317-095045.sql.gz' completed successfully
```

[NeckBeardPrince]

Same issue on 4.9.0 how has this not been addressed yet?!

[NeckBeardPrince]

Wait... by default you set --no-verify-ssl. That would skip SSL verification for https://s3.amazonaws.com.

[lukey78]

This problem seems to be fixed with the new v5.0 version.