[DPE-6762] Catch archive command timeout

[marceloneppel]

Issue

If pgBackRest cannot archive a WAL file in less than 60 seconds, it will fail it's check command and the charm will be stuck in a blocked status.

Solution

In the stanza check step, handle the error code 82 (Archive operation timeout) in the same way we already do for error code 49 (Cannot connect to host) in the stanza initialisation step:

postgresql-operator/src/backups.py

Line 643 in 6ffe0b4

if return_code == 49:

. This way, the user can fix the problem and run juju resolve.

Fixes #784

Tested manually by adding sleep 90 && before the actual archive command in templates/patroni.yml.j2 and deploying the charm + connecting it to Microceph's RadosGW.

pgBackRest error codes extracted from source code: error-codes.csv. I think we shouldn't turn the charm into an error state for every error code, so I'm limiting it to error code 82 for now. We may expand the list of error codes in which we do that in the charm in the future.

Also, sometimes the error message from pgBackRest is sent to stdout instead of stderr (and we don't log them, as seen in #1280), so it's important to check both. I ported the changes from #1320 to this PR to handle that situation.

Checklist

• I have added or updated any relevant documentation.
• I have cleaned any remaining cloud resources from my accounts.

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.83%. Comparing base (6ffe0b4) to head (b6819f1).

Files with missing lines	Patch %	Lines
src/backups.py	84.61%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1328      +/-   ##
==========================================
+ Coverage   75.75%   75.83%   +0.07%     
==========================================
  Files          16       16              
  Lines        4187     4217      +30     
  Branches      633      640       +7     
==========================================
+ Hits         3172     3198      +26     
- Misses        793      796       +3     
- Partials      222      223       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.