Charm in blocked status without clear messaging and not cleared after fixing connection to s3 endpoint

[mthaddon]

Steps to reproduce

First, ensure you cannot connect to the s3 backend you configure s3-integrator to connect to. Then:

juju deploy postgresql -n 3 --channel=14/stable
juju deploy s3-integrator --channel=edge
juju run-action s3-integrator/leader sync-s3-credentials access-key=<access-key-here> secret-key=<secret-key-here> --wait

# With swift credentials sourced:
swift list
# No output

juju config s3-integrator bucket='stg-postgresql-test'
juju config s3-integrator endpoint='https://myradosgwendpoint.example.com'
juju config s3-integrator path='/backups'
juju config s3-integrator s3-uri-style='path'

juju relate s3-integrator postgresql

The charm will now go into blocked status with a message of failed to initialize stanza.

Looking at the juju logs we see the following https://paste.ubuntu.com/p/mWzX92kNTB/ with the important part being:

ERROR: [049]: timeout connecting to 'myradosgwendpoint.example.com:443'

Realising this was due to firewall rules, these were updated. However, the charm stays in blocked status.

Expected behavior

The blocked status message is a little cryptic. I have no idea why it failed to initialize stanza, or what stanza it's referring to, or why that's relevant without looking deeper into the juju logs.

I'd expect the charm to recover when the firewall rules are fixed, and/or have some indication of what to do if manual intervention is required.

I believe some of the items mentioned here (particularly around status message) might be covered in #97 but others (like the recovery from inability to access the endpoint) may not.

Actual behavior

See above.

Versions

Operating system: Ubuntu 20.04.6 LTS

Juju CLI: 2.9.42-ubuntu-amd64

Juju agent: 2.9.42

Charm revision: 288

LXD: The provider here is openstack rather than LXD.

Log output

See above.

[github-actions]

https://warthogs.atlassian.net/browse/DPE-1712

[marceloneppel]

Hi @mthaddon! Yeah, some of the items should be solved with the fixes we are creating for #97.

About the charm recovering after the firewall (or some similar) issues are solved, and the S3 settings are later changed (like in the logs), the charm after the fixes related to recovering from the blocked state land for #97.

If a change that doesn't need the credentials or bucket/path change is blocking the charm (like firewall issues that are resolved by allowing traffic in the firewall rules without changing the S3 settings) is needed, we can probably use update status or another process that we run in parallel to the charm process to check it after an interval an unblock the charm.