Postgresql charm does not get out of blocked state after removing relation with s3-integrator

[gustavosr98]

Bug Description

I am trying to use the s3-integrator charm to relate to postgresql charm.
First I didn't have the bucket created because I was assuming the s3-integrator charm was going to create it.

After I got the blocked state in the postgresql charm:
I created an s3 bucket in AWS
Updated the s3-integrator charm to use the new existing bucket
But it was still blocked

A second intent after having the postgresql charm in blocked state was to remove the relation between the charms
Postgresql charm was still in blocked state even after relation was gone
Tried adding the relation back but nothing changed

@marceloneppel mentioned the blocked messaged could be related to permissions or something similar.
So, we could improve that message.

To Reproduce

$ juju deploy postgresql pg-m --channel edge --series jammy
$ juju deploy s3-integrator --channel edge --series jammy

# Wait deployment to settle

# How to configure S3 how to docs from mongodb charm
# https://charmhub.io/mongodb/docs/h-configure-s3?channel=5/edge
$ juju run-action s3-integrator/leader sync-s3-credentials access-key=<access-key-here> secret-key=<secret-key-here> --wait

$ aws s3 ls
# None

$ juju config s3-integrator \
    path="pg-m/0" \
    region="us-east-1" \
    bucket="test-bucket"

$ juju relate s3-integrator pg-m

Now in blocked state. Relevant log output

Try updating config with an existing S3 bucket

# Create S3 bucket
$ aws s3 mb s3://pg-bucket-gustavosr98
$ aws s3 ls                   
2023-03-29 14:38:03 pg-bucket-gustavosr98

$ juju config s3-integrator bucket="pg-bucket-gustavosr98"

# Still blocked

Try removing relation

$ juju remove-relation s3-integrator pg-m

$ juju status --relations | grep pg-m | grep s3-integrator
# None

# Still showing as blocked
# even after relation is gone

Try adding relation back

$ juju add-relation s3-integrator pg-m
# Still showing as blocked

Environment

juju: 2.9.42 (Client and Controller)
juju cloud substrate: LXD

postgresql charm: latest/edge (rev 281)
s3-integrator charm: latest/edge (rev 8)

s3 provider: AWS S3

Relevant log output

$ juju status
[..]
pg-m/0*             blocked   idle   0        10.46.90.187           failed to initialize stanza
s3-integrator/0*    active    idle   2        10.46.90.83

$ juju ssh pg-m/0 "sudo less /var/log/juju/unit-pg-m-0.lg"
2023-03-29 19:17:16 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-joined" hook (via hook dispatching script: dispatch)
2023-03-29 19:17:16 WARNING unit.pg-m/0.juju-log server.go:316 s3-parameters:4: Some mandatory fields: ['access-key', 'secret-key'] are not present, do not emit credential change event!
2023-03-29 19:17:17 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-changed" hook (via hook dispatching script: dispatch)
2023-03-29 19:17:17 ERROR unit.pg-m/0.juju-log server.go:316 s3-parameters:4: ERROR: [032]: 'pg-m/0' must begin with / for 'repo1-path' option

2023-03-29 19:17:18 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-changed" hook (via hook dispatching script: dispatch)
2023-03-29 19:18:31 WARNING unit.pg-m/0.juju-log server.go:316 No relation: certificates
2023-03-29 19:18:31 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-29 19:23:55 WARNING unit.pg-m/0.juju-log server.go:316 No relation: certificates
2023-03-29 19:23:55 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-29 19:29:53 WARNING unit.pg-m/0.juju-log server.go:316 No relation: certificates
2023-03-29 19:29:54 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-29 19:35:04 WARNING unit.pg-m/0.juju-log server.go:316 No relation: certificates
2023-03-29 19:35:04 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-29 19:38:20 WARNING unit.pg-m/0.juju-log server.go:316 s3-parameters:4: couldn't initialize stanza due to a blocked status

[gustavosr98]

Trying now on a new deployment using the existing s3 bucket

[gustavosr98]

Same output using the existing s3 bucket on a new deployment

Model      Controller  Cloud/Region         Version  SLA          Timestamp
pg-backup  lxd         localhost/localhost  2.9.42   unsupported  09:58:14-05:00

App            Version  Status   Scale  Charm          Channel  Rev  Exposed  Message
pg-b                    blocked      1  postgresql     edge     281  no       failed to initialize stanza
s3-integrator           active       1  s3-integrator  edge       8  no       

Unit              Workload  Agent  Machine  Public address  Ports  Message
pg-b/0*           blocked   idle   0        10.46.90.84            failed to initialize stanza
s3-integrator/0*  active    idle   1        10.46.90.246           

Machine  State    Address       Inst id        Series  AZ  Message
0        started  10.46.90.84   juju-46d4f8-0  jammy       Running
1        started  10.46.90.246  juju-46d4f8-1  jammy       Running

I am wondering if it is related to the path config value

2023-03-31 14:34:31 INFO juju.worker.uniter resolver.go:159 found queued "leader-elected" hook
2023-03-31 14:35:32 INFO juju.worker.uniter.operation runhook.go:159 ran "leader-elected" hook (via hook dispatching script: dispatch)
2023-03-31 14:35:33 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-31 14:35:33 INFO juju.worker.uniter resolver.go:159 found queued "start" hook
2023-03-31 14:35:33 INFO unit.pg-b/0.juju-log server.go:316 Running legacy hooks/start.
2023-03-31 14:35:41 INFO juju.worker.uniter.operation runhook.go:159 ran "start" hook (via hook dispatching script: dispatch)
2023-03-31 14:35:41 INFO unit.pg-b/0.juju-log server.go:316 database-peers:1: Starting cluster topology observer process
2023-03-31 14:35:41 INFO unit.pg-b/0.juju-log server.go:316 database-peers:1: Started cluster topology observer process with PID 16003
2023-03-31 14:35:42 INFO juju.worker.uniter.operation runhook.go:159 ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-03-31 14:35:44 INFO juju.worker.uniter.operation runhook.go:159 ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-03-31 14:37:09 WARNING unit.pg-b/0.juju-log server.go:316 No relation: certificates
2023-03-31 14:37:10 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-31 14:41:27 WARNING unit.pg-b/0.juju-log server.go:316 No relation: certificates
2023-03-31 14:41:27 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-31 14:46:57 WARNING unit.pg-b/0.juju-log server.go:316 No relation: certificates
2023-03-31 14:46:57 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-31 14:51:05 WARNING unit.pg-b/0.juju-log server.go:316 No relation: certificates
2023-03-31 14:51:05 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-31 14:56:08 WARNING unit.pg-b/0.juju-log server.go:316 No relation: certificates
2023-03-31 14:56:09 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-31 14:57:49 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-created" hook (via hook dispatching script: dispatch)
2023-03-31 14:57:50 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-joined" hook (via hook dispatching script: dispatch)
2023-03-31 14:57:51 WARNING unit.pg-b/0.juju-log server.go:316 s3-parameters:3: Some mandatory fields: ['access-key', 'secret-key'] are not present, do not emit credential change event!
2023-03-31 14:57:51 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-changed" hook (via hook dispatching script: dispatch)
2023-03-31 14:57:52 ERROR unit.pg-b/0.juju-log server.go:316 s3-parameters:3: ERROR: [032]: 'pg-b/0' must begin with / for 'repo1-path' option

2023-03-31 14:57:52 INFO juju.worker.uniter.operation runhook.go:159 ran "s3-parameters-relation-changed" hook (via hook dispatching script: dispatch)

[dragomirp]

Hi, I think you need to set a path configuration for the s3-integrator

[dragomirp]

You can take a look at the configuration we use in our integration tests

[gustavosr98]

@dragomirp path has a value configured

I tried using the same schema that is showed on the mongodb docs of something/something

$ juju config s3-integrator path
pg-b/0

On the logs you can see it is complaining about the value it has

2023-03-31 14:57:52 ERROR unit.pg-b/0.juju-log server.go:316 s3-parameters:3: ERROR: [032]: 'pg-b/0' must begin with / for 'repo1-path' option

[gustavosr98]

@dragomirp

I see, the configuration on the integration tests (/something/something) does look different from the one on the docs (something/something)

[gustavosr98]

Yeah, that value with an / at the begging of the path param fixed it.

The issues are then only:

• The documentation on how to configure s3
• That postgresql charm does not get out of blocked state after removing relation with s3-integrator
• Blocked message could be improved

[dragomirp]

Hi, glad that it worked. We've recently added some documentation of our own that should be published in due course, but it might still be useful to you in the mean time. The path suggested there looks correct. I'll try to double check if there's a typo in the MongoDB docs or if there's a divergence between the charms when I'm able.

[marceloneppel]

Hi @gustavosr98!

We'll be working to fix this on the next pulse.

[marceloneppel]

Hi @gustavosr98! I created a PR that solves this issue and #99, together with other fixes, but it got blocked due to that other fixes. I'm creating another PR with just this issue and #99 fixes.

[gustavosr98]

Awesome @marceloneppel, thanks for the update

[marceloneppel]

Hi @gustavosr98! I separated the fix initially in a k8s charmPR and I'll move it to review today, and them copy the fix to the VM charm (this charm). So, we should have new charm revisions with the fix this week.