[JUJU-3738] - Fix/backend ref count

[ycliuhw]

This PR fixes secret backend refCount issues for:

• model import(increment on target controller)/export(decrement on source controller) during model migration;
• model removal;
• model migration abort;
• secret drain;

drive-by:

• added upgrade steps to create at least one refCount for each secret backend;
• fixed a secret backend model config issue for model migration(including the backend ID in the model data for importing process to handle a potential backend renaming in the target controller);
• fixed a potential data race for the secret drain worker that could happen if the secret backend changed too frequently in a short time period.

Checklist

• Code style: imports ordered, good names, simple structure, etc
• Comments saying why design decisions were made
• Go unit tests, with comments saying what you're testing
• Integration tests, with comments saying what you're testing
• doc.go added or updated in changed packages

QA steps

juju add-secret-backend myvault vault --config ./vault.yaml

juju deploy snappass-test

for i in {1..5}; do juju exec --unit snappass-test/0  -- secret-add --owner unit owned-by=easyrsa/0; done

for i in {1..5}; do juju exec --unit snappass-test/0  -- secret-add owned-by=easyrsa; done

juju model-config secret-backend=myvault

juju model-config secret-backend
myvault

export backend_id=$(juju show-secret-backend myvault | yq .myvault.id)

juju bootstrap microk8s k2

juju add-secret-backend myvault1 vault --config ./vault.yaml --import-id $backend_id

juju switch k1:t1

juju migrate k1:t1 k2

juju switch k2:t1

juju model-config secret-backend
myvault1

# check refCount in mongo on the source controller;
juju:PRIMARY> db.globalRefcounts.find({_id: "secretbackend#revisions#64782241fe5c050027770143"}).pretty()
{
	"_id" : "secretbackend#revisions#64782241fe5c050027770143",
	"refcount" : 0,
	"txn-revno" : NumberLong(11)
}

# check refCount in mongo on the target controller;
juju:PRIMARY> db.globalRefcounts.find({_id: "secretbackend#revisions#64782241fe5c050027770143"}).pretty()
{
	"_id" : "secretbackend#revisions#64782241fe5c050027770143",
	"refcount" : 8,
	"txn-revno" : NumberLong(3)
}

juju remove-secret-backend myvault1
ERROR backend "myvault1" still contains secret content

juju destroy-model -y --debug --destroy-storage k2:t1

# check refCount in mongo on target controller again;
juju:PRIMARY> db.globalRefcounts.find({_id: "secretbackend#revisions#64782241fe5c050027770143"}).pretty()
{
	"_id" : "secretbackend#revisions#64782241fe5c050027770143",
	"refcount" : 0,
	"txn-revno" : NumberLong(3)
}

juju remove-secret-backend myvault1

Documentation changes

No

Bug reference

No

[ycliuhw]

/build

[ycliuhw]

/build

[wallyworld]

Thank you, just a couple of small things to fix.

[ycliuhw]

/merge