Fix/lp 1875481

[ycliuhw]

Checklist

• Checked if it requires a pylibjuju change?
• Added integration tests for the PR?
• Added or updated doc.go related to packages changed?
• Do comments answer the question of why design decisions were made?

---

Description of change

Upgrade-charm now is changed to always trigger pods to restart to finish the remote-init process because the previous way of remote-init for upgrade-charm was not possible to succeed in a container running using non-root USER.

Currently, we got below error when charm was upgraded;

application-mariadb-k8s: 14:05:27 ERROR juju.worker.caasoperator exited "mariadb-k8s/0": executing operation "remote init": caas-unit-init for unit "mariadb-k8s/0" with command: "/var/lib/juju/tools/jujud caas-unit-init --unit unit-mariadb-k8s-0 --charm-dir /tmp/unit-mariadb-k8s-0386991581/charm --upgrade" failed: : command terminated with exit code 1
application-mariadb-k8s: 14:05:30 ERROR juju.worker.uniter resolver loop error: executing operation "remote init": caas-unit-init for unit "mariadb-k8s/0" with command: "/var/lib/juju/tools/jujud caas-unit-init --unit unit-mariadb-k8s-0 --charm-dir /tmp/unit-mariadb-k8s-0972517399/charm --upgrade" failed: ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
: command terminated with exit code 1
application-mariadb-k8s: 14:05:30 ERROR juju.worker.caasoperator exited "mariadb-k8s/0": executing operation "remote init": caas-unit-init for unit "mariadb-k8s/0" with command: "/var/lib/juju/tools/jujud caas-unit-init --unit unit-mariadb-k8s-0 --charm-dir /tmp/unit-mariadb-k8s-0972517399/charm --upgrade" failed: ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
: command terminated with exit code 1
application-mariadb-k8s: 14:05:34 ERROR juju.worker.uniter resolver loop error: executing operation "remote init": caas-unit-init for unit "mariadb-k8s/0" with command: "/var/lib/juju/tools/jujud caas-unit-init --unit unit-mariadb-k8s-0 --charm-dir /tmp/unit-mariadb-k8s-0337241697/charm --upgrade" failed: ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
: command terminated with exit code 1
application-mariadb-k8s: 14:05:34 ERROR juju.worker.caasoperator exited "mariadb-k8s/0": executing operation "remote init": caas-unit-init for unit "mariadb-k8s/0" with command: "/var/lib/juju/tools/jujud caas-unit-init --unit unit-mariadb-k8s-0 --charm-dir /tmp/unit-mariadb-k8s-0337241697/charm --upgrade" failed: ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied
ERROR failed to remove unit tools dir /var/lib/juju/tools/unit-mariadb-k8s-0: unlinkat /var/lib/juju/tools/unit-mariadb-k8s-0/close-port: permission denied

QA steps

# This docker image sets USER to `mysql`;
$ juju run --unit mariadb-k8s/0  'whoami'
mysql

$ juju status --format json | jq '.applications["mariadb-k8s"]["charm-rev"]'
6

$ mkubectl -nt1 get pod/mariadb-k8s-0 -o json | jq '.metadata.annotations["juju.io/charm-modified-version"]'
null

$ mkubectl -nt1 get pod/mariadb-k8s-0 -o json | jq '.status.startTime'
"2020-07-20T05:22:00Z"

# run upgrade-charm
$ juju upgrade-charm mariadb-k8s --path /tmp/charm-builds/mariadb-k8s/ --debug --resource mysql_image=xxx/mariadb

$ juju status --format json | jq '.applications["mariadb-k8s"]["charm-rev"]'
7

$ mkubectl -nt1 get pod/mariadb-k8s-0 -o json | jq '.metadata.annotations["juju.io/charm-modified-version"]'
"7"

$ mkubectl -nt1 get pod/mariadb-k8s-0 -o json | jq '.status.startTime'
"2020-07-20T05:39:30Z"

Documentation changes

No

Bug reference

https://bugs.launchpad.net/juju/+bug/1875481

[wallyworld]

I think we can also include the removal of the UnitUpgrade caas-juju-init mode

[ycliuhw]

$$merge$$