Pebble Error cannot create directory: permission denied when pushing file to container on Juju 3.5.1

[NohaIhab]

Bug Description

On Juju 3.5.1

This issue is blocking the deployment and the CI for some CKF charms.
As reported in this Juju 3.5.1 bug, our charms are not able to push files with pebble to the workload container with the error:

ops.pebble.PathError: permission-denied - cannot create directory: mkdir <dir_name>: permission denied

where <dir_name> is different for every charm, depending on what file it is attempting to push.

It looks like pebble does not have permissions to write files to workload containers.

This bug is known to affect the following charms, where the CI is failing in the corresponding PRs:

• jupyter-ui PR
• kfp-profile-controller PR
• kfp-persistence PR (same PR as kfp-profile-controller)

In our CI, we have Juju pinned to 3.5/stable, and for the Juju snap 3.5/stable is pinned to 3.5.1:

snap info juju | grep 3.5/stable
tracking:     3.5/stable
  3.5/stable:    3.5.1             2024-05-30 (27225)  99MB -

i.e. we have 3.5.1 in all of our CI at the moment of creating this issue.

The CI was seen to fail in main for the listed charms. Similarly, we expect this to break in the track/<version> branches.

To Reproduce

1. Install Juju 3.5/stable
2. Install microk8s 1.25-strict/stable
3. Bootstrap controller and add model
4. Deploy one of the charms mentioned in the description from latest/edge
5. Check the debug logs with juju debug-log --replay

Environment

juju 3.5.1
charms channel: latest/edge

Relevant Log Output

jupyter-ui: https://pastebin.canonical.com/p/2XJmKhdKrn/
kfp-persistence: https://pastebin.canonical.com/p/5zT8qk6bBS/
kfp-profile-controller: https://pastebin.canonical.com/p/vv9MwYN9y5/

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5806.

This message was autogenerated

[NohaIhab]

Temporary fix

After discussing with the team, we agreed that in order to unblock we can modify the CI of the affected charms as follows:

1. Set the juju agent version to 3.5.0 where this bug does not exist, by setting bootstrap-options in the actions-operator step to "--agent-version 3.5.0"
2. Add a comment on the agent version change to revert this once the juju bug is released to 3.5/stable