mgr: remove privileged security context from mgr sidecar container

[rkachach]

Mgr sidecar was using a privileged security context because it needed
to create the /var/lib/rook directory, generate the config on it, and
then copy it to the /etc/ceph directory. This change creates empty
directories on /var/lib/rook and /etc/ceph so the mgr sidecar can use
them, removing the need for using a privileged security context.

Issue resolved by this Pull Request:
Fixes: #13719

Manual testing:

k get pod | grep mgr | awk '{print $1}' | (while read x; do echo "======= $x =====" ; k get pod $x -o=json | jq -r '.spec.containers[] | "\(.name) - \(.securityContext)"'; done)

======= rook-ceph-mgr-a-74f9fd45d4-lpdhl =====
mgr - {"privileged":false}
watch-active - {"privileged":false}
======= rook-ceph-mgr-b-6549f7bcd6-wz9r7 =====
mgr - {"privileged":false}
watch-active - {"privileged":false}

Mgr sidecar watch-active is running without issues both on the active and on the mgr standby pods:

k logs rook-ceph-mgr-a-74f9fd45d4-lpdhl -c watch-active | tail -10                                         ✔   
2024-02-12 09:52:00.872858 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:52:15.887326 I | op-mgr: Checking mgr_role label value of daemon a (prev active mgr was a)
2024-02-12 09:52:16.352018 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon a.
2024-02-12 09:52:16.352034 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:52:31.365846 I | op-mgr: Checking mgr_role label value of daemon a (prev active mgr was a)
2024-02-12 09:52:31.828295 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon a.
2024-02-12 09:52:31.828314 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:52:46.840292 I | op-mgr: Checking mgr_role label value of daemon a (prev active mgr was a)
2024-02-12 09:52:47.259273 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon a.
2024-02-12 09:52:47.259293 I | cephcmd: successfully checked mgr_role label. checking again in 15s

k logs rook-ceph-mgr-b-6549f7bcd6-wz9r7 -c watch-active | tail -10                                         ✔   
2024-02-12 09:52:32.478736 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:52:47.478856 I | op-mgr: Checking mgr_role label value of daemon b (prev active mgr was a)
2024-02-12 09:52:47.872795 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon b.
2024-02-12 09:52:47.872813 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:53:02.886029 I | op-mgr: Checking mgr_role label value of daemon b (prev active mgr was a)
2024-02-12 09:53:03.371399 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon b.
2024-02-12 09:53:03.371469 I | cephcmd: successfully checked mgr_role label. checking again in 15s
2024-02-12 09:53:18.381855 I | op-mgr: Checking mgr_role label value of daemon b (prev active mgr was a)
2024-02-12 09:53:18.789335 I | op-mgr: active mgr is still the same (a). No need to update mgr_role label on daemon b.
2024-02-12 09:53:18.789353 I | cephcmd: successfully checked mgr_role label. checking again in 15s

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[travisn]

The integration tests are passing, so if manual testing looks good, this should be good to go after a couple other small CI issues are fixed.