PVC longhorn has permission issues

[gdha]

Instead of using a local path (which is lost at every restart) we wanted to use longhorn to have some more persistent storage.
It fires up fine, but it does a crashloop and restarts and so on. The error is:

$ kubectl logs -n uptime-kuma uptime-kuma-0 -f
Welcome to Uptime Kuma
Your Node.js version: 20
Warning: Uptime Kuma is currently not stable on Node.js >= 20, please use Node.js 18.
2023-10-05T12:46:48Z [SERVER] INFO: Welcome to Uptime Kuma
2023-10-05T12:46:48Z [SERVER] INFO: Node Env: production
2023-10-05T12:46:48Z [SERVER] INFO: Importing Node libraries
2023-10-05T12:46:48Z [SERVER] INFO: Importing 3rd-party libraries
2023-10-05T12:46:53Z [SERVER] INFO: Creating express and socket.io instance
2023-10-05T12:46:53Z [SERVER] INFO: Server Type: HTTP
2023-10-05T12:46:53Z [SERVER] INFO: Importing this project modules
2023-10-05T12:46:54Z [NOTIFICATION] INFO: Prepare Notification Providers
2023-10-05T12:46:54Z [SERVER] INFO: Version: 1.22.1
Trace: Error: EACCES: permission denied, mkdir './data/upload/'
    at Object.mkdirSync (node:fs:1414:3)
    at Database.init (/app/server/database.js:99:16)
    at /app/server/server.js:176:14
    at Object.<anonymous> (/app/server/server.js:1586:3)
    at Module._compile (node:internal/modules/cjs/loader:1233:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: './data/upload/'
}
    at process.<anonymous> (/app/server/server.js:1825:13)
    at process.emit (node:events:514:28)
    at emit (node:internal/process/promises:150:20)
    at processPromiseRejections (node:internal/process/promises:284:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
2023-10-05T12:46:54Z [] INFO: Cannot write to error.log

This might be related - louislam/uptime-kuma#857 ?

[gdha]

Checked out longhorn/longhorn#918 and lsblk -f showed me the volume (the real number I got from the longhorn GUI):

root@n5:/mnt# mount /dev/sdg /mnt
root@n5:/mnt# ll
total 24
drwxr-xr-x  3 root root  4096 Oct  5 14:44 ./
drwxr-xr-x 21 root root  4096 Oct  4 17:56 ../
drwx------  2 root root 16384 Oct  5 14:44 lost+found/

As we see it is owned by root and the pod is using "uptime-kuma" as owner/group names.
I think we can only fix this via a sidecar container?

[gdha]

Simly did the following:

root@n5:/mnt# mount /dev/sdg /mnt
root@n5:/mnt# chown 3310:3310 /mnt

killed the pod and it restarted fine now. Proof that this storage is persistent.
To see what happened when the pod is running just mount it again:

root@n5:/# mount /dev/sdg /mnt
root@n5:/# ll /mnt
total 1044
drwxr-xr-x  4 3310 3310   4096 Oct  5 15:15 ./
drwxr-xr-x 21 root root   4096 Oct  4 17:56 ../
-rwxrwxr-x  1 3310 3310  61440 Oct  5 15:15 kuma.db*
-rwxrwxr-x  1 3310 3310  32768 Oct  5 15:22 kuma.db-shm*
-rwxrwxr-x  1 3310 3310 943512 Oct  5 15:22 kuma.db-wal*
drwx------  2 root root  16384 Oct  5 14:44 lost+found/
drwxr-xr-x  2 3310 3310   4096 Oct  5 15:15 upload/

[gdha]

However, still like to know how to fix it with a k8s sidecar...
Useful information - https://medium.com/bb-tutorials-and-thoughts/kubernetes-learn-sidecar-container-pattern-6d8c21f873d
After some investigation the solution is to use an initcontainer - see https://loft.sh/blog/kubernetes-init-containers/
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container