[FEATURE] add volume attribute for filesystem owner

[dmayle]

Is your feature request related to a problem? Please describe.
When provisioning volumes (dynamically or otherwise), they are unusable in non-root containers because the default ownership is root:root 755. This means that volumes used this way require additional configuration steps in order to be usable by non-root containers

Describe the solution you'd like
I'd like to see a CSI volume attribute added that can be used to set the initial ownership and permissions, something like 'initialOwner' and 'initialPermissions'.

Describe alternatives you've considered
I have a workaround where I use an init container to check if the volume is empty, and then set the permissions.

Additional context
The initial format happens in https://github.com/longhorn/longhorn-manager/blob/master/csi/node_server.go#L81 , where k8s.io/util/mount is used to transparently format the volume on first mount. Since this package neither supports this feature, nor reports back on whether or not it had to format the volume, the package would either have to be changed, or an alternative implementation would be required.

The source that matters in k8s.io/util/mount is:

• https://github.com/kubernetes/utils/blob/master/mount/mount.go#L144
• https://github.com/kubernetes/utils/blob/master/mount/mount_linux.go#L318
• https://github.com/kubernetes/utils/blob/master/mount/mount_windows.go#L217

[Rancheroo]

I have this issue also when running containers using a non-root limited access user.
It cannot access the mount with Permission denied.

I have tried setting the Security Context however that did not get me far.
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

The above post is very useful and I will give the workaround a try. However this would be a useful feature as root shouldn't be the default user in a container and I would expect many people to hit this once Longhorn is released and used widely in Enterprise.

[virus2016]

This might be the issue with my Mongodb instances. It's randomly after we take a snapshot or backup where the pod crashes and can not start backup because a file is locked and the user trying to access it can not remove the lock. As soon as I delete the pod, mongo starts up perfectly.

Does anyone else have this problem?

[yasker]

@virus2016 I am not sure you're experiencing the same issue here. Can you file a bug and provide the reproduce steps? If you can send us a support bundle (see the footer of UI) that would be even better.

[parsley42]

I'm definitely experiencing the same issue. For other folks who hit this, the fix for me was just setting fsGroup: x in the podSecurityContext (for me, x=1 -> "daemon"). When this is set, the kubelet will recursively update group perms on the mount.

[virus2016]

@virus2016 I am not sure you're experiencing the same issue here. Can you file a bug and provide the reproduce steps? If you can send us a support bundle (see the footer of UI) that would be even better.

Yes I will do. After a couple of weeks looking into this, it is definitely an issue with permissions specifically longhorn. I have run 3 deployments of mongodb replica as:

• no volumes attached
• digitalocean volumes attached
• longhorn volumes attached

I am using the rancher mongodb replica template in apps. All run perfect apart from longhorn.

At a random time, one mongodb pod with longhorn attached storage gets into a crash loop with the following:

2020-10-25T10:56:46.490+0000 I STORAGE  [main] Engine custom option: cache_size=600M
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=appdrift-mongodb-replicaset-2
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] db version v3.6.14
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] git version: cbef87692475857c7ee6e764c8f5104b39c342a1
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] allocator: tcmalloc
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] modules: none
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] build environment:
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten]     distarch: x86_64
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten]     target_arch: x86_64
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] options: { config: "/data/configdb/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { replSet: "appdriftlive" }, security: { authorization: "enabled", keyFile: "/data/configdb/key.txt" }, storage: { dbPath: "/data/db", wiredTiger: { engineConfig: { configString: "cache_size=600M" } } } }
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I STORAGE  [initandlisten] exception in initAndListen: DBPathInUse: Unable to create/open the lock file: /data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /data/db directory, terminating
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] now exiting
25/10/2020 10:56:46 2020-10-25T10:56:46.513+0000 I CONTROL  [initandlisten] shutting down with code:100

Soon after the other pods follow.

I have create the issue #1909.

Thanks for your help so far - I love Longhorn!

[samip5]

@dmayle Could you please share your workaround?

Longhorn definitely is not respecting podSecurityContext, even with 1.2.0 version of Longhorn.

[yasker]

@samip5 v1.2.0 has a bug in the security context. Can you check #2964 for a workaround?

I will close this issue since it's one year old. Feel free to file new issues if #2964 is not the issue you're experiencing.