xrdfs ls not working with POSIX system

[lucalavezzo]

Hi,

We have xrootd (v5.6.1) set up with the xrootd-multiuser plugin (2.1.2-1.osg36.el7), and are experiencing some problems with the xrdfs ls command.

While we are able to chmod, mkdir, touch etc. on xrdfs correctly, the ls for a directory that according to the POSIX permissions I should be able to view (and can view in a shell) does not seem to work; e.g. if I (lavezzo) want to cat a file that has 700 permissions I can, but if I want to ls a directory with the same permissions, I cannot:

[lavezzo@XXX]$ ll
drwx--S---  2 lavezzo lavezzo 4096 Aug  7 18:04 test
[lavezzo@XXX lavezzo]$ ll test/file.txt
-rwx------ 1 lavezzo lavezzo 13 Aug  7 18:04 test/file.txt
[lavezzo@XXX]$ xrdfs root://<redirector>/
[<redirector>:1094] / > cd /store/user/lavezzo
[<redirector>:1094] /store/user/lavezzo > ls -l
d--- 2023-08-07 22:04:49        4096 /store/user/lavezzo/test
[<redirector>:1094] /store/user/lavezzo > ls test
[<redirector>:1094] /store/user/lavezzo > ls test/file.txt
[ERROR] Server responded with an error: [3005] Unable to open directory /store/user/lavezzo/test/file.txt; not a directory

[<redirector>:1094] /store/user/lavezzo > cat test/file.txt
hello world!

In the logs, I see that multiuser is doing something, but then just hits me with permission denied:

230807 18:13:19 2363453 multiuser_UserSentry: Switching FS uid for user lavezzo
230807 18:13:19 2363453 multiuser_UserSentry: Switching FS uid for user lavezzo
230807 18:13:19 2363453 ofs_readdir: lavezzo.3189625:38@XXX Unable to read directory /store/user/lavezzo/test; permission denied

This is particularly damaging for us because by default when a directory is created through xrdfs mkdir, the permissions are set as such that other users don't have read access, and we haven't been able to successfully change this either.

Any clue whether this is an issue of xrootd-multiuser or of our setup?

[amadio]

How did you create the directory test? It seems to me that the server itself cannot read the directory or the file, so that's why you are getting permission denied. It's not that the server is denying you permission to list, but the server itself does not seem to be able to read the test directory or file.txt and gets permission denied by the OS.

[lucalavezzo]

Thank you for the quick response!

The problem happens both if I make a directory and make it readable only by me via a normal shell as well as via xrdfs.

When I make it via bash,

[lavezzo@XXX]$ mkdir testing_permissions
[lavezzo@XXX]$ echo "hello world!" > testing_permissions/file.txt
echo "hello world" > testing_permissions/file.txt
[lavezzo@XXX]$ chmod -R 700 testing_permissions/
[lavezzo@XXX]$ ll
total 24
drwx--S---  2 lavezzo lavezzo 4096 Aug  8 06:56 testing_permissions
[lavezzo@XXX]$ ll testing_permissions/
total 2
-rwx------ 1 lavezzo lavezzo 12 Aug  8 06:56 file.txt
[lavezzo@XXX]$ ls testing_permissions/
file.txt
[lavezzo@XXX]$ cat testing_permissions/file.txt
hello world
[lavezzo@XXX]$ xrdfs root://<REDIRECTOR>/
[<REDIRECTOR>:1094] / > cd /store/user/lavezzo
[<REDIRECTOR>:1094] /store/user/lavezzo > ls -l
d--- 2023-08-08 10:56:44        4096 /store/user/lavezzo/testing_permissions
[<REDIRECTOR>:1094] /store/user/lavezzo > ls testing_permissions
[<REDIRECTOR>:1094] /store/user/lavezzo > ls testing_permissions/file.txt
[ERROR] Server responded with an error: [3005] Unable to open directory /store/user/lavezzo/testing_permissions/file.txt; not a directory

[<REDIRECTOR>:1094] /store/user/lavezzo > cat testing_permissions/file.txt
hello world
[<REDIRECTOR>:1094] /store/user/lavezzo >

and via xrdfs,

[REDIRECTOR:1094] /store/user/lavezzo > mkdir testing_permissions2
[REDIRECTOR:1094] /store/user/lavezzo > ls -l
d--- 2023-08-08 10:56:44        4096 /store/user/lavezzo/testing_permissions
d--- 2023-08-08 11:00:26        4096 /store/user/lavezzo/testing_permissions2
[REDIRECTOR:1094] /store/user/lavezzo > ls testing_permissions2
[lavezzo@XXX]$ echo "hello world" > testing_permissions2/file.txt
[lavezzo@XXX]$ xrdfs root://REDIRECTOR/
[REDIRECTOR:1094] / > cd /store/user/lavezzo
[REDIRECTOR:1094] /store/user/lavezzo > ls testing_permissions2
[lavezzo@XXX]$ ls testing_permissions2
file.txt
[lavezzo@XXX]$

What would be preventing the server from reading the directory?

[amadio]

Is the server running as the same user as yourself? If permissions by the OS are like

drwx--S---  2 lavezzo lavezzo 4096 Aug  8 06:56 testing_permissions

only user lavezzo can do anything with the directory, although note that there is the S there for the group, so this may be preventing the server to access the directory. What is the output of umask for you? Can you try running the server with strace? That should show you exactly which system call failed with EPERM to give you extra clues. I don't think there is a bug in XRootD itself.

[lucalavezzo]

No the server is ran like xrootd-privileged and cmsd-privileged as a service.

As you say, the permissions should be that only I (lavezzo) can see this folder: but I would expect that the xrootd-multiuser, being able to recognize me as lavezzo, it would then give me permissions to see my own stuff. Alternatively, is there a way to force new directories created via xrdfs mkdir to have more loose permissions by default? Because at the moment if you create a directory with xrdfs mkdir, put files in it, you can't xrdfs ls, which is breaking some things for us.

umask is 0002. And at the moment we have multiuser.umask 0022 configured.

I tried running it with strace but I either did it wrong, or nothing interesting came out just things like

futex(0x613f08, FUTEX_WAKE_PRIVATE, 1)  = 1
futex(0x7ffc4ef90630, FUTEX_WAIT_PRIVATE, 0, NULL

I don't know if it's a bug in xrootd, xrootd-multiuser, or some misconfiguration on our end, but it would be great to understand that's going on and I thank you for the help in doing so!

[amadio]

In this case, it seems the problem is with the multiuser plugin, which is not part of the standard XRootD. I recommend trying to debug this issue with the developers of the plugin, which I do not know much about myself. If you have an example that can reproduce the problem with vanilla XRootD without the plugin, then we'll be happy to help with debugging the issue.

[lucalavezzo]

okay, I'll open an issue on their github then. Feel free to close this, thanks.

[amadio]

Thank you, please feel free to reopen if you see this issue without the plugin. Best regards,

[lucalavezzo]

For reference, this was an issue with xrootd-multiuser. A new version was released to address it, and it seems to be working. See opensciencegrid/xrootd-multiuser#49