Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Directory listing failures (bad permission reporting) #1456

Closed
juztas opened this issue May 13, 2021 · 6 comments
Closed

Directory listing failures (bad permission reporting) #1456

juztas opened this issue May 13, 2021 · 6 comments
Assignees
@simonmichal
Labels
bug (non-critical)

Comments

@juztas
Copy link
Contributor

juztas commented May 13, 2021

Hello,
I have a weird issue with xrootd 5.2 + multiuser plugin. Here are some details below:
Setup is simple 1 redirector and 4 data servers behind.
listing directory fails using root protocol via redirector:

$ gfal-ls root://xrootd-redir.ultralight.org/store/user/jbalcas/
gfal-ls error: 115 (Operation now in progress) - Failed reading directory: [ERROR] Error response: No such file or directory (Unknown error 400)

But it works listing specific file via redirector:

-bash-4.2$  gfal-ls -l root://xrootd-redir.ultralight.org//store/user/jbalcas/1GB_File
-r--------   1 3000  3000  1048576000 May 19  2020 root://xrootd-redir.ultralight.org//store/user/jbalcas/1GB_File

Directory listing works with davs protocol via redirector:

-bash-4.2$ gfal-ls -l davs://xrootd-redir.ultralight.org:1094/store/user/jbalcas/
drwxrwxrwx   0 0     0             5 Mar 18 18:09 core-dumps
-rwxrwxrwx   0 0     0         69465 May  1  2020 18Jan2019_job0_outHist.root
-rwxrwxrwx   0 0     0     1048576000 May 19  2020 1GB_File
-rwxrwxrwx   0 0     0         69465 May  1  2020 18Jan2019_job0_outHist.root1
drwxrwxrwx   0 0     0             2 Feb 22 07:15 for-xrootd-devs
drwxrwxrwx   0 0     0             0 May  4 13:11 test

Also, directory listing works using xrdfs via redirector:

-bash-4.2$ xrdfs xrootd-redir.ultralight.org
[xrootd-redir.ultralight.org:1094] / > ls -l /store/user/jbalcas/
-r-- 2020-05-01 20:17:28       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root
-r-- 2020-05-01 20:19:49       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root1
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas/1GB_File
dr-x 2021-03-19 01:09:34           5 /store/user/jbalcas/core-dumps
dr-x 2021-02-22 15:15:31           2 /store/user/jbalcas/for-xrootd-devs
dr-x 2021-05-04 20:11:59           0 /store/user/jbalcas/test
[xrootd-redir.ultralight.org:1094] / > stat /store/user/jbalcas/1GB_File
Path:   /store/user/jbalcas/1GB_File
Id:     983852453462060
Size:   1048576000
MTime:  2020-05-19 22:56:21
Flags:  16 (IsReadable)

Any directory or file listing via any of the 4 data servers works (root, davs, xrdfs). So something between redirector and data servers and especially using root protocol. With an increased debug, here is redirector log:

INFO in AuthzKey: Returning '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=jbalcas/CN=751133/CN=Justas Balcas::cms:/cms,/cms/becms,/cms/dcms,/cms/escms,/cms/itcms,/cms/uscms,::' of length 145 as key.
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdResponse: 0000 sending OK
jbalcas.3955907:51@login-1.hep.caltech.edu Protocol 'gsi'
jbalcas.3955907:51@login-1.hep.caltech.edu Name 'jbalcas'
jbalcas.3955907:51@login-1.hep.caltech.edu Host 'login-1.hep.caltech.edu'
jbalcas.3955907:51@login-1.hep.caltech.edu Vorg 'cms'
jbalcas.3955907:51@login-1.hep.caltech.edu Role ''
jbalcas.3955907:51@login-1.hep.caltech.edu Grps '/cms /cms/becms /cms/dcms /cms/escms /cms/itcms /cms/uscms'
jbalcas.3955907:51@login-1.hep.caltech.edu Caps ''
jbalcas.3955907:51@login-1.hep.caltech.edu Pidn 'jbalcas.3955907:51@login-1.hep.caltech.edu'
jbalcas.3955907:51@login-1.hep.caltech.edu Crlen 9266
jbalcas.3955907:51@login-1.hep.caltech.edu ueid  23
jbalcas.3955907:51@login-1.hep.caltech.edu uid   0
jbalcas.3955907:51@login-1.hep.caltech.edu gid   0
210513 15:05:34 28510 XrootdMonitor: 367 bytes sent to 169.228.130.91:9930 rc=0
210513 15:05:34 28510 XrootdMonitor: 367 bytes sent to xrootd-mon.unl.edu:9930 rc=0
210513 15:05:34 28510 XrootdXeq: jbalcas.3955907:51@login-1.hep.caltech.edu pub IPv4 login as jbalcas
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 req=stat dlen=49
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu ofs_stat:  fn=/store/
210513 15:05:34 28500 Receive localhost 30 bytes on 41983
210513 15:05:34 28500 Decode xrootd-redir redirects jbalcas.3955907:51@login-1.hep.caltech.edu to transfer-8.ultralight.org:1094 /store/
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 rc=-256 stat /store/
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 redirecting to transfer-8.ultralight.org:1094
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdResponse: 0100 sending 29 data bytes; status=4004
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 req=stat dlen=49
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu ofs_stat:  fn=/store/
210513 15:05:34 28500 Receive localhost 31 bytes on 43007
210513 15:05:34 28500 Decode xrootd-redir redirects jbalcas.3955907:51@login-1.hep.caltech.edu to transfer-10.ultralight.org:1094 /store/
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 rc=-256 stat /store/
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 redirecting to transfer-10.ultralight.org:1094
210513 15:05:34 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdResponse: 0100 sending 30 data bytes; status=4004
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdProtocol: 0100 req=dirlist dlen=8
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu ofs_opendir:  fn=/store/
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu oss_Opendir: lcl path /storage/cms/store/ (/store/)
210513 15:05:35 28510 ofs_opendir: jbalcas.3955907:51@login-1.hep.caltech.edu Unable to open directory /store/; no such file or directory
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrootdResponse: 0100 sending err 3011: Unable to open directory /store/; no such file or directory
210513 15:05:35 28510 XrootdXeq: jbalcas.3955907:51@login-1.hep.caltech.edu disc 0:00:01
210513 15:05:35 28510 XrootdMonitor: 56 bytes sent to 169.228.130.91:9930 rc=0
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrdPoll: Poller 2 removing FD 51
210513 15:05:35 28510 jbalcas.3955907:51@login-1.hep.caltech.edu XrdPoll: FD 51 detached from poller 2; num=3

From data server:

INFO in AuthzKey: Returning '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=jbalcas/CN=751133/CN=Justas Balcas::cms:/cms,/cms/becms,/cms/dcms,/cms/escms,/cms/itcms,/cms/uscms,::' of length 145 as key.
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrootdResponse: 0000 sending OK
jbalcas.3955907:41@login-1.hep.caltech.edu Protocol 'gsi'
jbalcas.3955907:41@login-1.hep.caltech.edu Name 'jbalcas'
jbalcas.3955907:41@login-1.hep.caltech.edu Host 'login-1.hep.caltech.edu'
jbalcas.3955907:41@login-1.hep.caltech.edu Vorg 'cms'
jbalcas.3955907:41@login-1.hep.caltech.edu Role ''
jbalcas.3955907:41@login-1.hep.caltech.edu Grps '/cms /cms/becms /cms/dcms /cms/escms /cms/itcms /cms/uscms'
jbalcas.3955907:41@login-1.hep.caltech.edu Caps ''
jbalcas.3955907:41@login-1.hep.caltech.edu Pidn 'jbalcas.3955907:41@login-1.hep.caltech.edu'
jbalcas.3955907:41@login-1.hep.caltech.edu Crlen 9266
jbalcas.3955907:41@login-1.hep.caltech.edu ueid  3
jbalcas.3955907:41@login-1.hep.caltech.edu uid   0
jbalcas.3955907:41@login-1.hep.caltech.edu gid   0
210513 08:05:35 372882 XrootdMonitor: 367 bytes sent to 169.228.130.91:9930 rc=0
210513 08:05:35 372882 XrootdMonitor: 367 bytes sent to xrootd-mon.unl.edu:9930 rc=0
210513 08:05:35 372882 XrootdXeq: jbalcas.3955907:41@login-1.hep.caltech.edu pub IPv4 login as jbalcas
210513 08:05:35 372882 multiuser_UserSentry: Switching FS uid for user jbalcas
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrootdProtocol: 0100 req=stat dlen=49
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu ofs_stat:  fn=/store/
210513 08:05:35 372882 multiuser_UserSentry: Switching FS uid for user jbalcas
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrootdProtocol: 0100 rc=0 stat /store/
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrootdResponse: 0100 sending 76 data bytes
210513 08:05:35 372882 XrootdXeq: jbalcas.3955907:41@login-1.hep.caltech.edu disc 0:00:01
210513 08:05:35 372882 multiuser_UserSentry: Switching FS uid for user jbalcas
210513 08:05:35 372882 XrootdMonitor: 56 bytes sent to 169.228.130.91:9930 rc=0
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrdPoll: Poller 0 removing FD 41
210513 08:05:35 372882 jbalcas.3955907:41@login-1.hep.caltech.edu XrdPoll: FD 41 detached from poller 0; num=0

And problematic detail seems to be here on redirector:

210512 20:44:56 25992 jbalcas.3134036:46@login-1.hep.caltech.edu oss_Opendir: lcl path /storage/cms/store/ (/store/)

Whenever I do directory listing using root protocol, xrootd redirector does lcl path mapping and asks dataserver /store and not /storage/cms/store/. In the case of davs protocol - lcl is done not on the redirector side but done on an individual server.
In our xrootd redir config we have cms.dfs lookup distrib mdhold 20m redirect immed - so that all lookups are redirected to data servers. Issues so far:

  1. What can be done to allow directory listing via redirector? Is this a bug or just a config issue? Let me know if you need a full redirector and data server configs.
  2. Directory listing permissions. Using root/davs protocol permissions and owner uid/gid reported back to the client are wrong.
-bash-4.2$ gfal-ls -l davs://xrootd-redir.ultralight.org:1094/store/user/jbalcas/1GB_File
-rwxrwxrwx   0 0     0     1048576000 May 19  2020 davs://xrootd-redir.ultralight.org:1094/store/user/jbalcas/1GB_File
-bash-4.2$ gfal-ls -l davs://transfer-10.ultralight.org:1094/store/user/jbalcas/1GB_File
-rwxrwxrwx   0 0     0     1048576000 May 19  2020 davs://transfer-10.ultralight.org:1094/store/user/jbalcas/1GB_File
-bash-4.2$ gfal-ls -l root://xrootd-redir.ultralight.org:1094/store/user/jbalcas/1GB_File
-r--------   1 3000  3000  1048576000 May 19  2020 root://xrootd-redir.ultralight.org:1094/store/user/jbalcas/1GB_File
-bash-4.2$ gfal-ls -l root://transfer-10.ultralight.org:1094/store/user/jbalcas/1GB_File
-r--------   1 3000  3000  1048576000 May 19  2020 root://transfer-10.ultralight.org:1094/store/user/jbalcas/1GB_File

while correct permissions are:

[root@transfer-10 ~]# ll /storage/cms/store/user/jbalcas/1GB_File
-rw-rw-r-- 1 root jbalcas 1048576000 May 19  2020 /storage/cms/store/user/jbalcas/1GB_File
[root@transfer-10 ~]# stat /storage/cms/store/user/jbalcas/1GB_File
  File: ‘/storage/cms/store/user/jbalcas/1GB_File’
  Size: 1048576000	Blocks: 2048000    IO Block: 4194304 regular file
Device: 2ch/44d	Inode: 1099511856847  Links: 1
Access: (0664/-rw-rw-r--)  Uid: (    0/    root)   Gid: ( 3000/ jbalcas)
Access: 2021-05-10 15:40:00.427772762 -0700
Modify: 2020-05-19 15:56:21.000000000 -0700
Change: 2021-05-10 15:40:00.428772803 -0700
 Birth: -
@abh3
Copy link
Member

abh3 commented May 13, 2021 via email

@juztas
Copy link
Contributor Author

juztas commented May 13, 2021

Using redirector (multiuser + cephfs):

[xrootd-redir.ultralight.org:1094] / > ls -l /store/user/jbalcas/
-r-- 2020-05-01 20:17:28       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root
-r-- 2020-05-01 20:19:49       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root1
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas/1GB_File
dr-x 2021-03-19 01:09:34           5 /store/user/jbalcas/core-dumps
dr-x 2021-02-22 15:15:31           2 /store/user/jbalcas/for-xrootd-devs
dr-x 2021-05-04 20:11:59           0 /store/user/jbalcas/test
[xrootd-redir.ultralight.org:1094] / > ls -l /store/user/jbalcas/1GB_File
[ERROR] Server responded with an error: [3005] Unable to open directory /store/user/jbalcas/1GB_File; not a directory

[xrootd-redir.ultralight.org:1094] / > stat /store/user/jbalcas/1GB_File
Path:   /store/user/jbalcas/1GB_File
Id:     983852453462060
Size:   1048576000
MTime:  2020-05-19 22:56:21
Flags:  16 (IsReadable)
[xrootd-redir.ultralight.org:1094] / >

Using data server (multiuser + cephfs):

-bash-4.2$ xrdfs transfer-10.ultralight.org
[transfer-10.ultralight.org:1094] / > ls -l /store/user/jbalcas/
-r-- 2020-05-01 20:17:28       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root
-r-- 2020-05-01 20:19:49       69465 /store/user/jbalcas/18Jan2019_job0_outHist.root1
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas/1GB_File
dr-x 2021-03-19 01:09:34           5 /store/user/jbalcas/core-dumps
dr-x 2021-02-22 15:15:31           2 /store/user/jbalcas/for-xrootd-devs
dr-x 2021-05-04 20:11:59           0 /store/user/jbalcas/test
[transfer-10.ultralight.org:1094] / > ls -l /store/user/jbalcas/1GB_File
[ERROR] Server responded with an error: [3005] Unable to open directory /store/user/jbalcas/1GB_File; not a directory

[transfer-10.ultralight.org:1094] / > stat /store/user/jbalcas/1GB_File
Path:   /store/user/jbalcas/1GB_File
Id:     983852453462060
Size:   1048576000
MTime:  2020-05-19 22:56:21
Flags:  16 (IsReadable)

hdfs redirector (5.2 release) backend with xrootd-hdfs plugins (it does not use multiuser):

-bash-4.2$ xrdfs xrootd.ultralight.org
[xrootd.ultralight.org:1094] / > ls /store/user/jbalcas/
/store/user/jbalcas//18Jan2019_job0_outHist.root
/store/user/jbalcas//18Jan2019_job0_outHist.root1
/store/user/jbalcas//1GB_File
/store/user/jbalcas//core-dumps
/store/user/jbalcas//for-xrootd-devs
/store/user/jbalcas//test
[xrootd.ultralight.org:1094] / > ls -l /store/user/jbalcas/
-r-- 2020-05-01 20:17:28       69465 /store/user/jbalcas//18Jan2019_job0_outHist.root
-r-- 2020-05-01 20:19:49       69465 /store/user/jbalcas//18Jan2019_job0_outHist.root1
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas//1GB_File
dr-x 2021-03-19 01:09:34        4096 /store/user/jbalcas//core-dumps
dr-x 2021-02-22 15:15:31        4096 /store/user/jbalcas//for-xrootd-devs
dr-x 2021-05-04 20:11:59        4096 /store/user/jbalcas//test
[xrootd.ultralight.org:1094] / > ls -l /store/user/jbalcas/1GB_File
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas/1GB_File//1GB_File
[xrootd.ultralight.org:1094] / > stat /store/user/jbalcas/1GB_File
Path:   /store/user/jbalcas/1GB_File
Id:     0
Size:   1048576000
MTime:  2020-05-19 22:56:21
Flags:  24 (Offline|IsReadable)

And data server with hdfs plugin (it does not use multiuser):

[transfer-2.ultralight.org:1094] / > ls -l /store/user/jbalcas/
-r-- 2020-05-01 20:17:28       69465 /store/user/jbalcas//18Jan2019_job0_outHist.root
-r-- 2020-05-01 20:19:49       69465 /store/user/jbalcas//18Jan2019_job0_outHist.root1
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas//1GB_File
dr-x 2021-03-19 01:09:34        4096 /store/user/jbalcas//core-dumps
dr-x 2021-02-22 15:15:31        4096 /store/user/jbalcas//for-xrootd-devs
dr-x 2021-05-04 20:11:59        4096 /store/user/jbalcas//test
[transfer-2.ultralight.org:1094] / > ls -l /store/user/jbalcas/1GB_File
-r-- 2020-05-19 22:56:21  1048576000 /store/user/jbalcas/1GB_File//1GB_File
[transfer-2.ultralight.org:1094] / > stat /store/user/jbalcas/1GB_File
Path:   /store/user/jbalcas/1GB_File
Id:     0
Size:   1048576000
MTime:  2020-05-19 22:56:21
Flags:  24 (Offline|IsReadable)

So xrdfs fails to do ls -l for particular file, but ok to list directory (xrootd+multiuser). For HDFS based endpoint (without multiuser) - everything works.

@abh3
Copy link
Member

abh3 commented May 13, 2021

Sure looks like a bug to me. I'll push this.

@abh3
Copy link
Member

abh3 commented May 13, 2021

So, this ticket contains three different problems:

  1. Can't list directory which appears to be a client problem,
  2. Wrong permissions and uid/gid being returned which is a server problem, and
  3. File mapping is inconsistent on the redirector.

In order to track this properly, could you split this into separate tickets and the solution for each is a separate task. As for the wrong permissions, what version is each server in your DFS config running? This may identify where is discrepancy is coming from.

As for the mapping, I will look at that but I don't see why that should be happening. So, yes, the config file would be helpful here.

@juztas
Copy link
Contributor Author

juztas commented May 13, 2021

Ok, Will create separate issues now.
Redirector:

[root@xrootd-redir ~]# yum list installed | grep xrootd
gfal2-plugin-xrootd.x86_64                 2.18.1-1.1.osg35up.el7      @osg-upcoming-development
xrootd.x86_64                              1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-client.x86_64                       1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-client-libs.x86_64                  1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-cmstfc.x86_64                       1.5.2-6.osg35.el7           @osg-contrib
xrootd-lcmaps.x86_64                       1.7.8-3.osgup.el7           @osg-upcoming-development
xrootd-libs.x86_64                         1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-multiuser.x86_64                    1.0.0-1.osg35up.el7         @osg-upcoming-development
xrootd-scitokens.x86_64                    1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-selinux.noarch                      1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-server.x86_64                       1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd-server-libs.x86_64                  1:5.2.0-0.rc1.1.osg35up.el7 @osg-upcoming-development
xrootd4-libs.x86_64                        4.12.6-1.el7                @epel

DataServer:

[root@transfer-10 ~]# yum list installed | grep xrootd
gfal2-plugin-xrootd.x86_64           2.18.1-1.1.osg35.el7           @osg
gratia-probe-xrootd-storage.x86_64   1.23.2-1.osg35.el7             @osg
gratia-probe-xrootd-transfer.x86_64  1.23.2-1.osg35.el7             @osg
xrootd.x86_64                        1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-client.x86_64                 1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-client-libs.x86_64            1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-cmstfc.x86_64                 1.5.2-6.osg35.el7              @osg-contrib
xrootd-lcmaps.x86_64                 1.7.8-3.osgup.el7              @osg-upcoming-testing
xrootd-libs.x86_64                   1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-multiuser.x86_64              1.0.0-0.2.rc.2.osg35up.el7     @osg-upcoming-minefield
xrootd-scitokens.x86_64              1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-selinux.noarch                1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-server.x86_64                 1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd-server-libs.x86_64            1:5.2.0-0.rc1.1.osg35up.el7    @osg-upcoming-development
xrootd4-client-libs.x86_64           4.12.6-1.el7                   @epel
xrootd4-libs.x86_64                  4.12.6-1.el7                   @epel

Redirector (DataServer) same config:

[root@xrootd-redir ~]# cat /etc/xrootd/config.d/99-local.cfg
# ===================================================================
# WARNING! THIS FILE WAS AUTOMATICALLY PREPARED BY PUPPET
# ANY MODIFICATIONS TO THIS FILE WILL BE OVERWRITTEN ON PUPPET RUN
# IN CASE YOU WANT TO DO MODIFICATIONS, MAKE SURE YOU DISABLE PUPPET!
# ===================================================================
# XrootD Security
# ---------------------------------------
xrootd.seclib /usr/lib64/libXrdSec.so
sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrootd/xrootdcert.pem -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzto:900 -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0
acc.authdb /etc/xrootd/auth_file
ofs.authorize
macaroons.secretkey /etc/xrootd/macaroon-secret
ofs.authlib ++ libXrdMacaroons.so
ofs.authlib ++ libXrdAccSciTokens.so
# --------------------------------------
# XrootD Monitoring
# --------------------------------------
# Monitoring for AAA Dashboard :
xrd.report 169.228.130.91:9931 every 30s all sync
xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest files io info user 169.228.130.91:9930 dest fstat info user xrd-mon.osgstorage.org:9930
all.sitename T2_US_Caltech
# -------------------------------------
# Configure redirector/server
# -------------------------------------
set xrdr = xrootd-redir.ultralight.org
xrd.port 1094
all.manager $(xrdr):1213
if $(xrdr)
  # It's role is manager
  all.role manager
  # The known managers (fnal redirector)
  all.manager meta all cmsxrootd.fnal.gov+ 1213
  # Redirect all lookup calls to original data servers. Redirector does not have visibility of FS
  cms.dfs lookup distrib mdhold 20m redirect immed
else
  # Role is server
  all.role server
  # The known managers (local redirector)
  all.manager meta xrootd-redir.ultralight.org:1213
fi
# -------------------------------------
# Allow only specific path, checksum config
# -------------------------------------
# Allow any path to be exported; this is further refined in the authfile.
all.export /

# Hosts allowed to use this xrootd cluster
cms.allow host *

xrootd.chksum max 10 adler32

# Disable async. Related issue: https://github.com/xrootd/xrootd/issues/1113
xrootd.async off
# -------------------------------------
# Integrate with CMS TFC, placed in /etc/storage.xml
# It will see files under /store/...
# -------------------------------------
oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage.xml?protocol=ceph
# -------------------------------------
# Configure davs/https for TPC
# -------------------------------------
# Enable https over XrootD
if exec xrootd
  xrd.protocol http:1094 /usr/lib64/libXrdHttp.so
  http.cadir /etc/grid-security/certificates
  http.cert /etc/grid-security/xrootd/xrootdcert.pem
  http.key /etc/grid-security/xrootd/xrootdkey.pem
  http.secxtractor /usr/lib64/libXrdLcmaps.so
  http.secretkey NNNNNNNNNN
  # Enable third-party-copy
  http.exthandler xrdtpc libXrdHttpTPC.so
  # Pass the bearer token to the Xrootd authorization framework.
  http.header2cgi Authorization authz
  http.listingdeny yes
  http.desthttps yes
  http.selfhttps2http no
  http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
  http.exthandler xrdmacaroons libXrdMacaroons.so
fi

storage.xml:

[root@xrootd-redir ~]# cat /etc/xrootd/storage.xml
<storage-mapping>
   <!-- XRootD Server Ceph hack -->
   <lfn-to-pfn protocol="ceph" destination-match=".*" path-match="/+(store/.*)" result="/storage/cms/$1"/>
</storage-mapping>

This was referenced May 13, 2021
Closed
Open
Closed
@juztas
Copy link
Contributor Author

juztas commented May 13, 2021

3 separate tickets created. Closing this one

@juztas juztas closed this as completed May 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@simonmichal simonmichal

Labels
bug (non-critical)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@juztas @abh3 @simonmichal