Xrootd stop working after update to version 4.0.0

[marcelkuri]

At the beginning of September xrootd updated to version 4.0.0.
Since then our /atlas mountpoint do not show the data servers space. This is our /etc/fstab
xrootdfs /atlas fuse rw,rdr=root://lipnode01:1094//data/xrootdfs,uid=496 0 0

With "df -h" it is showed:
Filesystem Size Used Avail Use% Mounted on
xrootdfs 1.6T 135G 1.5T 9% /atlas

We have 3 data servers with a shared partition (oss.localroot) of 1.6 Tb each.
So "df -h" would show 1.6T x 4 = 64T. But only 1.6T is showed.

The copy command with xrdcp works.

[wyang007]

can you umount /atlas and mount it again? if it still doesn't work, can you do this and let me know the output?
getfattr --only-value -n xrootdfs.fs.dataserverlist /atlas

In general, i don't recommend using xrootdfs in xrootd4 until we have release 4.0.4 or 4.1, which will fix a number of incompatibility issues.

[marcelkuri]

I umounted and mounted /atlas but the problem persist.
The output of
getfattr --only-value -n xrootdfs.fs.dataserverlist /atlas

is:
getfattr: Removing leading '/' from absolute path names
lipnode01:1094

What alternative can I use for xrootfs?

[xrootd-dev]

Hi, is lipnode01 a redirector or data server? (according to your first e-mail, it is a redirector, I just want to confirm this). What are the names of the data servers? if you do

xrdfs your_redirector:1094 locate -d /data/xrootdfs

what is the output?

Wei Yang | yangw@slac.stanford.edu | 650-926-3338(O)

On Sep 27, 2014, at 2:28 AM, marcelkuri notifications@github.com wrote:

I umounted and mounted /atlas but the problem persist.
The output of
getfattr --only-value -n xrootdfs.fs.dataserverlist /atlas

is:
getfattr: Removing leading '/' from absolute path names
lipnode01:1094

What alternative can I use for xrootfs?

—
Reply to this email directly or view it on GitHub.

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

[marcelkuri]

Yes, Mr. Wei.
lipnode01 is our redirector.
Our data servers are lipnode02, lipnode03... and so on.

The output is
xrdfs lipnode01:1094 locate -d /data/xrootdfs
[::192.168.0.1]:1094 Server ReadWrite

Marcel Kuriyama

[wyang007]

Hi Marcel,

I supposed 192.168.0.1 is lipnode01? So xrdfs can’t find your data servers. Does your data servers all have “all.export /data/xrootdfs” in the xrootd configuration file? Also, can you try

xrd lipnode01:1094 locateall /data/xrootdfs?

what is the output (xrd use an old xrootd client library, while xrdfs use the new xrootd client library).

regards,
Wei Yang | yangw@slac.stanford.edu | 650-926-3338(O)

On Sep 28, 2014, at 12:33 PM, marcelkuri notifications@github.com wrote:

Yes, Mr. Wei.
lipnode01 is our redirector.
Our data servers are lipnode02, lipnode03... and so on.

The output is
xrdfs lipnode01:1094 locate -d /data/xrootdfs
[::192.168.0.1]:1094 Server ReadWrite

Marcel Kuriyama

—
Reply to this email directly or view it on GitHub.

[marcelkuri]

Yes. 192.168.0.1 is lipnode01, the redirector.
And, all my data servers have "all.export /data/xrootdfs" in their xrootd-clustered.cfg file.

The output of "xrd lipnode01:1094 locateall /data/xrootdfs" is:

InfoType: kXrdcLocDataServer
CanWrite: true
Location: '192.168.0.1:1094

Marcel Kuriyama

This is our xrootd-clustered.cfg of all nodes:

all.export /data/xrootdfs
set xrdr=lipnode01
all.manager $(xrdr):1213
cms.allow host *

if $(xrdr) && named cns
      all.export /data/inventory
      xrd.port 1094
else if $(xrdr)
      all.role manager
      xrd.port 1095
else
      all.role server
      oss.localroot /data/files
      ofs.notify closew create mkdir mv rm rmdir trunc | /usr/bin/XrdCnsd -d -D 2 -i 90 -b $(xrdr):1094:/data/inventory
      cms.space min 20g 50g
fi

#Proof configuration
if exec xrootd
        xrd.protocol xproofd:1093 /usr/local/Root/lib/root/libXrdProofd.so
fi

### ROOTSYS
xpd.rootsys /usr/local/Root
xpd.workdir /tmp/proofbox
xpd.resource static /usr/local/Root/etc/proof/proof.conf

xpd.role worker
if lipnode01
  xpd.role master
fi

[wyang007]

Hi Marcel,

thanks for the config file. So lipnode01:1094 is a CNS server, while your cluster is lipnode01:1095. In /etc/fstab you should change to port from 1094 to 1095. A few more comments:

1. You must get this configuration template from a long time ago. CNS is no longer needed unless you have a good reason. You can comment out the following lines if you will not use CNS and inventory:

if $(xrdr) && named cns
all.export /data/inventory
xrd.port 1094

ofs.notify closew create mkdir mv rm rmdir trunc | /usr/bin/XrdCnsd -d -D 2 -i 90 -b $(xrdr):1094:/data/inventory

1. Putting xrootdfs in /etc/fstab sometimes don’t work well at system start up time because the mount command was run at a very earlier stage. the man page of xrootdfs described how to put it under autofs.

regards,
Wei Yang | yangw@slac.stanford.edu | 650-926-3338(O)

On Sep 28, 2014, at 5:21 PM, marcelkuri notifications@github.com wrote:

Yes. 192.168.0.1 is lipnode01, the redirector.
And, all my data servers have "all.export /data/xrootdfs" in their xrootd-clustered.cfg file.

The output of "xrd lipnode01:1094 locateall /data/xrootdfs" is:
InfoType: kXrdcLocDataServer
CanWrite: true
Location: '192.168.0.1:1094

Marcel Kuriyama

This is our xrootd-clustered.cfg of all nodes:

all.export /data/xrootdfs
set xrdr=lipnode01
all.manager $(xrdr):1213
cms.allow host *

if $(xrdr) && named cns
all.export /data/inventory
xrd.port 1094
else if $(xrdr)
all.role manager
xrd.port 1095
else
all.role server
oss.localroot /data/files
ofs.notify closew create mkdir mv rm rmdir trunc | /usr/bin/XrdCnsd -d -D 2 -i 90 -b $(xrdr):1094:/data/inventory
cms.space min 20g 50g
fi

#Proof configuration
if exec xrootd
xrd.protocol xproofd:1093 /usr/local/Root/lib/root/libXrdProofd.so
fi

ROOTSYS

xpd.rootsys /usr/local/Root
xpd.workdir /tmp/proofbox
xpd.resource static /usr/local/Root/etc/proof/proof.conf

xpd.role worker
if lipnode01
xpd.role master
fi

—
Reply to this email directly or view it on GitHub.

[marcelkuri]

Hi Wei,
So, as you say:
1. You must get this configuration template from a long time ago. CNS is no longer needed unless you have a good reason.

I must also edit "/etc/sysconfig/xroot" to remove any reference to CNS:

XROOTD_USER=xrootd
XROOTD_GROUP=xrootd
XROOTD_DEFAULT_OPTIONS="-l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-clustered.cfg -k 7"
#XROOTD_CNS_OPTIONS="-k 7 -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-clustered.cfg"
CMSD_DEFAULT_OPTIONS="-l /var/log/xrootd/cmsd.log -c /etc/xrootd/xrootd-clustered.cfg -k 7"
XFRD_DEFAULT_OPTIONS="-l /var/log/xrootd/xfrd.log -c /etc/xrootd/xrootd-clustered.cfg -k 7"
#XROOTD_INSTANCES="default cns"
XROOTD_INSTANCES="default"
CMSD_INSTANCES="default"
XFRD_INSTANCES="default"

After these changes everything is working:

1. /atlas is showed as a big space, not just as the space of the local disk only.
2. xrdcp is working.

Thanks a lot for you patience and attention
Best regards

Marcel Kuriyama

[abh3]

Looks like this was resolved. I am closing this ticket.