xrdfs query prepare bug?

[maksiks]

Hi Everyone,
I am using custom plugin option for xrdfs query prepare calls.
Xrootd config file snippet is:
ofs.preplib libXrdOfsPrepGPI.so -debug -admit all -run /cms/ops/prepare/prep

It looks to me that the call works fine till the return json is below 2017 (well, looks like the cut is between 2005 and 2017) characters long. Once one goes above that then the error is returned. Below is the examples of two commands, in the first case the return json is slightly shorter, in the second example we reach 2017 character long return json.

Would you please take a look if indeed my conclusion are correct? Is there anything that can be done to remedy the situation?

Thank you, --Max

xrdfs root://dtn20.nese.rc.fas.harvard.edu:1094 query prepare 111 /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary
{"request_id":"111","responses":[{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""}]}

xrdfs root://dtn20.nese.rc.fas.harvard.edu:1094 query prepare 111 /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary
No information available.

[xrootd-dev]

Hi Max, Yes, there is an limit of 2048 characters. There is no particular reason for this limit other than we used the "simple" interface to return messages. We would have to switch to a more complex interface where large messages are allowed. There is nothing you can do, this needs to be fixed in the source. Fortunately, we have a new release coming in a couple or so weeks and hopefully we can add this there. Of course, we could give you a source fix (requires compilation at your site) and you could get it faster and also tell us if it works for you. Andy

On Thu, 1 Jun 2023, maksiks wrote: Hi Everyone, I am using custom plugin option for xrdfs query prepare calls. Xrootd config file snippet is: ofs.preplib libXrdOfsPrepGPI.so -debug -admit all -run /cms/ops/prepare/prep It looks to me that the call works fine till the return json is below 2017 (well, looks like the cut is between 2005 and 2017) characters long. Once one goes above that then the error is returned. Below is the examples of two commands, in the first case the return json is slightly shorter, in the second example we reach 2017 character long return json. Would you please take a look if indeed my conclusion are correct? Is there anything that can be done to remedy the situation? Thank you, --Max xrdfs root://dtn20.nese.rc.fas.harvard.edu:1094 query prepare 111 /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary {"request_id":"111","responses":[{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"

1685643952","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"1685643953","error_text":""},{"path":"/store/temp/user/test1G.binary","path_exists":true,"on_tape":true,"online":true,"requested":true,"has_reqid":true,"req_time":"16856 43953","error_text":""}]}

…

xrdfs root://dtn20.nese.rc.fas.harvard.edu:1094 query prepare 111 /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary No information available. -- Reply to this email directly or view it on GitHub: #2023 You are receiving this because you are subscribed to this thread. Message ID: ***@***.***> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

[maksiks]

Hi Andy,
It looks like this limit causes problems for T2_US_MIT_Tape transfers on the 10% level. FTS/Rucio issue 'xrdfs query prepare' calls either for several files or for a file with a really long name (and full CMS file names can be very very long). Increasing maximum length limit will go a long way to resolve issues we are having, please please consider it for the upcoming new release.
Thank you, --Max

[abh3]

It's already on the list to be addressed as soon as possible. We recognize this is causing problems that needn't be there.

…

On Thu, 1 Jun 2023, maksiks wrote: Hi Andy, It looks like this limit causes problems for T2_US_MIT_Tape transfers on the 10% level. FTS/Rucio issue 'xrdfs query prepare' calls either for several files or for a file with a really long name (and full CMS file names can be very very long). Increasing maximum length limit will go a long way to resolve issues we are having, please please consider it for the upcoming new release. Thank you, --Max -- Reply to this email directly or view it on GitHub: #2023 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[maksiks]

Hi Everyone,
I am testing xrdfs query prepare call in xrootd version 5.6.1 and would like to ask the following question just to make sure what I see is correct: to me it looks like the length of the string the call can handle is increased but still there is a limit on the overall length; the call can't handle arbitrarily long string. It is right?
Thank you, --Max

[abh3]

Hi Max, That is correct. We increased the default allowable length to cover all the cases we could think of. You can, ofcourse, make it largeer using the -maxresp option: https://xrootd.slac.stanford.edu/doc/dev56/ofs_config.htm#_Toc136617302 In any case, the size is fixed at the set value. Allowing arbitrary lengths would be a considerable development effort. Andy

…

On Tue, 1 Aug 2023, maksiks wrote: Hi Everyone, I am testing xrdfs query prepare call in xrootd version 5.6.1 and would like to ask the following question just to make sure what I see is correct: to me it looks like the length of the string the call can handle is increased but still there is a limit on the overall length; the call can't handle arbitrarily long string. It is right? Thank you, --Max -- Reply to this email directly or view it on GitHub: #2023 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>

[maksiks]

Hi Everyone,
I just upgraded one of our doors to xrootd 5.6.2 (el8, osg-testing repo). I do not see any changes in this call from previous versions. I also do not see any effect from any of the options that are designed to work with longer file name. Please advise as I do not understand if it is me doing something wrong, or code changes did not propagate into newer versions of xrootd rpms that are distributed to us through osg.

Thank you, --Max

[abh3]

I assume you specified the -maxresp <sz> option. The patch was 564f0b2 and it appears to be in master and appeared in 5.6.0. If you did specify the option what value did you use? Where there and error messages when the plugin was initialized?

…

On Tue, 26 Sep 2023, maksiks wrote: Hi Everyone, I just upgraded one of our doors to xrootd 5.6.2 (el8, osg-testing repo). I do not see any change in this call from previous versions. I also do see any effect from any os the options that are designed to work with longer file name. Please advise as I do not understand if it is me doing something wrong, or code changes did not propagate into newer versions of xrootd. Thank you, --Max -- Reply to this email directly or view it on GitHub: #2023 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>

[maksiks]

The whole plugin line looks like this at the moment: ofs.preplib +noauth libXrdOfsPrepGPI.so -maxresp 14m -maxfiles 256 -maxquery 16 -maxreq 8 -debug -admit all -run /cms/ops/prepare/prep I did try different combination of options and different -maxresp options, however it does not increase the length of the string that can be passed in. Once you are above that limit the call just simply returns "No information available.?" like it used to. ? Thank you, --Max

________________________________ From: Andrew Hanushevsky ***@***.***> Sent: Tuesday, September 26, 2023 6:23 PM To: xrootd/xrootd Cc: Maxim Goncharov; Author Subject: Re: [xrootd/xrootd] xrdfs query prepare bug? (Issue #2023) I assume you specified the -maxresp <sz> option. The patch was 564f0b2 and it appears to be in master and appeared in 5.6.0. If you did specify the option what value did you use? Where there and error messages when the plugin was initialized?

On Tue, 26 Sep 2023, maksiks wrote: Hi Everyone, I just upgraded one of our doors to xrootd 5.6.2 (el8, osg-testing repo). I do not see any change in this call from previous versions. I also do see any effect from any os the options that are designed to work with longer file name. Please advise as I do not understand if it is me doing something wrong, or code changes did not propagate into newer versions of xrootd. Thank you, --Max -- Reply to this email directly or view it on GitHub: #2023 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>

- Reply to this email directly, view it on GitHub<#2023 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUWS7SC7QHETZOQIQ4IWV3X4NIVJANCNFSM6AAAAAAYXK3OSU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[abh3]

Hi, I looked at he code once again and it seems correct. However, I noticed that you used the word "door" is this dCache or a regular xroot server? Also, we've had this happen before that a user upgraded but the config file still pointed to the old plugin. So, could you verify that the ofs.preplib is actually pointing to the 5.6.2 version and not via an absolute path to an older version.

[maksiks]

Hi Andrew, This is a regular xrootd server installed on Linux8 OS. The command I execute I paste below, you can see response I get. If you remove a couple of /store/... entries you would get expected json back. Also, I list below xrootd rpm packages installed on that node, and a ls -l for the library in question. As far as I can see the library was updated a couple of weeks ago as one would expectand there are no other ones the system might be using underneath, do you agree? xrdfs root://dtn17.nese.rc.fas.harvard.edu:1094 query prepare 111 /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary /store/temp/user/test1G.binary No information available. sudo rpm -qa | grep xrootd | sort xrootd-5.6.2-2.2.osg36.el8.x86_64 xrootd-client-5.6.2-2.2.osg36.el8.x86_64 xrootd-client-compat-5.6.2-2.2.osg36.el8.x86_64 xrootd-client-libs-5.6.2-2.2.osg36.el8.x86_64 xrootd-cmstfc-1.5.2-6.osg36.el8.x86_64 xrootd-lcmaps-1.7.8-3.osgup.el8.x86_64 xrootd-libs-5.6.2-2.2.osg36.el8.x86_64 xrootd-multiuser-2.1.2-1.osg36.el8.x86_64 xrootd-scitokens-5.6.2-2.2.osg36.el8.x86_64 xrootd-selinux-5.6.2-2.2.osg36.el8.noarch xrootd-server-5.6.2-2.2.osg36.el8.x86_64 xrootd-server-libs-5.6.2-2.2.osg36.el8.x86_64 xrootd-voms-5.6.2-2.2.osg36.el8.x86_64 sudo ls -l /usr/lib64/libXrdOfsPrepGPI-5.so -rwxr-xr-x 1 root root 36736 Sep 19 10:21 /usr/lib64/libXrdOfsPrepGPI-5.so sudo locate libXrdOfsPrepGPI /usr/lib64/libXrdOfsPrepGPI-5.so /var/lib/containers/storage/overlay/520583f38f09ba169d1efd9a75f6ac0eeb9b0f6dc5798cfc1851c001553a4ee2/merged/usr/lib64/libXrdOfsPrepGPI-5.so /var/lib/containers/storage/overlay/df3d17560aed71561eb73f6cd62b3c76aa5885be040133ee1ae105dfea52edb2/diff/usr/lib64/libXrdOfsPrepGPI-5.so Thank you, --Max

…

________________________________ From: Andrew Hanushevsky ***@***.***> Sent: Tuesday, October 3, 2023 8:30 AM To: xrootd/xrootd ***@***.***> Cc: Maxim Goncharov ***@***.***>; Author ***@***.***> Subject: Re: [xrootd/xrootd] xrdfs query prepare bug? (Issue #2023) Hi, I looked at he code once again and it seems correct. However, I noticed that you used the word "door" is this dCache or a regular xroot server? Also, we've had this happen before that a user upgraded but the config file still pointed to the old plugin. So, could you verify that the ofs.preplib is actually pointing to the 5.6.2 version and not via an absolute path to an older version. — Reply to this email directly, view it on GitHub<#2023 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUWS7UPDQVIETS5SWKSCCLX5QAPHAVCNFSM6AAAAAAYXK3OSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBUHA3TAMBSGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[maksiks]

I am sorry, bu is there any update on this issue?
Thank you, --Max