FTS error TRANSFER [5] DESTINATION MAKE_PARENT HTTP 500 : Unexpected server error: 500 with url davs://cmsio5.rc.ufl.edu...

[bockjoo]

Hi,
In the xrootd 5.2.0 servers with the Lustre backend storage, I am seeing this error:
TRANSFER [5] DESTINATION MAKE_PARENT HTTP 500 : Unexpected server error: 500 with url davs://cmsio5.rc.ufl.edu:1094/store/mc/RunIISummer20UL18RECO/ST_t-channel_antitop_4f_InclusiveDecays_wtop1p3_TuneCP5_13TeV-powheg-madspin-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v3/130000",

Brian described this in
#1466 (comment)

Is this an issue with the xrootd davs protocol or should this be fixed in FTS?
Thanks,
Bockjoo

[abh3]

The issue here is that the "stat" optimization has been enabled in the configuration file for the xrootd server being used. If you read past Brian's comment you will see that there is currently no way we can avoid that problem with that optimization enabled relative to the sequence of operations that FTS does. We are exploring several solutions to this problem but the only bypass available today is to either not specify the "mdhold" option on the "cms.dfs" directive or pre-create the directory. I know that neither solution is appealing.

[bockjoo]

Then, I guess I will try it without the mdhold option.
Thanks,
Bockjoo

[bockjoo]

I removed 'mdhold 20m' in the cms.dfs directive, but I still see the FTS error:
[root@cmsio8 ~]# grep cms.dfs /etc/xrootd/xrootd-clustered.cfg
#cms.dfs lookup distrib redirect immed
#cms.dfs lookup central mdhold 20m redirect verify
cms.dfs lookup central redirect verify

[root@cmsio8 ~]# grep "Unable to mkdir" /var/log/xrootd/clustered/xrootd.log | grep /store/mc/RunIISummer20UL18RECO/ST_t-channel_top_4f_hdampdown_InclusiveDecays_TuneCP5_13TeV-powheg-madspin-pythia8
210715 21:27:08 28972 ofs_mkdir: cmsprod.17946:118@cmsfts3.fnal.gov Unable to mkdir /store/mc/RunIISummer20UL18RECO/ST_t-channel_top_4f_hdampdown_InclusiveDecays_TuneCP5_13TeV-powheg-madspin-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v3/130000; no such file or directory

Is 'cms.dfs lookup central redirect verify' the correct config that you suggested in #1479 (comment) ?
Thanks,
Bockjoo

[abh3]

Hi Bockjoo, That would be correct. Not specifying mdhold causes the redirector to not hold on to the success or failure of a directory lookup. So, the question is who is actually issuing that message; the redirector or the server? Andy

…

On Thu, 15 Jul 2021, Bockjoo Kim wrote: I removed 'mdhold 20m' in the cms.dfs directive, but I still see the FTS error: ***@***.*** ~]# grep cms.dfs /etc/xrootd/xrootd-clustered.cfg #cms.dfs lookup distrib redirect immed #cms.dfs lookup central mdhold 20m redirect verify cms.dfs lookup central redirect verify ***@***.*** ~]# grep "Unable to mkdir" /var/log/xrootd/clustered/xrootd.log | grep /store/mc/RunIISummer20UL18RECO/ST_t-channel_top_4f_hdampdown_InclusiveDecays_TuneCP5_13TeV-powheg-madspin-pythia8 210715 21:27:08 28972 ofs_mkdir: ***@***.*** Unable to mkdir /store/mc/RunIISummer20UL18RECO/ST_t-channel_top_4f_hdampdown_InclusiveDecays_TuneCP5_13TeV-powheg-madspin-pythia8/AODSIM/106X_upgrade2018_realistic_v11_L1v1-v3/130000; no such file or directory Is 'cms.dfs lookup central redirect verify' the correct config that you suggested in #1479 (comment) ? Thanks, Bockjoo -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1479 (comment)

[bockjoo]

The server xrootd.log on cmsio8 has the error 'Unable to mkdir' and cmsio8 is the server.
Is this answer to your question? Are you asking something else?

[abh3]

That's exactly what I needed. Then indeed, on of the directory path elements must be missing. I should say that FTS should make the whole directory path and I am suprised that it does not.

…

On Thu, 15 Jul 2021, Bockjoo Kim wrote: The server xrootd.log on cmsio8 has the error 'Unable to mkdir' and cmsio8 is the server. Is this answer to your question? Are you asking something else? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1479 (comment)

[bockjoo]

The directory is created: Initially on cmsio8, it fails to create the directory but later the directory is created on another server.
FTS seems to be seeing only the initial directory creation failure and issues the error, I think.

[bockjoo]

By the way, I can not reproduce this by submitting an FTS transfer for a file with non-existing parent directory.
So, there may be something that I am missing.
I will test some more much later today and report back.
Thanks!

[abh3]

Then I am confused. The setup is using a DFS so it should not matter where the directories are created. So, maybe this is a misleading error issued by FTS and it later recovers and all is well?

…

On Thu, 15 Jul 2021, Bockjoo Kim wrote: The directory is created: Initially on cmsio8, it fails to create the directory but later the directory is created on another server. FTS seems to be seeing only the initial directory creation failure and issues the error, I think. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1479 (comment)

[abh3]

Can we close this ticket?

[bockjoo]

I've lost touch of the issue and don't remember how to continue the test.
Yes, please close this one.