xrdcl-http stating wrong path #1394
Comments
|
I can't repeat this error. Is this from the xrootd log? Can I take a look at your xcache config file, and the script you used to starte the xcache? |
|
Yes it is from the xrootd logs here I use this command when I see this problem I get the problem only when the cache is empty. When the file was fetch through xroot and is already present on the cache and I use the same commend as above it works. Here are the xcache and xrdcl http config files I am launching xcache with |
|
Thanks for more info. I was having difficulty to have my log showing the "[XrdClHttp ] HttpFileSystemPlugIn...", until I realized I was using root://xcache//https://data_source. Those info will show up in the log if I use https://xcache//https//data_source. This is a known problem in xrdcl-http. It has been fixed (xrootd/xrdcl-http@323be8f#diff-5c4ea2f2572ee031fc0af08f4074c2d2c7c8aeb39751d20efacedc20b5a47e3dL152), and is now committed to the xrdcl-http. For a while I was a bit hesitate to commit the change. There are a few other places like this in HttpFileSysmtePlugIn.cc (and I don't understand the original author's intention, some said it was there to handle relative paths). |
|
Seems to work in RC7 but it doesn't looks like it was what I thought was the origin of another problem. And in the xcache logs I get Should I open a new issue ? |
|
Does this happen to all files or just this file? will it work with files from other site? I have a similar setup using 5.1.0.rc7 and it works for me, thouhgh it is very slow (<500kb/s) to get the 768MB file |
|
The particular error is triggered by a TLS failure which causes this. It
would be interesting to know if this is repeatable as we can't seem to
repeat it on our end. If so, you could up the debug level and then we
would have something to go on.
…On Wed, 10 Feb 2021, Wei Yang wrote:
Does this happen to all files or just this file? will it work with files from other site?
I have a similar setup using 5.1.0.rc7 and it works for me, thouhgh it is very slow (<500kb/s) to get the 768MB file
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#1394 (comment)
|
|
Really strange. No I dont have the problem with three of the four sites I tried yesterday. Still happens with an EOS instance at Cern Which debug level should I rise ? Is there still an issue where xrdcl-http doesnt check in |
|
Since this is a client problem, set the following envar and restart:
XRD_LOGLEVEL=Debug
We can turn on additional debugging if this doesn't yield sufficient
information. The log file will be rather large so just send it to me via
email or provide a pointer so I can fetch it. The log file may have
informtion that you would rather not make public so avoid posting it.
Andy
…On Thu, 11 Feb 2021, pmusset wrote:
Really strange. No I dont have the problem with three of the four sites I tried yesterday. Still happens with an EOS instance at Cern
Normally nothing has changed since yesterday. Or I didnt pull correctly my docker image with the rc7 and it only worked today.
Which debug level should I rise ?
Is there still an issue where xrdcl-http doesnt check in `/etc/grid-security/certificates` for the certificates ? Could it be linked to that ? On XrootD 4 I know I had to do
```
RUN cp /etc/grid-security/certificates/*.pem /etc/pki/ca-trust/source/anchors/ && update-ca-trust extract
```
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#1394 (comment)
|
|
This seems to have stalled and this bug is rated as critical. Perhaps it's not critical and we should downgrade it. What's the status? |
|
Sorry I was in holiday last week. |
|
@pmusset is this completely reproducible? Can you try running with a higher xrdcl log-level, like this: This will produce a lot of output, so start with lower level (say, 1 or 2) if this takes time to happen (note, the numbers are not progressive in the log level). It seems this should be reproducible by just running xrdcp with the same remote, i.e., I'm puzzled by the errno 42, do you have hitchhiker's hiding around your server room? |
|
Error 42 is a fallout of the posix layer not recognizing a new error code
that the xroot client is returning. It's a catchall response. I really
need to correct that.
Andy
…On Tue, 23 Feb 2021, Matev? Tadel wrote:
@pmusset is this completely reproducible? Can you try running with a higher xrdcl log-level, like this:
```
# To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2 Warning, 1 Info)
pss.setopt DebugLevel 4
pss.trace all
```
This will produce a lot of output, so start with lower level (say, 1 or 2) if this takes time to happen (note, the numbers are not progressive in the log level).
It seems this should be reproducible by just running xrdcp with the same remote, i.e.,
```
xrdcp -A -d2 -f https://ccdcalitest10.in2p3.fr:2880//pnfs/in2p3.fr/data/escape/cc_in2p3_dcache/NESSIE/35/6b/test_iorandom_768MB_8.out /dev/null
```
I'm puzzled by the errno 42, do you have hitchhiker's hiding around your server room?
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#1394 (comment)
|
|
@pmusset, from the xrootd server log file you sent to Andy, i see these lines:
This tells me that the xrdcl-http stat() request was successful because it got the correct file size. I can do the same thing using BTW, xrdcl-http open() was also successful, but I guess open() is probably meaningless in HTTP protocol. It is only meaningful in the Davix library. Later in the xrootd server log, I see:
After seeing this, I tried to use curl to copy (with my ATLAS and dTeam certificates). Though I got a "HTTP/1.1 200 OK", the content I got back is not your data but a html file telling me "There was an error loading the page you requested — This page may have been deleted or moved." along with a few links to CERN and a CERN logo. So, question: what certificate should I use to access this file. If I don't have the right x509 certificate, I expect the web server NOT to tell me "HTTP/1.1 200 OK" but something like "HTTP/1.1 403" (btw, if I don't have an x509 certificate at all, and use the -k option with curl, I do get "HTTP/1.1 403 token authorization failed"). So what a mess in the world of industry standard? |
|
OK, what the last log shows me is that this is an error that happens server-side when doing async I/O and is related to it running out of AIO objects in certain cases. This has been fixed just a couple of days ago and the fix will appear in 5.1.1. That problem has been in the code since day one and only rears it's ugly head when either the server is overloaded or when the client is relatively slow. Anyway, I assume the "wrong path" problem has been fixed and this is simply a new problem, yes? |
|
Yes. The ticket can be closed I think |
|
The initial problem has been resolved. |
I am trying to use xrdcl-http with xcache on xrootd 5.1-rc6 and he seems that when it is trying to access to a file at the origin, it is putting the filepath twice. For example, if I try to access
https://origin.in2p3.fr:2880/path/of/the/file, it seems like it is trying to accesshttps:/origin.in2p3.fr:2880/path/of/the/file/path/of/the/fileThe text was updated successfully, but these errors were encountered: