XrdCl-Http client transfer to EOS storage finishing with Segmentation fault and incomplete file transferred. #1891
Assignees
Labels
Pending Info
Waiting on additional information from issue reporter
Comments
|
For small files the transfer seems to work fine: |
|
|
The issue here is with Davix. Using xrdcp in the way you do requires that
Davix first write out the data into a local file before transferring the
local file to EOS. As you see, your local file system runs our of space to
hold a copy of a 4GB file (this file, by default, is written to /tmp which
simply does nor have enough space). After that, everything goes downhill.
The SEGV is clearly an issue but with atleast a stack trace with statement
numbers there is not much we can do.
The solution here, since you are transferring an actual file into EOS, is
to use a tool like curl or some other http based copy program.
Andy
…On Thu, 2 Feb 2023, Jaroslav Guenther wrote:
Reproduction of the problem transferring over http to EOS storage.
Preparation checks:
```
***@***.*** ~]# ll /var/tmp/4G.file
-rw-r--r--. 1 root root 4294967296 Jan 25 16:53 /var/tmp/4G.file
***@***.*** ~]# rpm -ql xrdcl-http
/etc/xrootd/client.plugins.d/xrdcl-http-plugin.conf
/usr/lib64/libXrdClHttp-5.so
***@***.*** ~]# rpm -qa xrdcl-http
xrdcl-http-5.5.1-1.el7.x86_64
***@***.*** ~]# which xrdcp
/usr/bin/xrdcp
***@***.*** ~]# xrdcp --version
v5.5.1
***@***.*** ~]# ll /tmp/x509up_u12354
-rw-------. 1 root root 6325 Feb 2 14:12 /tmp/x509up_u12354
***@***.*** ~]# export X509_USER_PROXY=/tmp/x509up_u12354
***@***.*** (mgm:master mq:slave) ~]$ eos ls -l /eos/ams02/opstest/guenther/4G.file
Unable to stat /eos/ams02/opstest/guenther/4G.file; No such file or directory (errc=2) (No such file or directory)
```
The execution of the transfer command:
```
***@***.*** ~]# time XrdSecPROTOCOL=gsi,unix xrdcp -f --debug 2 --allow-http /var/tmp/4G.file "https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.ruid=12354&eos.rgid=2763"
[2023-02-02 14:13:03.709247 +0100][Debug ][Utility ] Env: overriding entry: tlsnodata=0 with 1
[2023-02-02 14:13:03.709485 +0100][Debug ][Utility ] CopyProcess: 2 jobs to prepare
[2023-02-02 14:13:03.709556 +0100][Debug ][Utility ] Creating a classic copy job, from file://localhost/var/tmp/4G.file to https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354
[2023-02-02 14:13:03.709579 +0100][Debug ][Utility ] Monitor library name not set. No monitoring
[2023-02-02 14:13:03.709652 +0100][Debug ][Utility ] Opening file://localhost/var/tmp/4G.file for reading
[2023-02-02 14:13:03.709741 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending an open command
[2023-02-02 14:13:03.709839 +0100][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference
[2023-02-02 14:13:03.709849 +0100][Debug ][Poller ] Available pollers: built-in
[2023-02-02 14:13:03.709854 +0100][Debug ][Poller ] Attempting to create a poller according to preference: built-in
[2023-02-02 14:13:03.709859 +0100][Debug ][Poller ] Creating poller: built-in
[2023-02-02 14:13:03.709870 +0100][Debug ][Poller ] Creating and starting the built-in poller...
[2023-02-02 14:13:03.710119 +0100][Debug ][Poller ] Using 1 poller threads
[2023-02-02 14:13:03.710136 +0100][Debug ][TaskMgr ] Starting the task manager...
[2023-02-02 14:13:03.710175 +0100][Debug ][TaskMgr ] Task manager started
[2023-02-02 14:13:03.710181 +0100][Debug ][JobMgr ] Starting the job manager...
[2023-02-02 14:13:03.710255 +0100][Debug ][JobMgr ] Job manager started, 3 workers
[2023-02-02 14:13:03.710264 +0100][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2023-02-02 14:13:03 +0100]
[2023-02-02 14:13:03.710375 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Open has returned with status [SUCCESS]
[2023-02-02 14:13:03.710400 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] successfully opened at localhost, handle: 0x7, session id: 1
[2023-02-02 14:13:03.710483 +0100][Debug ][Utility ] Opening https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296 for writing
[2023-02-02 14:13:03.710536 +0100][Debug ][XrdClHttp ] HttpFilePlugin constructed.
[2023-02-02 14:13:03.747783 +0100][Debug ][XrdClHttp ] Open: URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296, XRootD flags: 34, POSIX flags: 578
[2023-02-02 14:13:03.766476 +0100][Debug ][XrdClHttp ] Opened: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
[2023-02-02 14:13:03.766583 +0100][Error ][Utility ] The given URL is empty
[2023-02-02 14:13:03.766661 +0100][Error ][PostMaster ] Unable to get transport handler for protocol
[2023-02-02 14:13:03.784673 +0100][Debug ][XrdClHttp ] Stat-ed URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
[2023-02-02 14:13:03.784786 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
[2023-02-02 14:13:03.784917 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
[2023-02-02 14:13:03.784994 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
[2023-02-02 14:13:03.785017 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
[2023-02-02 14:13:03.811201 +0100][Debug ][XrdClHttp ] Wrote 8388608 bytes, at offset 0, to URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
[8MB/4GB][ 0%][> ][8MB/s] [2023-02-02 14:13:03.811375 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
[2023-02-02 14:13:03.836593 +0100][Error ][XrdClHttp ] Could not write URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296, error: [ERROR] Internal error: not a directory: Impossible to write to fdNo space left on device
[2023-02-02 14:13:03.838100 +0100][Debug ][XrdClHttp ] Closing davix fd: 20923904
Segmentation fault
real 0m0.317s
user 0m0.078s
sys 0m0.139s
```
Result is an incomplete file present on the storage:
```
***@***.*** (mgm:master mq:slave) ~]$ eos ls -l /eos/ams02/opstest/guenther/4G.file
-rw-r--r-- 2 guenther it 8454144 Feb 2 14:17 4G.file
``
I do have space in that path on EOS:
```
***@***.*** (mgm:master mq:slave) ~]$ eos quota ls /eos/ams02/opstest/guenther/
??> Quota Node: /eos/ams02/opstest/guenther/
???????????????????????????????????????????????????????????????????????????????????????????????????????????????
?user ?used bytes?logi bytes?used files?aval bytes?aval logib?aval files? filled[%]?vol-status?ino-status?
???????????????????????????????????????????????????????????????????????????????????????????????????????????????
guenther 33.82 MB 16.91 MB 4 2.00 TB 1.00 TB 0 0.00 % ok ignored
```
--
Reply to this email directly or view it on GitHub:
#1891
You are receiving this because you are subscribed to this thread.
Message ID: ***@***.***>
|
|
Have you solved this issue and can I close this ticket? |
|
Given no response then we assume this problem has at least a bypass. So, I am closing this ticket. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reproduction of the problem transferring over http to EOS storage.
Preparation checks:
The execution of the transfer command:
Result is an incomplete file present on the storage:
[root@eosams02-ip563 (mgm:master mq:slave) ~]$ eos quota ls /eos/ams02/opstest/guenther/
┏━> Quota Node: /eos/ams02/opstest/guenther/
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│user │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
guenther 33.82 MB 16.91 MB 4 2.00 TB 1.00 TB 0 0.00 % ok ignored
The text was updated successfully, but these errors were encountered: