You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was hunting for a more recent fork of bbcp - this one seems maintained and with more recent commits. Unfortunately - I had some issues using it. The copy seems to happen with no issues, but there's an issue with cleanup at the end and the return status of the program is not 0. I tried -s 1 and I still had the same problem.
Went with https://github.com/scaleengine/bbcp and no such issues. I can help debug if you guys are interested in getting to the bottom of this.
BTW I had it built in a container too if you want to easily reproduce it and you don't have access to debian or ubuntu.
FROM ruby:2.7-slim as BBCP_BUILDER
RUN apt-get update && \
apt-get install -y build-essential libssl-dev git
RUN git clone https://github.com/ICRAR/bbcp.git
# uname -i is not available in docker. Hardcode arch.
RUN sed -i 's/`\/bin\/uname -i`/x86_64/' bbcp/src/Makefile
RUN cd bbcp/src && make clean && make
Some examples:
free(): invalid pointer, killed by signal 6
bbcp -s 1 bbcp.C costi@ryzen7.home:/home/costi/
bbcp: Copy process 815813 was killed via signal 6
free(): invalid pointer
bbcp: Copy process 815807 was killed via signal 6
Copy process was killed via signal 11
bbcp -s 1 foo costi@ryzen7.home:/home/costi/
bbcp: Copy process 815345 was killed via signal 11
bbcp: Copy process 815341 was killed via signal 11
Full debug:
costi@bf3a566c00b4:/app$ bbcp --debug -i /ssh/id_rsa blah costi@ryzen7.home:/home/costi/
bbcp_CTL 946: Optimum MT level is 8
bbcp_CTL: Running as pid 947: ssh -x -a -oFallBackToRsh=no -i /ssh/id_rsa -l costi ryzen7.home bbcp SNK
bbcp_CTL: Sending to ryzen7.home: -D -m 644 -s 4 -W 131072 -Y 80dd4960 -H none:0
bbcp_bbcp_CTLCTL: Sending to ryzen7.home: 947/home/costi/:
PATH=
bbcp_CTL/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin: Sending to
ryzen7.home:
bbcp_SNK 17887: MaxWin=425984 MaxSeg=536 AT=1
bbcp_SNK 17887: Window set to 131072 AT=1
bbcp_SNK 17887: rwbsz=163840 BNum=12
bbcp_SNK 17887: Optimum MT level is 9
bbcp_SNK 17888: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
bbcp_SNK 17887: Thread 140132234241792 monitoring process 17835
bbcp_SNK 17887: bind recv window set to -1 (actual=131072 segsz=536)
bbcp_CTL 946: Received from ryzen7.home: 200 Port: 5031 Host: ryzen7
bbcp_CTL: Running as pid 948: bbcp SRC
bbcp_CTL: Sending to bf3a566c00b4: -D -m 644 -s 4 -W 131072 -Y 80dd4960 -H ryzen7:5031
bbcp_CTL: Sending to bf3a566c00b4: blah
bbcp_CTL: Sending to bf3a566c00b4:
bbcp_CTL 948: PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
bbcp_SRC 948: Max Window=425984 MaxSeg=536 AT=1
bbcp_SRC 948: Window set to 131072 AT=1
bbcp_SRC 948: rwbsz=163840 BNum=12
bbcp_SRC 948: Optimum MT level is 8
bbcp_SRC 949: PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
bbcp_SRC 948: Thread 139768807147264 monitoring process 877
bbcp_SRC 948: getFS: blah running in /tmp
bbcp_SRC 948: getFS norm blah
bbcp_SRC 948: FS_Unix::Stat for blah
bbcp_SRC.Stat: No such file or directory processing blah
bbcp_SRC 948: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 948: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login ctlr 80dd4960 wsz: +131072 ver: 17.12.00.00.0 dsz: 163840
bbcp_SRC 948: Received from ryzen7.home: 204 loginok wsz: 131072 163840
bbcp_SNK 17887: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17887: Received from costi-linux-zuper.home: login ctlr 80dd4960 wsz: +131072 ver: 17.12.00.00.0 dsz: 163840
bbcp_SNK: Sending to costi-linux-zuper.home: 204 loginok wsz: 131072 163840
bbcp_SRC 948: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 948: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 80dd4960
bbcp_SRC 948: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 948: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 80dd4960
bbcp_SNK 17887: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17887: Received from costi-linux-zuper.home: login data 80dd4960
bbcp_SNK 17887: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17887: Received from costi-linux-zuper.home: login data 80dd4960
bbcp_SRC 948: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 948: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 80dd4960
bbcp_SRC 948: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 948: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 80dd4960
bbcp_SNK 17887: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17887: Received from costi-linux-zuper.home: login data 80dd4960
bbcp_SRC 948: Allocated 12 163840 (159744) byte buffs in the net pool.
bbcp_SRC: Sending to ryzen7.home: eol
bbcp_SNK 17887: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17887: Received from costi-linux-zuper.home: login data 80dd4960
bbcp_SNK 17887: Allocated 12 167936 (163840) byte buffs in the net pool.
bbcp_SNK 17887: getFS: /home/costi/ running in /tmp
bbcp_SNK 17887: getFS norm /home/costi/
bbcp_SNK 17887: FS_Unix::Stat for /home/costi/
bbcp_SNK: Sending to costi-linux-zuper.home: flist
bbcp_SNK 17887: Received from costi-linux-zuper.home: eol
bbcp_bbcp_CTL 946: Received from ryzen7.home: 200 End: 0 0
bbcp_CTL 946: At end files=0 bytes=0
SNK: Sending to costi-linux-zuper.home: exit 0
bbcp_CTL 946: Process 947 ended; rc=0
bbcp_CTL.Waitpid: Copy process 948 was killed via signal 11
bbcp_CTL 946: Process 948 ended; rc=255
bbcp_CTL 946: Ending; rc=255 files=0 bytes=0
costi@bf3a566c00b4:/app$ bbcp --debug -i /ssh/id_rsa blah costi@ryzen7.home:/home/costi/
bbcp_CTL 955: Optimum MT level is 8
bbcp_CTL: Running as pid 956: ssh -x -a -oFallBackToRsh=no -i /ssh/id_rsa -l costi ryzen7.home bbcp SNK
bbcp_bbcp_CTLCTL: Sending to ryzen7.home: -D -m 644 -s 4 -W 131072 -Y 9736e633 -H none:0
956
: PATH=bbcp_CTL: Sending to ryzen7.home/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:
/home/costi/
bbcp_CTL: Sending to ryzen7.home:
bbcp_SNK 17948: MaxWin=425984 MaxSeg=536 AT=1
bbcp_SNK 17948: Window set to 131072 AT=1
bbcp_SNK 17948: rwbsz=163840 BNum=12
bbcp_SNK 17948: Optimum MT level is 9
bbcp_SNK 17949: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
bbcp_SNK 17948: Thread 140651933218560 monitoring process 17896
bbcp_SNK 17948: bind recv window set to -1 (actual=131072 segsz=536)
bbcp_CTL 955: Received from ryzen7.home: 200 Port: 5031 Host: ryzen7
bbcp_CTL: Running as pid 957: bbcp SRC
bbcp_CTL: Sending to bf3a566c00b4: -D -m 644 -s 4 -W 131072 -Y 9736e633 -H ryzen7:5031
bbcp_CTL: Sending to bf3a566c00b4: blah
bbcp_CTL: Sending to bf3a566c00b4:
bbcp_CTL 957: PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
bbcp_SRC 957: Max Window=425984 MaxSeg=536 AT=1
bbcp_SRC 957: Window set to 131072 AT=1
bbcp_SRC 957: rwbsz=163840 BNum=12
bbcp_SRC 957: Optimum MT level is 8
bbcp_SRC 958: PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
bbcp_SRC 957: Thread 140299876443904 monitoring process 877
bbcp_SRC 957: getFS: blah running in /tmp
bbcp_SRC 957: getFS norm blah
bbcp_SRC 957: FS_Unix::Stat for blah
bbcp_SRC.Stat: No such file or directory processing blah
bbcp_SRC 957: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 957: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login ctlr 9736e633 wsz: +131072 ver: 17.12.00.00.0 dsz: 163840
bbcp_SRC 957: Received from ryzen7.home: 204 loginok wsz: 131072 163840
bbcp_SNK 17948: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17948: Received from costi-linux-zuper.home: login ctlr 9736e633 wsz: +131072 ver: 17.12.00.00.0 dsz: 163840
bbcp_SNK: Sending to costi-linux-zuper.home: 204 loginok wsz: 131072 163840
bbcp_SRC 957: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 957: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 9736e633
bbcp_SRC 957: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 957: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 9736e633
bbcp_SRC 957: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 957: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 9736e633
bbcp_bbcp_SNK 17948: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17948: Received from costi-linux-zuper.home: login data 9736e633
SRC 957: connect send window set to -1 (actual=16384 segsz=536)
bbcp_SRC 957: Connected to: ryzen7.home:5031
bbcp_SRC: Sending to ryzen7.home: login data 9736e633
bbcp_SNK 17948: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17948: Received from costi-linux-zuper.home: login data 9736e633
bbcp_SRC 957: Allocated 12 163840 (159744) byte buffs in the net pool.
bbcp_SRC: Sending to ryzen7.home: eol
free(): invalid pointer
bbcp_SNK 17948: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17948: Received from costi-linux-zuper.home: login data 9736e633
bbcp_SNK 17948: accept recv window set to -1 (actual=131072 segsz=1448)
bbcp_SNK 17948: Received from costi-linux-zuper.home: login data 9736e633
bbcp_SNK 17948: Allocated 12 167936 (163840) byte buffs in the net pool.
bbcp_SNK 17948: getFS: /home/costi/ running in /tmp
bbcp_SNK 17948: getFS norm /home/costi/
bbcp_SNK 17948: FS_Unix::Stat for /home/costi/
bbcp_SNK: Sending to costi-linux-zuper.home: flist
bbcp_SNK 17948: Received from costi-linux-zuper.home: eol
bbcp_SNK: Sending to costi-linux-zuper.home: exit 0
bbcp_CTL 955: Received from ryzen7.home: 200 End: 0 0
bbcp_CTL 955: At end files=0 bytes=0
bbcp_CTL 955: Process 956 ended; rc=0
bbcp_CTL.Waitpid: Copy process 957 was killed via signal 6
bbcp_CTL 955: Process 957 ended; rc=255
bbcp_CTL 955: Ending; rc=255 files=0 bytes=0
The text was updated successfully, but these errors were encountered:
@costi if you are happy to debug the problem that's great 👍. We maintain this fork at a very low capacity (basically fixing things we have found during the years too), and while I'll be happy to merge back a fix for this issue I don't think we can invest any time on our end to hunt it down.
I was hunting for a more recent fork of
bbcp
- this one seems maintained and with more recent commits. Unfortunately - I had some issues using it. The copy seems to happen with no issues, but there's an issue with cleanup at the end and the return status of the program is not0
. I tried-s 1
and I still had the same problem.Went with https://github.com/scaleengine/bbcp and no such issues. I can help debug if you guys are interested in getting to the bottom of this.
BTW I had it built in a container too if you want to easily reproduce it and you don't have access to debian or ubuntu.
Some examples:
free(): invalid pointer, killed by signal 6
Copy process was killed via signal 11
Full debug:
The text was updated successfully, but these errors were encountered: