blocksync.py python3 bug

[devZer0]

i'm getting this when using python3 version of blocksync. pyhton2 version works

/root/borg extract --stdout /backup/test::second | mbuffer -q | /root/blocksync.py - localhost /dev/sdc

Dry run : False
Local : True
Block size : 128.0 KB
Skipped : 0 blocks
Hash alg : sha512
Crypto alg : aes128-cbc
Compression : False
Read cache : True
SRC command : /root/blocksync.py - localhost /dev/sdc
DST command : /root/blocksync.py server /dev/sdc -a sha512 -b 131072 -k 0

Synching...
Traceback (most recent call last):
File "/root/blocksync.py", line 386, in
sync(srcpath, dsthost, dstpath)
File "/root/blocksync.py", line 246, in sync
for (l_block, l_sum) in getblocks(f):
File "/root/blocksync.py", line 110, in getblocks
block = f.read(options.blocksize)
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 0: invalid continuation byte
Traceback (most recent call last):
File "/root/blocksync.py", line 378, in
server(dstpath)
File "/root/blocksync.py", line 149, in server
res, complen = in_line.split(":")
ValueError: not enough values to unpack (expected 2, got 1)
mbuffer: error: outputThread: error writing to at offset 0x20000: Broken pipe
mbuffer: warning: error during output to : Broken pipe
Local Exception
Traceback (most recent call last):
File "borg/archiver.py", line 5168, in main
File "borg/archiver.py", line 5099, in run
File "borg/archiver.py", line 183, in wrapper
File "borg/archiver.py", line 198, in wrapper
File "borg/archiver.py", line 917, in do_extract
File "borg/archive.py", line 781, in extract_item
BrokenPipeError: [Errno 32] Broken pipe

Platform: Linux debian 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.3 Python: CPython 3.9.16 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 1711 CWD: /root
sys.argv: ['/root/borg', 'extract', '--stdout', '/backup/test::second']
SSH_ORIGINAL_COMMAND: None

[shodanshok]

I just commited a patch on the python3 branch to fix the issue, please let me know if it works.

[devZer0]

thanks very much!

unfortunately, still errors out with the same error

[shodanshok]

You are right, an issue remained when destination was a blockdev. I fixed it, let it try now.

[devZer0]

i'm really sorry, but it still doesn't work

/root/borg extract --stdout /backup/test::second | mbuffer -q | /root/blocksync3.py - localhost /dev/sdc

Dry run : False
Local : True
Block size : 128.0 KB
Skipped : 0 blocks
Hash alg : sha512
Crypto alg : aes128-cbc
Compression : False
Read cache : True
SRC command : /root/blocksync3.py - localhost /dev/sdc
DST command : /root/blocksync3.py server /dev/sdc -a sha512 -b 131072 -k 0

Synching...
Traceback (most recent call last):
File "/root/blocksync3.py", line 392, in
sync(srcpath, dsthost, dstpath)
File "/root/blocksync3.py", line 250, in sync
for (l_block, l_sum) in getblocks(f):
File "/root/blocksync3.py", line 110, in getblocks
block = f.read(options.blocksize)
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 0: invalid continuation byte
mbuffer: error: outputThread: error writing to at offset 0x20000: Broken pipe
mbuffer: warning: error during output to : Broken pipe
Local Exception
Traceback (most recent call last):
File "borg/archiver.py", line 5168, in main
File "borg/archiver.py", line 5099, in run
File "borg/archiver.py", line 183, in wrapper
File "borg/archiver.py", line 198, in wrapper
File "borg/archiver.py", line 917, in do_extract
File "borg/archive.py", line 781, in extract_item
BrokenPipeError: [Errno 32] Broken pipe

Platform: Linux debian 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.3 Python: CPython 3.9.16 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 7324 CWD: /root
sys.argv: ['/root/borg', 'extract', '--stdout', '/backup/test::second']
SSH_ORIGINAL_COMMAND: None

[shodanshok]

Don't be sorry, rather, thanks for your help!

This issue turned to be a rather fundamental problem with how python3 encodes string (compared to python2). I hope to have fixed it, can you try it now?

[devZer0]

i'm very happy that things moving forward !

it loooks good now and seems to work. i did not yet test for consistency/data validity, but restore works.

i observe slightly performance loss. the old version does about 255MB/s whereas the new one does not perform above 242MB/s. it was no real performance test, though... not an issue, just wanted to let you know.

thank you very much !

[shodanshok]

I'm glad to hear that it now works!

it loooks good now and seems to work. i did not yet test for consistency/data validity, but restore works.

Be sure to thoroughly test it, taking and comparing some hashes from both source and destination, before using it for anything valuable.

[devZer0]

looks good, so far

# /root/borg extract --stdout /backup/test::second | mbuffer -q | /root/blocksync3.py - localhost /dev/sdc


Dry run     : False
Local       : True
Block size  : 128.0 KB
Skipped     : 0 blocks
Hash alg    : sha512
Crypto alg  : aes128-cbc
Compression : False
Read cache  : True
SRC command : /root/blocksync3.py - localhost /dev/sdc
DST command : /root/blocksync3.py server /dev/sdc -a sha512 -b 131072 -k 0


Synching...
skipped: 0, same: 262135, diff: 9, 262144/1, 220.8 MB/s

Completed in 148 seconds

# md5sum /dev/sdc
ff27c3dd96d7abd93bfb76176eaf6293  /dev/sdc

# dd if=/dev/urandom of=/dev/sdc1 bs=1024k count=1
1+0 Datensätze ein
1+0 Datensätze aus
1048576 Bytes (1,0 MB, 1,0 MiB) kopiert, 0,00430363 s, 244 MB/s

# dd if=/dev/urandom of=/dev/sdc2 bs=1024k count=1
dd: Fehler beim Schreiben von '/dev/sdc2': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
1+0 Datensätze ein
0+0 Datensätze aus
1024 Bytes (1,0 kB, 1,0 KiB) kopiert, 0,00460056 s, 223 kB/s

# dd if=/dev/urandom of=/dev/sdc3 bs=1024k count=1
1+0 Datensätze ein
1+0 Datensätze aus
1048576 Bytes (1,0 MB, 1,0 MiB) kopiert, 0,00327241 s, 320 MB/s


# /root/borg extract --stdout /backup/test::second | mbuffer -q | /root/blocksync3.py - localhost /dev/sdc


Dry run     : False
Local       : True
Block size  : 128.0 KB
Skipped     : 0 blocks
Hash alg    : sha512
Crypto alg  : aes128-cbc
Compression : False
Read cache  : True
SRC command : /root/blocksync3.py - localhost /dev/sdc
DST command : /root/blocksync3.py server /dev/sdc -a sha512 -b 131072 -k 0


Synching...
skipped: 0, same: 262135, diff: 9, 262144/1, 218.1 MB/s

Completed in 150 seconds

# md5sum /dev/sdc
ff27c3dd96d7abd93bfb76176eaf6293  /dev/sdc