Fail when renaming with different case on case insensitive file system

[mkllnk]

I'm using rdiff-backup 1.2.8 to backup files on a Windows machine with cygwin. The source is the local hard drive and the backup destination is an external hard drive, both NTFS. Renaming a folder called archive to Archive causes an AssertError assert not incrp.lstat(), incrp. This has been reported before: https://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/case-insensitive-filesystems-assert-not-incrp-lstat-100362/

Renaming the folder to its original name works after running rdiff-backup --check-destination-dir "$dst".

I wrote a script to reproduce that error:

#!/bin/sh

src="$(mktemp -d)"
dst="$(mktemp -d)"
echo "backup source: $src"
echo "backup destination: $dst"

mkdir "$src/archive"
rdiff-backup "$src" "$dst"

sleep 1
mv "$src/archive" "$src/Archive-tmp"
mv "$src/Archive-tmp" "$src/Archive"
rdiff-backup "$src" "$dst"

ls -la "$src"
ls -la "$dst"
rm "$src" -rf
rm "$dst" -rf

I ran that script on my linux machine with a FAT32 usb drive: TMPDIR=/media/C808-E853/ sh rdiff-backup-23.sh

It gave me a different error, but the bug is probably related:

backup source: /media/C808-E853/tmp.a0Op9QfBAo
backup destination: /media/C808-E853/tmp.l1QgeTHVzJ
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.l1QgeTHVzJ/rdiff-backup-data
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.l1QgeTHVzJ/rdiff-backup-data
Exception '[Errno 17] File exists: '/media/C808-E853/tmp.l1QgeTHVzJ/Archive'' raised of class '<type 'exceptions.OSError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)
OSError: [Errno 17] File exists: '/media/C808-E853/tmp.l1QgeTHVzJ/Archive'

[davekempe]

Hi there,
I know is is an annoying answer, but can you try 1.3.3? There are many fixes for this sort of problem in this version. It is very stable, despite the name.

[mkllnk]

Sure, almost no problem. But I can't compile it even though I have python2.7-dev and librsync-dev installed (Debian Wheezy). I tried it with python2.6 as well. No success.

python2.7 ./dist/setup.py build

running build
running build_py
copying rdiff_backup/SetConnections.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rpath.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/log.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/fs_abilities.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/eas_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/win_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/metadata.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/robust.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Main.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Globals.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/statistics.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/FilenameMapping.py -> build/lib.linux-x86_64-2.7/rdiff_backup
running build_ext
building 'rdiff_backup.C' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cmodule.c -o build/temp.linux-x86_64-2.7/cmodule.o
gcc: error: cmodule.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4

[davekempe]

Hrmm, not sure why it isn't working for you. I built it from source on a fresh wheezy container just for testing:

root@wheezy-test:~# apt-get install librsync-dev python-dev wget
dev@wheezy-test:~$ wget http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-1.3.3.tar.gz
dev@wheezy-test:~$ tar zxf rdiff-backup-1.3.3.tar.gz
dev@wheezy-test:~/rdiff-backup-1.3.3$ python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/TempFile.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/log.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/SetConnections.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Globals.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/longname.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/static.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Hardlink.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/metadata.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rpath.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/librsync.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/fs_abilities.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/__init__.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/hash.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Main.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/user_group.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rorpiter.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/eas_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Rdiff.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Time.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/restore.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/FilenameMapping.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/lazy.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/compare.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/statistics.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/win_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/selection.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/backup.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/iterfile.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/manage.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/regress.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/robust.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Security.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/increment.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/connection.py -> build/lib.linux-x86_64-2.7/rdiff_backup
running build_ext
building 'rdiff_backup.C' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cmodule.c -o build/temp.linux-x86_64-2.7/cmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/cmodule.o -o build/lib.linux-x86_64-2.7/rdiff_backup/C.so
building 'rdiff_backup._librsync' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _librsyncmodule.c -o build/temp.linux-x86_64-2.7/_librsyncmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/_librsyncmodule.o -lrsync -o build/lib.linux-x86_64-2.7/rdiff_backup/_librsync.so
running build_scripts
creating build/scripts-2.7
copying and adjusting rdiff-backup -> build/scripts-2.7
copying and adjusting rdiff-backup-statistics -> build/scripts-2.7
changing mode of build/scripts-2.7/rdiff-backup from 644 to 755
changing mode of build/scripts-2.7/rdiff-backup-statistics from 644 to 755
dev@wheezy-test:~/rdiff-backup-1.3.3$ which python
/usr/bin/python
dev@wheezy-test:~/rdiff-backup-1.3.3$ ls -al /usr/bin/python
lrwxrwxrwx 1 root root 9 Sep 29  2013 /usr/bin/python -> python2.7
dev@wheezy-test:~/rdiff-backup-1.3.3$

I don't know what you did differently, but it does work building from source on wheezy.

[mkllnk]

Thank you. That worked. I tried to use the code from the github repository. The directory structure is different and setup.py is in the dist folder.

Anyway, I was able to build 1.3.3 and ran my script again on a FAT32 usb stick. Here is the error:

Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.AOP5zanMyu/rdiff-backup-data
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.AOP5zanMyu/rdiff-backup-data
Exception '[Errno 17] File exists: '/media/C808-E853/tmp.AOP5zanMyu/Archive'' raised of class '<type 'exceptions.OSError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)

Traceback (most recent call last):
  File "./rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)
OSError: [Errno 17] File exists: '/media/C808-E853/tmp.AOP5zanMyu/Archive'

[LeonardMichlmayr]

I have just noticed some trouble with rdiff-backup on vfat in linux. The filesystem is not completely case-insensitive when mounted with the option iocharset=utf8.
$ touch A ; touch a
will result in an error, but
$ touch a; touch A
will not.

$ stat a
will not recognize the file A, but
$ touch a
is not allowed to create the file a if A exists.

rdiff-backup should always assume case insensitive behavior on vfat! If not, you will run into above error, as soon as there are two files with the same lowercase representation. Currently, rdiff-backup mistakes vfat as case sensitive!

[ericzolf]

OK, quick check on an USB stick with vfat and the result is... baaaad:

$ PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7 PATH=$PWD/build/scripts-3.7:$PATH rdiff-backup ../rdiff-backup_testfiles/various_file_types /run/media/myuser/MYUSBSTICK/backup1
Warning: hard linking not supported by filesystem at /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup-data
b'SpecialFileError fifo [Errno 1] Operation not permitted'
b"SpecialFileError symbolic_link [Errno 1] Operation not permitted: b'regular_file' -> b'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.8'"
OSError while renaming /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13 to /run/media/myuser/MYUSBSTICK/backup1/ث�;087b��]��;021v;042�;015!�;062�;089����p��;019k;029���e�;085��;085;086���4��;088;003�;007sΞ��4;004�;023 ����جڅ�;075v;067�#���ɷ�_;015�g�;066;017;060=^�;077;019�c��;124;042;034;092'^$@#!(){};063+ ~`;032
b'UpdateError rdiff-backup.tmp.13 [Errno 22] Invalid argument: b\'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13\' -> b"/run/media/myuser/MYUSBSTICK/backup1/\\xd8\\xab\\xb1;087b\\xae\\xc5]\\x8a\\xbb;021v;042\\xf4;015!\\xf9;062\\xe2;089\\x86\\xbb\\xab\\xdbp\\xb0\\x84;019k;029\\xc2\\xf1\\xf5e\\xa5;085\\x82\\x9a;085;086\\xa0\\xf4\\xdf4\\xba\\xfd;088;003\\x82;007s\\xce\\x9e\\x8b\\xb34;004\\x9f;023 \\xf4\\x8f\\xa6\\xfa\\x97\\xab\\xd8\\xac\\xda\\x85\\xdc;075v;067\\xfa#\\x94\\x92\\x9e\\xc9\\xb7\\xc3_;015\\x84g\\x9a;066;017;060=^\\xdb;077;019\\x96c\\x8b\\xa7;124;042;034;092\'^$@#!(){};063+ ~`;032"'

After a sudo mkfs.ntfs -L MYUSBSTICK /dev/sda1, 2nd try on NTFS doesn't look better:

$ PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7 PATH=$PWD/build/scripts-3.7:$PATH rdiff-backup ../rdiff-backup_testfiles/various_file_types /run/media/myuser/MYUSBSTICK/backup2
b"UpdateError two_hardlinked_files2 Updated mirror temp file b'/run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.11' does not match source"
OSError while renaming /run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.12 to /run/media/myuser/MYUSBSTICK/backup2/Ø«±Wb®Å]»;021v;042ô;015!ù;062âY»«Ûp°
                                         ;019k;029Âñõe¥UV ôß4ºýX;003;007sγ4;004¦ú«Ø¬Ú
ÜKvCú#
      g;017;060=^ÛM;019c§;124;042;034;092'^$@#!(){};063+ ~` 
b'UpdateError rdiff-backup.tmp.12 [Errno 22] Invalid argument: b\'/run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.12\' -> b"/run/media/myuser/MYUSBSTICK/backup2/\\xc3\\x98\\xc2\\xab\\xc2\\xb1Wb\\xc2\\xae\\xc3\\x85]\\xc2\\x8a\\xc2\\xbb;021v;042\\xc3\\xb4;015!\\xc3\\xb9;062\\xc3\\xa2Y\\xc2\\x86\\xc2\\xbb\\xc2\\xab\\xc3\\x9bp\\xc2\\xb0\\xc2\\x84;019k;029\\xc3\\x82\\xc3\\xb1\\xc3\\xb5e\\xc2\\xa5U\\xc2\\x82\\xc2\\x9aUV\\xc2\\xa0\\xc3\\xb4\\xc3\\x9f4\\xc2\\xba\\xc3\\xbdX;003\\xc2\\x82;007s\\xc3\\x8e\\xc2\\x9e\\xc2\\x8b\\xc2\\xb34;004\\xc2\\x9f;023 \\xc3\\xb4\\xc2\\x8f\\xc2\\xa6\\xc3\\xba\\xc2\\x97\\xc2\\xab\\xc3\\x98\\xc2\\xac\\xc3\\x9a\\xc2\\x85\\xc3\\x9cKvC\\xc3\\xba#\\xc2\\x94\\xc2\\x92\\xc2\\x9e\\xc3\\x89\\xc2\\xb7\\xc3\\x83_;015\\xc2\\x84g\\xc2\\x9aB;017;060=^\\xc3\\x9bM;019\\xc2\\x96c\\xc2\\x8b\\xc2\\xa7;124;042;034;092\'^$@#!(){};063+ ~` "'
Exception '[Errno 84] Invalid or incomplete multibyte or wide character: b'/run/media/myuser/MYUSBSTICK/backup2/\xd8\xab\xb1Wb\xae\xc5]\x8a\xbb\x15v*\xf4\x0f!\xf9>\xe2Y\x86\xbb\xab\xdbp\xb0\x84\x13k\x1d\xc2\xf1\xf5e\xa5U\x82\x9aUV\xa0\xf4\xdf4\xba\xfdX\x03\x82\x07s\xce\x9e\x8b\xb34\x04\x9f\x17 \xf4\x8f\xa6\xfa\x97\xab\xd8\xac\xda\x85\xdcKvC\xfa#\x94\x92\x9e\xc9\xb7\xc3_\x0f\x84g\x9aB\x11<=^\xdbM\x13\x96c\x8b\xa7|*"\\\'^$@#!(){}?+ ~` '' raised of class '<class 'OSError'>':
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 384, in error_check_Main
    Main(arglist)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 406, in Main
    take_action(rps)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 342, in take_action
    Backup(rps[0], rps[1])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 431, in Backup
    backup.Mirror(rpin, rpout)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 39, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 264, in patch
    ITR(diff.index, diff)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rorpiter.py", line 316, in __call__
    last_branch.fast_process(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 577, in fast_process
    self.CCPP.get_rorps(index), self.basis_root_rp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 231, in get_mirror_inc_rps
    mirror_rp, alt_mirror, alt_inc = mir_triple_new(new_rorp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 195, in mir_triple_new
    mirror_rp = check_new_index(mirror_root, new_rorp.index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 140, in check_new_index
    rp = wrap_call(base.new_index, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 126, in wrap_call
    result = func(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1378, in new_index
    return self.__class__(self.conn, self.base, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/FilenameMapping.py", line 147, in __init__
    super().__init__(connection, base, index, data)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1041, in __init__
    if data is None: self.setdata()
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1067, in setdata
    self.data = self.conn.rpath.make_file_dict(self.path)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 355, in make_file_dict
    statblock = os.lstat(filename)

Traceback (most recent call last):
  File "/home/myuser/Public/rdiff-backup/build/scripts-3.7/rdiff-backup", line 32, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 384, in error_check_Main
    Main(arglist)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 406, in Main
    take_action(rps)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 342, in take_action
    Backup(rps[0], rps[1])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 431, in Backup
    backup.Mirror(rpin, rpout)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 39, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 264, in patch
    ITR(diff.index, diff)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rorpiter.py", line 316, in __call__
    last_branch.fast_process(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 577, in fast_process
    self.CCPP.get_rorps(index), self.basis_root_rp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 231, in get_mirror_inc_rps
    mirror_rp, alt_mirror, alt_inc = mir_triple_new(new_rorp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 195, in mir_triple_new
    mirror_rp = check_new_index(mirror_root, new_rorp.index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 140, in check_new_index
    rp = wrap_call(base.new_index, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 126, in wrap_call
    result = func(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1378, in new_index
    return self.__class__(self.conn, self.base, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/FilenameMapping.py", line 147, in __init__
    super().__init__(connection, base, index, data)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1041, in __init__
    if data is None: self.setdata()
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1067, in setdata
    self.data = self.conn.rpath.make_file_dict(self.path)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 355, in make_file_dict
    statblock = os.lstat(filename)
OSError: [Errno 84] Invalid or incomplete multibyte or wide character: b'/run/media/myuser/MYUSBSTICK/backup2/\xd8\xab\xb1Wb\xae\xc5]\x8a\xbb\x15v*\xf4\x0f!\xf9>\xe2Y\x86\xbb\xab\xdbp\xb0\x84\x13k\x1d\xc2\xf1\xf5e\xa5U\x82\x9aUV\xa0\xf4\xdf4\xba\xfdX\x03\x82\x07s\xce\x9e\x8b\xb34\x04\x9f\x17 \xf4\x8f\xa6\xfa\x97\xab\xd8\xac\xda\x85\xdcKvC\xfa#\x94\x92\x9e\xc9\xb7\xc3_\x0f\x84g\x9aB\x11<=^\xdbM\x13\x96c\x8b\xa7|*"\\\'^$@#!(){}?+ ~` '

I don't try with exFAT but I expect a similar result. It doesn't look like rdiff-backup was ever meant to work from POSIX to non-POSIX conform file systems (it probably works more or less as long as no file is "strange"). It'll be quite an effort to make this kind of thing work reliably, don't hold your breath.

[ikus060]

Dod you test with master or with 1.2.8 ? Cause here, I'm using rdiff-backup 1.2.8 from Windows to Linux all the time. From NTFS or FAT32 to linux debian server.

I've never ran into issues.

[ikus060]

It make me think we should find a way to have integration test like this in our pipeline to start various backup with different filesystem and os for source and destination. I will keep this in mind and think about.

[ericzolf]

Windows to Linux is probably no issue, as it would be NTFS/FAT to ext4 (or other POSIX file system), the other way around is the issue here. Also, I thought that in general, the issue should be addressed together with longnames and other potential issues with filenames (like forbidden characters). Actually, there should be a central mapping function which derives a "valid and unique name for the target file system" for each filename it gets, either the filename itself, or if it's too long, wrong encoding, wrong characters, wrong type, etc... a replacement name with a mapping of some kind.

It's no simple task and requires some re-shuffling of the code, and probably a change in repository format, which I wouldn't like to address in the currently upcoming version. Actually I'd like first to simplify the code, remove some technical debt, before we go into implementing bigger functions. And as ikus060 highlighted, automated testing isn't a small feat either.

[ikus060]

Actually I'd like first to simplify the code, remove some technical debt, before we go into implementing bigger functions.

Completely agreed here. Let improve the current code base without adding too many feature. Let focus on have a robust release first.

[ollytom]

For comparison unison, when copying from a case-sensitive filesystem (e.g. Unix-like) to a case-insensitive one (e.g. NTFS on Windows or APFS on macOS), refuses to synchronise any files which have the same “spelling” but different case. A warning is printed and logged for each. See Case conflicts in “Cross-Platform Synchronization” in the unison documentation.