Fail when renaming with different case on case insensitive file system #11
Comments
|
Hi there, |
|
Sure, almost no problem. But I can't compile it even though I have python2.7-dev and librsync-dev installed (Debian Wheezy). I tried it with python2.6 as well. No success.
|
|
Hrmm, not sure why it isn't working for you. I built it from source on a fresh wheezy container just for testing: root@wheezy-test:~# apt-get install librsync-dev python-dev wget dev@wheezy-test:~$ wget http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-1.3.3.tar.gz dev@wheezy-test:~$ tar zxf rdiff-backup-1.3.3.tar.gz dev@wheezy-test:~/rdiff-backup-1.3.3$ python setup.py build running build running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/TempFile.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/log.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/SetConnections.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Globals.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/longname.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/static.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Hardlink.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/metadata.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/rpath.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/librsync.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/fs_abilities.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/__init__.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/hash.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Main.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/user_group.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/rorpiter.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/eas_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Rdiff.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Time.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/restore.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/FilenameMapping.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/lazy.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/compare.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/statistics.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/win_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/selection.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/backup.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/iterfile.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/manage.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/regress.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/robust.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/Security.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/increment.py -> build/lib.linux-x86_64-2.7/rdiff_backup copying rdiff_backup/connection.py -> build/lib.linux-x86_64-2.7/rdiff_backup running build_ext building 'rdiff_backup.C' extension creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cmodule.c -o build/temp.linux-x86_64-2.7/cmodule.o gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/cmodule.o -o build/lib.linux-x86_64-2.7/rdiff_backup/C.so building 'rdiff_backup._librsync' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _librsyncmodule.c -o build/temp.linux-x86_64-2.7/_librsyncmodule.o gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/_librsyncmodule.o -lrsync -o build/lib.linux-x86_64-2.7/rdiff_backup/_librsync.so running build_scripts creating build/scripts-2.7 copying and adjusting rdiff-backup -> build/scripts-2.7 copying and adjusting rdiff-backup-statistics -> build/scripts-2.7 changing mode of build/scripts-2.7/rdiff-backup from 644 to 755 changing mode of build/scripts-2.7/rdiff-backup-statistics from 644 to 755 dev@wheezy-test:~/rdiff-backup-1.3.3$ which python /usr/bin/python dev@wheezy-test:~/rdiff-backup-1.3.3$ ls -al /usr/bin/python lrwxrwxrwx 1 root root 9 Sep 29 2013 /usr/bin/python -> python2.7 dev@wheezy-test:~/rdiff-backup-1.3.3$ I don't know what you did differently, but it does work building from source on wheezy. |
|
Thank you. That worked. I tried to use the code from the github repository. The directory structure is different and setup.py is in the dist folder. Anyway, I was able to build 1.3.3 and ran my script again on a FAT32 usb stick. Here is the error: |
|
I have just noticed some trouble with rdiff-backup on vfat in linux. The filesystem is not completely case-insensitive when mounted with the option iocharset=utf8. $ stat a rdiff-backup should always assume case insensitive behavior on vfat! If not, you will run into above error, as soon as there are two files with the same lowercase representation. Currently, rdiff-backup mistakes vfat as case sensitive! |
|
OK, quick check on an USB stick with vfat and the result is... baaaad: $ PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7 PATH=$PWD/build/scripts-3.7:$PATH rdiff-backup ../rdiff-backup_testfiles/various_file_types /run/media/myuser/MYUSBSTICK/backup1
Warning: hard linking not supported by filesystem at /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup-data
b'SpecialFileError fifo [Errno 1] Operation not permitted'
b"SpecialFileError symbolic_link [Errno 1] Operation not permitted: b'regular_file' -> b'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.8'"
OSError while renaming /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13 to /run/media/myuser/MYUSBSTICK/backup1/ث�;087b��]��;021v;042�;015!�;062�;089����p��;019k;029���e�;085��;085;086���4��;088;003�;007sΞ��4;004�;023 ����جڅ�;075v;067�#���ɷ�_;015�g�;066;017;060=^�;077;019�c��;124;042;034;092'^$@#!(){};063+ ~`;032
b'UpdateError rdiff-backup.tmp.13 [Errno 22] Invalid argument: b\'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13\' -> b"/run/media/myuser/MYUSBSTICK/backup1/\\xd8\\xab\\xb1;087b\\xae\\xc5]\\x8a\\xbb;021v;042\\xf4;015!\\xf9;062\\xe2;089\\x86\\xbb\\xab\\xdbp\\xb0\\x84;019k;029\\xc2\\xf1\\xf5e\\xa5;085\\x82\\x9a;085;086\\xa0\\xf4\\xdf4\\xba\\xfd;088;003\\x82;007s\\xce\\x9e\\x8b\\xb34;004\\x9f;023 \\xf4\\x8f\\xa6\\xfa\\x97\\xab\\xd8\\xac\\xda\\x85\\xdc;075v;067\\xfa#\\x94\\x92\\x9e\\xc9\\xb7\\xc3_;015\\x84g\\x9a;066;017;060=^\\xdb;077;019\\x96c\\x8b\\xa7;124;042;034;092\'^$@#!(){};063+ ~`;032"'After a I don't try with exFAT but I expect a similar result. It doesn't look like rdiff-backup was ever meant to work from POSIX to non-POSIX conform file systems (it probably works more or less as long as no file is "strange"). It'll be quite an effort to make this kind of thing work reliably, don't hold your breath. |
|
Dod you test with master or with 1.2.8 ? Cause here, I'm using rdiff-backup 1.2.8 from Windows to Linux all the time. From NTFS or FAT32 to linux debian server. I've never ran into issues. |
|
It make me think we should find a way to have integration test like this in our pipeline to start various backup with different filesystem and os for source and destination. I will keep this in mind and think about. |
|
Windows to Linux is probably no issue, as it would be NTFS/FAT to ext4 (or other POSIX file system), the other way around is the issue here. Also, I thought that in general, the issue should be addressed together with longnames and other potential issues with filenames (like forbidden characters). Actually, there should be a central mapping function which derives a "valid and unique name for the target file system" for each filename it gets, either the filename itself, or if it's too long, wrong encoding, wrong characters, wrong type, etc... a replacement name with a mapping of some kind. It's no simple task and requires some re-shuffling of the code, and probably a change in repository format, which I wouldn't like to address in the currently upcoming version. Actually I'd like first to simplify the code, remove some technical debt, before we go into implementing bigger functions. And as ikus060 highlighted, automated testing isn't a small feat either. |
Completely agreed here. Let improve the current code base without adding too many feature. Let focus on have a robust release first. |
|
For comparison unison, when copying from a case-sensitive filesystem (e.g. Unix-like) to a case-insensitive one (e.g. NTFS on Windows or APFS on macOS), refuses to synchronise any files which have the same “spelling” but different case. A warning is printed and logged for each. See Case conflicts in “Cross-Platform Synchronization” in the unison documentation. |
|
The current approach of rdiff-backup, despite bugs, is to map such names to unique valid names and restore the files based on the mapping, which I think is a good approach for a backup tool. |
|
Hello @mklink! The code base has been rewritten for Python 3 and there will be no further releases for the 1.x series in Python 2 anymore. Could you please help our efforts and test again with the latest development version (https://github.com/rdiff-backup/rdiff-backup#installing-the-latest-development-version) if you can still reproduce this and provide us with a test case (steps on how to reproduce the issue) that applies for latest development version? With your help, if we can reproduce this problem still with the 2.x code base if might be fixed it in upcoming 2.x series releases. |
|
Hi @ottok! Thank you for following up. I just did the test again. The latest development version doesn't have instructions for Windows where I encountered the problem but your latest release seems to be up-to-date (7 days old). I downloaded rdiff-backup-1.9.1b0.win32exe.zip from the "3rd beta release before 2.0.0". Is that a 2.0 preview or how do I test this? Anyway, the tested 1.9.1b0 still encounters the same error. Here is how I reproduced it: mkdir s
mkdir d
mkdir d/test
touch d/test/a
./rdiff-backup.exe s d
mv s/test s/Test
./rdiff-backup.exe s dI got the following output: |
|
Hi @ottok! I managed to replicate this issue on my Debian system. I created a FAT container and had the backup source and destination within that filesystem. Unfortunately, I didn't manage to run the latest master because of a signing key problem. Anyway, here is how to re-produce it with the old version: cd /tmp/
fallocate -l 1m disk
/sbin/mkfs.vfat disk
udisksctl loop-setup -f disk
udisksctl mount -b /dev/loop0 # My system did that automatically.
cd /media/$USER/*/ # Select the right one if you have multiple mounts.
mkdir -p src/test
touch src/test/a
mkdir dst
rdiff-backup src dst
mv src/test/ src/Testt
mv src/Testt/ src/Test # You can't move it straight from `test` to `Test` because the FS reckons it's the same.
rdiff-backup src dst
# Exception 'QuotedPath: dst/rdiff-backup-data/increments/test.2020-03-05T09;05816;05839+11;05800.dir
# ... |
|
Hello I encountered the same error on Windows 10 using rdiff-backup 2.0.3. I encountered the same error as mkllnk by renaming a folder: I also tested the issue by renaming a file instead of a folder. This resulted in a different issue which can lead to data loss. Renaming a file to the same name with another case and backup up again will delete the file from the backup: |
ericzolf
changed the title
ericzolf
changed the title
|
Looking back into older issues. Here the problem is making a back-up from case-insensitive to case-insensitive file system. From case-sensitive to case-insensitive works just fine because rdiff-backup then quotes upper-case letters in names. This brings us a workaround to this issue: force the quoting characters with something like The actual solution to the issue is more complicated because it means comparing files in a case insensitive manner and I'm not sure at how many places exactly, so the proper solution will have to wait a bit more until the code has been cleaned-up. Hopefully, it'll then become clearer to me where to best do the changes. |
|
Looking back at this bug, I don't think it'll ever get fixed:
I'm still open to arguments but I would rather document the workaround from my previous comment than spend time on this issue. |
DOC: document how to backup between case aware file systems (VFAT or NTFS), closes #11
DOC: document how to backup between case aware file systems (VFAT or NTFS), closes #11
|
Nice work @ericzolf. I think this makes sense.
It looks like you and others have been very busy over the past couple of years!
I'm not using rdiff-backup but I'd like to say thanks for all your hard work!
|
I'm using rdiff-backup 1.2.8 to backup files on a Windows machine with cygwin. The source is the local hard drive and the backup destination is an external hard drive, both NTFS. Renaming a folder called
archivetoArchivecauses an AssertErrorassert not incrp.lstat(), incrp. This has been reported before: https://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/case-insensitive-filesystems-assert-not-incrp-lstat-100362/Renaming the folder to its original name works after running
rdiff-backup --check-destination-dir "$dst".I wrote a script to reproduce that error:
I ran that script on my linux machine with a FAT32 usb drive:
TMPDIR=/media/C808-E853/ sh rdiff-backup-23.shIt gave me a different error, but the bug is probably related:
The text was updated successfully, but these errors were encountered: