Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Unable to translate Unicode character \uDC63 at index 72 to specified code page. #480

Closed
kenkendk opened this issue Aug 5, 2014 · 15 comments

Comments

@kenkendk
Copy link
Member

kenkendk commented Aug 5, 2014

From lou...@gmail.com on October 03, 2011 16:02:55

What steps will reproduce the problem? 1. Attempted to create my first back up of an entire drive
2. Used the backup wizard and set to back up to an external HD
3. Error occured while compiling file list, no files backed up. What is the expected output? What do you see instead? I expected a backup to occur, instead I got this message:

---BEGIN message---

Error: Unable to translate Unicode character \uDC63 at index 72 to specified code page.

Error: System.Text.EncoderFallbackException: Unable to translate Unicode character \uDC63 at index 72 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char_& chars)
at System.Text.UTF8Encoding.GetBytes(Char_ chars, Int32 charCount, Byte* bytes, Int32 byteCount, EncoderNLS baseEncoder)
at System.Text.EncoderNLS.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, Boolean flush)
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Dispose(Boolean disposing)
at System.IO.TextWriter.Dispose()
at Duplicati.Library.Compression.FileArchiveZip.WriteAllLines(String file, String[] data)
at Duplicati.Library.Main.RSync.RSyncDir.MakeMultiPassDiff(ICompression signaturefile, ICompression contentfile, Int64 volumesize)
at Duplicati.Library.Main.Interface.Backup(String[] sources)
at Duplicati.GUI.DuplicatiRunner.ExecuteTask(IDuplicityTask task)
Cleanup output:

---END message--- What version of the product are you using? On what operating system? Using version 1.2.2 win 64, on winXP 64 bit edition. (latest patches installed) What backend (destination) are you using? External HD, NTFS formatted. Please provide any additional information below. No additional information.

Original issue: http://code.google.com/p/duplicati/issues/detail?id=480

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on October 03, 2011 12:42:27

Ok, it seems that some filename is not representable as utf-8. Do you have any idea what that file/folder is called?

Status: Accepted
Owner: kenneth@hexad.dk

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From lou...@gmail.com on October 03, 2011 13:34:28

How do I find out what file it choked on? or what folder it was in when it choked?

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on October 04, 2011 01:28:47

I was hoping that would be obvious to you. It appear to happen when Duplicati is writing a list of included folders. Unfortunately I cannot display that character on my system. Could you try to go to: http://www.sjsu.edu/faculty/watkins/unicodeviewer2.htm Enter the value "DC63" in the first field, and the press submit. It will show you what character that it fails on. This will allow you to work around the problem, but it should be fixed.

Unfortunately it seems that the problem is a bit deeper than I had hoped: http://wcfpro.wordpress.com/2011/01/11/encoderfallbackexception-when-passing-utf8-characters/

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From lou...@gmail.com on October 04, 2011 04:52:58

Well I'm trying to back up an entire volume which contains many folders and many files. I'm not sure how I can even search for the offending character because it is a surrogate character.

would this help? http://cygwin.com/ml/cygwin-developers/2009-09/msg00067.html

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on October 05, 2011 00:51:53

I have no quick solution for this, but I would assume that other programs would have trouble with that folder as well?

My only quick-fix solution is to try backing up each of the root folders separately until you find the one that makes trouble, then repeat with the subfolders until you locate it.

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From lou...@gmail.com on October 05, 2011 05:00:44

I will follow your suggestion and find out what the offending file is. I'll report back when I find it.

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From lou...@gmail.com on October 05, 2011 09:36:31

I found the culprit:
The drive I was trying to backup had a hidden 'RECYCLER' folder in it. (the windows recycle bin for that drive)

In the recycler folder was this strange folder name that does not even appear in windows explorer, I can only see it when browsing the folder with the dos box. (see attached image)

I was able to rmdir the offending folder. and it worked.

Attachment: badfilename.jpg

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on October 06, 2011 00:45:17

Good detective work!
So it was a file Windows Explorer did not like.
I'll put this at low priority as you immediate problem was fixed.
Duplicati should probably just ignore files/folders like that.

Labels: -Priority-Medium Priority-Low

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From daniel.g...@gmail.com on July 12, 2012 16:04:48

Another thing: I got the same error and don't know which file is causing the issue.

I just downloaded the source to fix the bug by myself BUT the itneresting thing: The error does only appear with the release version, the debug version runs fine. :-(

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on July 13, 2012 01:49:22

Can you reproduce if you compile with the Release configuration?

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From daniel.g...@gmail.com on July 13, 2012 02:04:17

As I said: When i choose "Release" configuration in the solution explorer, the crash happens. When I select "Debug" as configuration, it does NOT crash.

Do you mean something different with "Release configuration"? :-)

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on July 13, 2012 02:57:32

I assumed you meant "Release" as in the build from the "Downloads" page, so you could not reproduce when building on your own machine :).

You should be able to set a breakpoint, even if you are in "Release" mode? https://code.google.com/p/duplicati/source/browse/branches/1.3.x/Duplicati/Library/Compression/FileArchiveZip.cs#321 Looking at that line now, I realize that it does not seem to enforce UTF-8 encoding as it should.
You should be able to change it to:
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(CreateFile(file), System.Text.Encoding.Utf8))

And it should now work.

If this fixes the issue, we should also change: https://code.google.com/p/duplicati/source/browse/branches/1.3.x/Duplicati/Library/Compression/FileArchiveZip.cs#266 So it assumes UTF-8 encoding, but I am not a 100% sure that is backwards compatible.

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From daniel.g...@gmail.com on July 13, 2012 03:29:06

Thank you, your patch works perfectly!

Using "Encoding.UTF8" (big letters "UTF") did the trick.

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From kenneth@hexad.dk on July 14, 2012 03:36:57

This issue was updated by revision r1358 .

Changed to correctly encode filenames as UTF-8 in metadata files.

Status: Fixed

@kenkendk
Copy link
Member Author

kenkendk commented Aug 5, 2014

From rst...@gmail.com on July 31, 2012 13:40:53

The changes are now available in the latest development snapshot: https://code.google.com/p/duplicati/wiki/Downloads?tm=2

@kenkendk kenkendk closed this as completed Aug 5, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant