Fix stripping (japanese) unicode characters #879
Conversation
* Accents are getting removed from Unicode characters to generate similar "words". * Non-latin characters which cannot be processed that way (eg. japanese, greek, russian, etc.) should not be filtered out at all otherwise files are erroneously skipped or detected as dupes if only some characters make it passed the filter. * Starting from an arbitrary unicode codepoint (converted to decimal), above which we know it is pointless to try any sort of processing, we leave the characters as is. * Fix arsenetar#878.
| # HACK We shouldn't ignore non-ascii characters altogether. Any Unicode char | ||
| # above common european characters that cannot be "sanitized" (ie. stripped | ||
| # of their accents, etc.) are preserved as is. The arbitrary limit is | ||
| # obtained from this one: ord("\u037e") GREEK QUESTION MARK | ||
| s = "".join( |
There was a problem hiding this comment.
This is the statement that could be dropped maybe. But this would require some tweak in the test suite as well (fortunately, not a lot, as I have already tried).
|
Last time I looked at the file name comparison based matches it seemed in general unicode was not really supported due to the way the various functions work. Probably worth a look over as likely to be other issues with unicode values within. I'm wondering here if we just remove stripping out characters completely and just allow all unicode characters to pass, things like emoticons in filenames may cause similar issues. |
|
I'm going to merge this, but really this character processing probably needs to be revisited. |
|
I agree. I hope to find some time later to look it over more thoroughly. |
This "stripping of accents" doesn't look right to me. It would be just as good to keep words accentuated and compare them that way.
For example, "ébé.mp3" and "ebe.mp3" appear in the results as "100%" match, which obviously isn't right at all.
This algorithm might be a bit dated now, and perhaps there is some sort of technical debt left from the python2 to python3 migration (since Unicode strings are now the default).
Maybe it could be safe to remove this altogether, I'm not sure. For now I've left it as is, but it might be worth considering.