Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Linux] Support non-utf8 locale for filename #5719

Closed
guruz opened this issue Apr 21, 2017 · 3 comments
Closed

[Linux] Support non-utf8 locale for filename #5719

guruz opened this issue Apr 21, 2017 · 3 comments
Assignees
@ogoffart
Labels
Discussion
Milestone
2.5.0

Comments

@guruz
Copy link
Contributor

guruz commented Apr 21, 2017

FYI @ogoffart @DeepDiver1975

Related #5676 [Feature] Filename encoding with gbk standard should be supported

<ogoffart> There is no concept of fs encoding on linux
<ogoffart> Everything has to be encoded with the locale
<jturcotte> You'll also have to deal with chars that you can't convert to his encoding, and you want to make sure that if the user modifies that file that it's sent back to the server with the right name.
<jturcotte> And if the user sees blah???.png weird characters in the file, maybe he'll feel like he has to rename it too, so it shouldn't look too weird.

Challenges: How to handle invalid encodings?
#4811 (comment)
@ogoffart ogoffart changed the title Convert local file name encoding to utf8 on server? [Linux] Support non-utf8 locale for filename Apr 22, 2017
@ogoffart
Copy link
Contributor

(I changed the title because there is no need to involve the server, it's purely a client issue)

How to handle invalid encoding?: The way we currently do it in SyncEngine::treewalkFile, for example.

client/src/libsync/syncengine.cpp

Line 364 in e86499d

qWarning() << "File ignored because of invalid utf-8 sequence: " << file->path;

(however this has to happen at the time of conversion from the system locale, which has to happen in csync, before the reconcile phase.)

@ogoffart ogoffart mentioned this issue Jul 4, 2017
Merged
@guruz guruz added this to the 2.4.0 milestone Jul 4, 2017
ogoffart added a commit that referenced this issue Jul 4, 2017
@ogoffart
csync: Use Qt for encodeing/decoding filesystem strings
b040bd4 
Issues:
 - #5661   On mac, iconv did not support all of unicode and some
           files with emoji in the filename could not be uploaded

 - #5719 , #5676  On linux, we will now support non utf-8 locale
ogoffart added a commit that referenced this issue Jul 13, 2017
@ogoffart
csync: Use Qt for encodeing/decoding filesystem strings
acf65b4 
Issues:
 - #5661   On mac, iconv did not support all of unicode and some
           files with emoji in the filename could not be uploaded

 - #5719 , #5676  On linux, we will now support non utf-8 locale
@ogoffart ogoffart added the ReadyToTest QA, please validate the fix/enhancement label Jul 17, 2017
@guruz
Copy link
Contributor Author

guruz commented Nov 6, 2017

Moving to 2.5 as #5676 was moved too

@guruz guruz modified the milestones: 2.4.0, 2.5.0 Nov 6, 2017
@guruz guruz removed the ReadyToTest QA, please validate the fix/enhancement label Nov 20, 2017
@ogoffart
Copy link
Contributor

Duplicate of #5676

@ogoffart ogoffart marked this as a duplicate of #5676 Nov 21, 2017
ogoffart added a commit that referenced this issue Nov 21, 2017
@ogoffart
csync_update: ignore remote files that cannot be encoded
967eb0f 
This is mainly for linux, whose local is not UTF-8.
For example, in latin1, it is not possible to encode emoji or chinese character.
If there are such character in the filename, Qt would just save the file using
the replacement character ('?'). Then, on the next sync, client would rename
the files using this replacement character.

Avoid this by ignoring the files which cannot be downloaded because the
filename cannot be represented with the user's locale

Relates to issue #5676 and #5719
@ogoffart ogoffart mentioned this issue Nov 21, 2017
Merged
ogoffart added a commit that referenced this issue Nov 23, 2017
@ogoffart
csync_update: ignore remote files that cannot be encoded
72809ef 
This is mainly for linux, whose local is not UTF-8.
For example, in latin1, it is not possible to encode emoji or chinese character.
If there are such character in the filename, Qt would just save the file using
the replacement character ('?'). Then, on the next sync, client would rename
the files using this replacement character.

Avoid this by ignoring the files which cannot be downloaded because the
filename cannot be represented with the user's locale

Relates to issue #5676 and #5719
ckamm added a commit that referenced this issue Jan 10, 2018
@ckamm
Ignore files that can't be encoded for the filesystem
e50b023 
There's an upstream bug where QTextCodec::canEncode returns true even
though it should be false. This works around that issue and adds a test.

The original work was done in 72809ef

See #6287, #5676, #5719
See https://bugreports.qt.io/browse/QTBUG-6925
@ckamm ckamm mentioned this issue Jan 10, 2018
Merged
ckamm added a commit that referenced this issue Jan 10, 2018
@ckamm
Ignore files that can't be encoded for the filesystem
25f7dba 
There's an upstream bug where QTextCodec::canEncode returns true even
though it should be false. This works around that issue and adds a test.

The original work was done in 72809ef

See #6287, #5676, #5719
See https://bugreports.qt.io/browse/QTBUG-6925
ckamm added a commit that referenced this issue Jan 10, 2018
@ckamm
Ignore files that can't be encoded for the filesystem
7d70f1b 
There's an upstream bug where QTextCodec::canEncode returns true even
though it should be false. This works around that issue and adds a test.

The original work was done in 72809ef

See #6287, #5676, #5719
See https://bugreports.qt.io/browse/QTBUG-6925
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ogoffart ogoffart

Labels
Discussion
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
2 participants
@ogoffart @guruz