UnicodeEncodeError when attempting to sync paths that contain surrogates escapes
#230
Comments
|
Uh, that looks like a non-utf-8 character in a file name. Which encoding is your file system system using? You can find out in Python with: import sys
sys.getfilesystemencoding() |
|
This error seems to happen because Python cannot decode the bytes provided for a file path. From their documentation:
Unfortunately, sqlite3 does not like surrogate escapes. |
|
Hello my drive is formatted in f2fs.
Yesterday I made a wrong copy of one folder and this error appeared.
I deleted it but still error appears and Maestral can t sync.
Don't know what to do
When it worked it was great cause maestral is much lighter than official
Dropbox daemon
(I am on manjaro xfce)
…On Thu, 19 Nov 2020, 22:48 SamSchott, ***@***.***> wrote:
Uh, that looks like a non-utf-8 character in a file name. Which encoding
is your file system system using? You can find out in Python with:
import syssys.getfilesystemencoding()
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#230 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGHD3WNN3FMHO6U4BCX5R6DSQWHDTANCNFSM4T27TMOQ>
.
|
|
f2fs is the file system, however the issue seems to be a non-unicode character in one of the filenames. It would be easiest if you find the offending file and rename it. If you set the log level to debug with When you are sure that the problematic file name is gone but the error persists, a brute-force method would be to rebuild the index. You can stop the daemon first and then schedule a rebuild for the next startup with While such filenames will almost certainly be also rejected by Dropbox servers it would still be nice to handle those errors more gracefully. I'll see what I can do for the next release. |
samschott
changed the title
UnicodeEncodeError when querying the database with paths that contain surrogates escapes
samschott
changed the title
UnicodeEncodeError when querying the database with paths that contain surrogates escapesUnicodeEncodeError when attempting to sync paths that contain surrogates escapes
|
Thanks for your reply
I ll try this tomorrow and keep you updated.
Best regards
…On Thu, 19 Nov 2020, 23:29 SamSchott, ***@***.***> wrote:
f2fs is the file system, however the issue seems to be a non-unicode
character in one of the filenames.
It would be easiest if you find the offending file and rename it. If you
set the log level to debug with maestral log level DEBUG and try syncing
again, the log will show which file it last attempted to sync before the
error. You can show the logs with maestral log show -e.
When you are sure that the problematic file name is gone but the error
persists, a brute-force method would be to rebuild the index. You can stop
the daemon first and then schedule a rebuild for the next startup with maestral
rebuild-index.
While such filenames will almost certainly be also rejected by Dropbox
servers it would still be nice to handle those errors more gracefully. I'll
see what I can do for the next release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#230 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGHD3WKLAY2TOUCEWZ57PETSQWL5NANCNFSM4T27TMOQ>
.
|
Log result2020-11-18 23:34:07 maestral.sync INFO: Creating folders... Don t know which file is making trouble.... and i set to maestral log level DEBUG |
|
Hrm, ok, this is more difficult than I thought. How comfortable are you with Python? Could you run the following code snippet and let me know what it prints? from maestral.main import Maestral
from maestral.sync import DirectorySnapshot
m = Maestral()
snapshot = DirectorySnapshot(m.dropbox_path)
for path in snapshot.paths:
try:
path.encode()
except UnicodeEncodeError:
print(path) |
|
Unfortunately not so experienced, I ll need more explanation what to do and
how :-/
…On Fri, 20 Nov 2020, 22:58 SamSchott, ***@***.***> wrote:
Hrm, ok, this is more difficult than I thought. How comfortable are you
with Python? Could you run the following code snippet and let me know what
it prints?
from maestral.main import Maestralfrom maestral.sync import DirectorySnapshot
m = Maestral()snapshot = DirectorySnapshot(m.dropbox_path)
for path in snapshot.paths:
try:
path.encode()
except UnicodeEncodeError:
print(path)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#230 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGHD3WMJAAN3RVAJVEB7E2TSQ3Q7TANCNFSM4T27TMOQ>
.
|
|
Sure. It will be easiest if you create a file
Any file paths that are printed to the terminal are problematic. |
|
the terminal answers this:
Traceback (most recent call last):
File "test.py", line 9, in <module>
path.encode()
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce4' in
position 213: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 11, in <module>
print(path)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce4' in
position 213: surrogates not allowed
|
|
see attached
|
|
Of course 🤦♂️ A path with cannot be encoded cannot be printed to the terminal. Could you try again with the modifications below? from maestral.main import Maestral
from maestral.sync import DirectorySnapshot
m = Maestral()
snapshot = DirectorySnapshot(m.dropbox_path)
for path in snapshot.paths:
try:
path.encode()
except UnicodeEncodeError:
encoded_path = path.encode("utf-8", "surrogateescape")
print(encoded_path) |
|
From terminal:
*File "test.py", line 1 from maestral.main import Maestral from
maestral.sync import DirectorySnapshot m = Maestral() snapshot =
DirectorySnapshot(m.dropbox_path) for path in snapshot.paths: try:
path.encode() except UnicodeEncodeError: encoded_path =
path.encode("utf-8", "surrogateescape") print(encoded_path)
^SyntaxError: invalid syntax*
At least I understood a bit more, one of my file has this character in file
name: � and I need to find it.
The thing is that we are oftenly saving files in czech language and it can
happen that some letters are wrongly written....
Sorry to bother you with this :-)
|
|
Are you sure you copied exactly the Python code which I posted? It looks like there are two print statements in your file...
No worries, this error should have been handled better in the first place. As the user base grows, I encounter more niche cases. |
|
See attached ;-)
|
|
Github does not allow email attachments, unfortunately. Could you post here instead? |
|
Sure, no problem.
I wrote in the file test.py this:
from maestral.main import Maestral from maestral.sync import
DirectorySnapshot m = Maestral() snapshot =
DirectorySnapshot(m.dropbox_path)
for path in snapshot.paths: try: path.encode() except UnicodeEncodeError:
encoded_path = path.encode("utf-8", "surrogateescape") print(encoded_path)
|
|
Ah, ok. You'll need to copy-paste exactly what I posted, including all of the spaces. This is the file which I have used: And it runs without syntax errors. |
|
Awesome, thanks!
I found 3 files with wrong letter, modified their titles and now it seems
it works.
Will keep you updated but actually it looks like Maestral is syncing again
:-)
Many thanks
|
|
Excellent, glad to hear. I'll work on handling such errors more gracefully than a hard crash. |
|
Dear Sam,
If I can, I also found out that the gui crashes very oftenly when I want to
select which files and folders to sync.
At the end it works but gui crashes 2-3 times before it works and I can
choose directories
Just for info
|
|
Is this on Linux? There has been a similar issue on macOS which I just fixed. I’ll look into it.
|
|
I reinstalled this morning my computer and reinstalled Maestral v 1.2.2 on
manjaro Linux.
When you run Maestral GUI and click on preferences, it opens the window
with token and then starts to sync and window crashes, then when you reopen
window to choose which folders to sync, you can choose but no way to close.
The window does not answer.
…On Sat, 21 Nov 2020, 14:08 SamSchott, ***@***.***> wrote:
Is this on Linux? There has been a similar issue on macOS which I just
fixed. I’ll look into it.
On 21 Nov 2020, at 11:19, adrienlourdin ***@***.***> wrote:
Dear Sam,
If I can, I also found out that the gui crashes very oftenly when I want
to
select which files and folders to sync.
At the end it works but gui crashes 2-3 times before it works and I can
choose directories
Just for info
Adrien LOURDIN - Societe Alkastan
Tel: 00 420 732 867 341
Email: ***@***.***
*Sídlo společnosti:*
Na Kopaninách 125
<https://maps.google.com/?q=Na+Kopanin%C3%A1ch+125&entry=gmail&source=g>
251 69 Velké Popovice
*Provozovna (adresa pro doručení):*
Alkastan s.r.o.
Areál NAKO - Ke Mlýnu 19
252 42 Jesenice - Osnice
On Sat, 21 Nov 2020 at 01:20, SamSchott ***@***.***> wrote:
> Excellent, glad to hear. I'll work on handling such errors more
gracefully
> than a hard crash.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#230 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AGHD3WOHLUFIQHP4UUHFSS3SQ4BWJANCNFSM4T27TMOQ>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<
#230 (comment)>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AFAAVUQMR7O44CWB76J2XILSQ6O3JANCNFSM4T27TMOQ>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#230 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGHD3WOAZJJOHEVSMLENBS3SQ63VLANCNFSM4T27TMOQ>
.
|
|
[adrien@adrien-ux360ca ~]$ maestral gui
Unauthorized Memory Access (SIGSEGV) (core dumped)
|
|
Wow thanks a lot!
…On Mon, 23 Nov 2020, 23:12 SamSchott, ***@***.***> wrote:
Closed #230 <#230> via #238
<#238>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#230 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGHD3WNIFAG42CKCCJ2YXEDSRLM27ANCNFSM4T27TMOQ>
.
|
|
Yeah, the issue was a lot more difficult to fix than I had thought. But it should be good now. It'll be rolled out in the next release... |
Daemon os starting but stops syncing due to unexpected error:
Traceback (most recent call last):
File "/home/adrien/.local/lib/python3.8/site-packages/maestral/sync.py", line 3864, in startup_worker
sync.upload_local_changes_while_inactive()
File "/home/adrien/.local/lib/python3.8/site-packages/maestral/sync.py", line 1782, in upload_local_changes_while_inactive
events, local_cursor = self._get_local_changes_while_inactive()
File "/home/adrien/.local/lib/python3.8/site-packages/maestral/sync.py", line 1819, in _get_local_changes_while_inactive
ctime_check = now > stats.st_ctime > self.get_last_sync(dbx_path_lower)
File "/home/adrien/.local/lib/python3.8/site-packages/maestral/sync.py", line 1102, in get_last_sync
res = self._db_session.query(IndexEntry).get(dbx_path.lower())
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1018, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity
return q.one()
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in iter
return self._execute_and_instances(context)
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self.handle_dbapi_exception(
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1514, in handle_dbapi_exception
util.raise(exc_info[1], with_traceback=exc_info[2])
File "/usr/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce4' in position 192: surrogates not allowed
The text was updated successfully, but these errors were encountered: