Error parsing labels #545

HeinrichHartmann · 2018-05-03T09:56:53Z

General informations

system/distribution (with version): 7.2.0 (git master)
offlineimap version (offlineimap -V): offlineimap v7.2.0, imaplib2 v2.57 (bundled), Python v2.7.10, OpenSSL 0.9.8zh 14 Jan 2016
Python version: Python 2.7.10
server name or domain:
CLI options: -a Gmail-Private

Configuration file offlineimaprc

...

Logs, error

 ERROR: ERROR in syncfolder for Gmail-Private folder [Gmail]/All Mail: Traceback (most recent call last):
  File "/Users/hartmann/git/offlineimap/offlineimap/accounts.py", line 633, in syncfolder
    remotefolder.cachemessagelist()
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Gmail.py", line 168, in cachemessagelist
    labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 162, in imapsplit
    (quoted, rest) = __split_quoted(workstr)
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 276, in __split_quoted
    raise ValueError("can't find ending quote '%s' in '%s'"% (q, s))
ValueError: can't find ending quote '"' in '"Webserver (RW.net'

  can't find ending quote '"' in '"Webserver (RW.net'
Account sync Gmail-Private:
 *** Finished account 'Gmail-Private' in 0:50

Steps to reproduce the error

Hit the above error message when syncing my Gmail account.

Get a Gmail Account
Make a label "Webserver (RW.net)"
Run offlineimap

The text was updated successfully, but these errors were encountered:

HeinrichHartmann · 2018-05-03T10:04:02Z

__split_qouted is called with this string "Webserver (RW.net so the problem seems to be further up.

HeinrichHartmann · 2018-05-03T10:06:44Z

Inserted a few more prints. Here is how imapsplit is called:

imapsplit('X-GM-LABELS ("Webserver (RW.net)" "\\Important") UID 82031 FLAGS ()')
imapsplit('"Webserver (RW.net')

Looks like the parens are not matched correctly.

nicolas33 · 2018-05-03T17:22:37Z

Could you show the string messagestr in folder/Gmail.py in the for loop (line 150), please?

HeinrichHartmann · 2018-05-04T18:59:30Z

This is the value of the offending message str:
121251 (X-GM-LABELS ("Webserver (RW.net)" "\\Inbox" GInbox) UID 275440 FLAGS (\Seen))

For reference, this is hot the full print log looks like:

messagestr:'121251 (X-GM-LABELS ("Webserver (RW.net)" "\\Inbox" GInbox) UID 275440 FLAGS (\Seen))'
imapsplit('X-GM-LABELS ("Webserver (RW.net)" "\\Inbox" GInbox) UID 275440 FLAGS (\Seen)')
imapsplit('"Webserver (RW.net')
__split_qouted('"Webserver (RW.net')

nicolas33 · 2018-05-04T22:53:52Z

Thank you.

The issue stands in folder/Gmail.py:

165                 m = re.search('\(([^\)]*)\)', options['X-GM-LABELS'])
167                     labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])

At line 165, options is {'FLAGS': '(\\Seen)', 'X-GM-LABELS': '("Webserver (RW.net)" "\\Inbox" GInbox)', 'UID': '275440'}

In the console, I get this output:

>>> labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "offlineimap/imaputil.py", line 162, in imapsplit
    (quoted, rest) = __split_quoted(workstr)
  File "offlineimap/imaputil.py", line 276, in __split_quoted
    raise ValueError("can't find ending quote '%s' in '%s'"% (q, s))
ValueError: can't find ending quote '"' in '"Webserver (RW.net'
>>> m.group(1)
'"Webserver (RW.net'

So, our regex fails at matching the first label inside the double quotes. The parenthesis are not offending.
Now, it's time to find the fix.

Github-ref: OfflineIMAP#545 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 · 2018-05-04T23:28:54Z

@HeinrichHartmann Could you please try the above version with a fix?

HeinrichHartmann · 2018-05-07T11:24:02Z

@nicolas33 Sure!
Unfortunately it does not seem to help:

hartmann@HHMB:/Users/hartmann/git/offlineimap git:(4313355)
$ python offlineimap.py -a Gmail-Private                                                                                                                                                                      2018-05-07 11:22:27 [6.338s]
OfflineIMAP 7.2.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.57 (bundled), Python v2.7.15, OpenSSL 1.0.2o  27 Mar 2018
Account sync Gmail-Private:
 *** Processing account Gmail-Private
 Establishing connection to imap.gmail.com:993 (Gmail-Private-remote)
Folder [Gmail]/All Mail [acc: Gmail-Private]:
 Syncing [Gmail]/All Mail: Gmail -> GmailMaildir
 Collecting data from messages on [Gmail]/All Mail
 ERROR: ERROR in syncfolder for Gmail-Private folder [Gmail]/All Mail: Traceback (most recent call last):
  File "/Users/hartmann/git/offlineimap/offlineimap/accounts.py", line 633, in syncfolder
    remotefolder.cachemessagelist()
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Gmail.py", line 169, in cachemessagelist
    labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 162, in imapsplit
    (quoted, rest) = __split_quoted(workstr)
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 276, in __split_quoted
    raise ValueError("can't find ending quote '%s' in '%s'"% (q, s))
ValueError: can't find ending quote '"' in '"Webserver (RW.net'

  can't find ending quote '"' in '"Webserver (RW.net'
Account sync Gmail-Private:
 *** Finished account 'Gmail-Private' in 0:42
ERROR: Exceptions occurred during the run!
ERROR: ERROR in syncfolder for Gmail-Private folder [Gmail]/All Mail: Traceback (most recent call last):
  File "/Users/hartmann/git/offlineimap/offlineimap/accounts.py", line 633, in syncfolder
    remotefolder.cachemessagelist()
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Gmail.py", line 169, in cachemessagelist
    labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 162, in imapsplit
    (quoted, rest) = __split_quoted(workstr)
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 276, in __split_quoted
    raise ValueError("can't find ending quote '%s' in '%s'"% (q, s))
ValueError: can't find ending quote '"' in '"Webserver (RW.net'

  can't find ending quote '"' in '"Webserver (RW.net'
hartmann@HHMB:/Users/hartmann/git/offlineimap git:(4313355)
$ git status                                                                                                                                                                                                 2018-05-07 11:23:16 [42.420s]
HEAD detached at N33/b545-gmail-error-parsing-label
nothing to commit, working tree clean

HeinrichHartmann · 2018-05-07T11:31:01Z

Also I don't see how your fix would help with that string:

String: ("Webserver (RW.net)" "\\Inbox" GInbox) UID 275440 FLAGS (\Seen)
Regexp: '^[(]([^\)]*)[)]'

Matches "Webserver (RW.net ...

Looking for balanced parens would also not fix the issue, since the label can contain arbitrary bracket expressions.
A proper solution would probably need to match the quoted string properly, allowing escaped quotations marks in the middle. Not sure how to do this best.

HeinrichHartmann · 2018-05-07T12:25:00Z

Note that reproducing this is very easy: Just add a label XXX-) to a new email and run offlineimap.

Github-ref: OfflineIMAP#545 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 · 2018-05-07T16:57:04Z

Sorry, for the noise. I've forgot to change the middle of the regexp. Here is the fix that I intend.

HeinrichHartmann · 2018-05-08T12:28:16Z

Looks good. I am still waiting for my full mailbox to be synced up (200K+ Mails), before I can be 100% sure that it worked.

nicolas33 · 2018-05-08T23:33:42Z

Ok, let us know.

HeinrichHartmann · 2018-05-09T19:55:20Z

New error:

Thread 'Copy message from Gmail-Private-remote:[Gmail]/All Mail' terminated with exception:
Traceback (most recent call last):
  File "/Users/hartmann/git/offlineimap/offlineimap/threadutil.py", line 160, in run
    Thread.run(self)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Gmail.py", line 290, in copymessageto
    super(GmailFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Base.py", line 839, in copymessageto
    message = self.getmessage(uid)
  File "/Users/hartmann/git/offlineimap/offlineimap/folder/Gmail.py", line 84, in getmessage
    labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 162, in imapsplit
    (quoted, rest) = __split_quoted(workstr)
  File "/Users/hartmann/git/offlineimap/offlineimap/imaputil.py", line 276, in __split_quoted
    raise ValueError("can't find ending quote '%s' in '%s'"% (q, s))
ValueError: can't find ending quote '"' in '"Webserver (RW.net'

Looks like this regexp is also problematic:

https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/Gmail.py?utf8=%E2%9C%93#L82

Fix ?!

diff --git a/offlineimap/folder/Gmail.py b/offlineimap/folder/Gmail.py
index 51cd635..af0d0b2 100644
--- a/offlineimap/folder/Gmail.py
+++ b/offlineimap/folder/Gmail.py
@@ -79,7 +79,7 @@ class GmailFolder(IMAPFolder):

         # Embed the labels into the message headers
         if self.synclabels:
-            m = re.search('X-GM-LABELS\s*\(([^\)]*)\)', data[0][0])
+            m = re.search('X-GM-LABELS\s*\((.*)\)', data[0][0])
             if m:
                 labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
             else:

Originally-written-by: Heinrich Hartmann <Heinrich@HeinrichHartmann.com> Github-ref: OfflineIMAP#545

nicolas33 · 2018-05-09T22:44:52Z

Your fix looks good. I've changed the backslashes to the brackets that I find more readable and for consistency.

HeinrichHartmann · 2018-05-17T17:23:57Z

Cool. This solves the issue for me. Thanks for fixing it so quickly!

nicolas33 · 2018-05-17T18:18:15Z

Thanks for the feedback.

Reported-by Heinrich Hartmann <Heinrich@HeinrichHartmann.com> Tested-by Heinrich Hartmann <Heinrich@HeinrichHartmann.com> Github-ref: #545 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 added a commit to nicolas33/offlineimap that referenced this issue May 4, 2018

fix attempt

4313355

Github-ref: OfflineIMAP#545 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 added a commit to nicolas33/offlineimap that referenced this issue May 7, 2018

fix attempt

33cd887

Github-ref: OfflineIMAP#545 Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>

nicolas33 added a commit to nicolas33/offlineimap that referenced this issue May 9, 2018

gmail: fix regexp

306f3ab

Originally-written-by: Heinrich Hartmann <Heinrich@HeinrichHartmann.com> Github-ref: OfflineIMAP#545

nicolas33 closed this as completed May 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error parsing labels #545

Error parsing labels #545

HeinrichHartmann commented May 3, 2018

HeinrichHartmann commented May 3, 2018

HeinrichHartmann commented May 3, 2018

nicolas33 commented May 3, 2018

HeinrichHartmann commented May 4, 2018

nicolas33 commented May 4, 2018

nicolas33 commented May 4, 2018

HeinrichHartmann commented May 7, 2018

HeinrichHartmann commented May 7, 2018

HeinrichHartmann commented May 7, 2018

nicolas33 commented May 7, 2018

HeinrichHartmann commented May 8, 2018

nicolas33 commented May 8, 2018

HeinrichHartmann commented May 9, 2018

nicolas33 commented May 9, 2018

HeinrichHartmann commented May 17, 2018

nicolas33 commented May 17, 2018

Error parsing labels #545

Error parsing labels #545

Comments

HeinrichHartmann commented May 3, 2018

General informations

Configuration file offlineimaprc

Logs, error

Steps to reproduce the error

HeinrichHartmann commented May 3, 2018

HeinrichHartmann commented May 3, 2018

nicolas33 commented May 3, 2018

HeinrichHartmann commented May 4, 2018

nicolas33 commented May 4, 2018

nicolas33 commented May 4, 2018

HeinrichHartmann commented May 7, 2018

HeinrichHartmann commented May 7, 2018

HeinrichHartmann commented May 7, 2018

nicolas33 commented May 7, 2018

HeinrichHartmann commented May 8, 2018

nicolas33 commented May 8, 2018

HeinrichHartmann commented May 9, 2018

nicolas33 commented May 9, 2018

HeinrichHartmann commented May 17, 2018

nicolas33 commented May 17, 2018