IMAP import from gmail isn't working

[DrVanSteiner]

Description
IMAP import from gmail isn't working in Docker environment running version 2.0.0rc45 and before.

Expected
Attachements of Emails get imported

Actual
Papermerge can connect the IMAP Server and reports the count of UNSEEN messages but afterwards nothing happens not even an error. (Even with logging enabled as described here: #118 (comment)) i see only the following in the logfile:

MAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 2 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 2 count
IMAP Import: UNSEEN messages 2 count

Configuration of IMAP:
IMPORT_MAIL_HOST = "imap.gmail.com"
IMPORT_MAIL_USER = "blabla"
IMPORT_MAIL_PASS = "blabla"
IMPORT_MAIL_BY_USER = False
IMPORT_MAIL_BY_SECRET = False
IMPORT_MAIL_DELETE = False

(Importing from local folder is working...)

Info:

• Papermerge Version [2.0.0rc45

[DrVanSteiner]

What i see so far is that contains_attachments always returns false.

[ciur]

@DrVanSteiner, can you please check if release 2.0.0rc48 fixes this issue ?

[DrVanSteiner]

No it's not fixed with that version. The worker still only reports the number of unseen messages and nothing happens.

[l4rm4nd]

Can confirm the issue.

Mails are correctly identified, but not imported. Should be a bug in import_attachment() of papermerge-core's imap.py.

Did some basic testing.

• IMAP login, inbox selection and e-mail fetching work without an issue:

from papermerge.core.importers.imap import *
imap_client = login("imap.gmail.com","test@gmail.com","MySecurePassword")
imap_client = select_inbox(imap_client, "INBOX")
messages = imap_client.search(['UNSEEN'])
print("IMAP Import: UNSEEN messages %s" % (messages))

• However, fetched e-mails don't seem to have an attachement. This is weird, since my sent e-mails clearly have!

messages_structure = imap_client.fetch(messages, ['BODYSTRUCTURE'])
for uid, structure in messages_structure.items():
  if not contains_attachments(structure[b'BODYSTRUCTURE']):
    print('no attach - weird, I did sent a file!')

@ciur: So this seems to be a bug in your validation contains_attachments(). The mails are removed from the messages object and therefore not processed - although having an attachement!

Note: The IMAPClient library provides a special gmail search function. Maybe we can utilize this instead of contains_attachments() for gmail specifically to fix the issue? Haven't had a look at your contains_attachments() implementation yet.

# This method only works for IMAP servers that support X-GM-RAW, which is only likely to be Gmail
messages = imap_client.gmail_search("has:attachment in:unread")

[ciur]

@l4rm4nd, thanks for investigation; I will have a look as well.

[ciur]

Maybe we can utilize this instead of contains_attachments() for gmail specifically to fix the issue? Haven't had a look at your contains_attachments() implementation yet.

No. By adding gmail, gmx... etc specific code is simple no go. Doing so, long term will lead to very brittle code base.

I think the problem is in contains_attachments method - it just looks wrong and difficult to understand how it does what it is supposed to do.

I will refactor contains_attachments function which hopefully will fix this issue.

[ciur]

Fix is now part of version 2.0

[guim31]

I post here because i configured my papermerge.conf like this :

IMPORT_MAIL_HOST ="imap.gmail.com"
IMPORT_MAIL_USER ="gu@gmail.com"
IMPORT_MAIL_PASS ="xxxx"
IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com"

The thing is : ALL my mails with attachments are imported by papermerge... Not only the ones coming from FromBrotherDevice@brother.com

Is there any way to fix this ?

[citizen-17]

IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com"

You have to set IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com" to True, IMPORT_MAIL_BY_USER = True and then configure the rest in the papermerge webui