FETCH: Invalid messageset

[tobwen]

problem

When waiting for a specific subject, the mail server throws an error as soon as a mail with the subject comes in. After relaunching the script, the mail gets detected without a problem and the loop starts from scratch (and breaks when a new mail comes in etc.)

code

#!/usr/bin/env python3

import time
from imap_tools import MailBox, A, AND, NOT

class GetOutOfLoop( Exception ):
    pass

with MailBox('x.x.x.x').login('xyz', 'xyz', 'INBOX') as mailbox:
    try:
        t_end = time.time() + 60 * 2
        while time.time() < t_end:
            for msg in mailbox.fetch(AND(subject='Access'), headers_only=True):
                print(msg.subject)
                print(msg.uid)
                mailbox.delete([msg.uid])
                raise GetOutOfLoop
            print('waiting...')
            time.sleep(5)
    except GetOutOfLoop:
        pass

traceback

Traceback (most recent call last):
  File "./get-fortitoken.py", line 13, in <module>
    for msg in mailbox.fetch(AND(subject='Access'), headers_only=True):
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 101, in fetch
    for fetch_item in (self._fetch_in_bulk if bulk else self._fetch_by_one)(nums, message_parts, reverse):  # noqa
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 66, in _fetch_by_one
    fetch_result = self.box.fetch(message_num, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 534, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 1196, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.7/imaplib.py", line 1027, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.error: FETCH command error: BAD [b'Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs).']

[ikvk]

Try implement without raise, is it helps?

[tobwen]

Try implement without raise, is it helps?

No, sorry. I tried it earlier without telling.

current code with the same problem and without try/catch:

#!/usr/bin/env python3

import time
from imap_tools import MailBox, A, AND, NOT

with MailBox('x.x.x.x').login('xyz', 'xyz', 'INBOX') as mailbox:
    t_end = time.time() + 60 * 2
    while time.time() < t_end:
        # time.sleep(30)
        for msg in mailbox.fetch(AND(subject='Access'), limit=1, reverse=True, headers_only=True, miss_defect=False, miss_no_uid=False):
            print(msg.subject)
            print(msg.uid)
            mailbox.delete([msg.uid])
            t_end = 1
            break
        if t_end > 1:
            print('waiting...')
            time.sleep(45)
        else:
            break 

crazy thing

When relaunching the script, it immediately finds the mail and breaks as expected. I also tried waiting the fetch and raising it to 60 seconds.

$ ./runme.py
waiting...
waiting...
waiting...
[ I'm sending a mail with matching subject ]
waiting...
[ mail with matching subject comes in ]
Traceback (most recent call last):
$ ./runme.py
AccessToken: 1234
55
$ ./runme.py
waiting...
waiting...
waiting...
[ I'm sending a mail with matching subject ]
waiting...
[ mail with matching subject comes in ]
Traceback (most recent call last):
$ ./runme.py
AccessToken: 1234
56

further thoughts & questions

• Perhaps IMAP-Idling is needed?
• I should use openssl or telnet to check if there intervals are too short.
• Can a debugging message mode be activated?

[ikvk]

May be this?
jstedfast/MailKit#572 (comment)

[tobwen]

Fetching them all also doesn't help... I need to go low-level now.

#!/usr/bin/env python3

import time
from imap_tools import MailBox

with MailBox('x.x.x.x').login('xyz', 'xyz', 'INBOX') as mailbox:
    while True:
        # also tried: bulk=True
        for msg in mailbox.fetch():
            print(msg.subject)
            mailbox.delete(msg.uid)
        print('waiting...')
        time.sleep(10)

waiting...
waiting...
Traceback (most recent call last):
  File "./get-fortitoken.py", line 8, in <module>
    for msg in mailbox.fetch():
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 103, in fetch
    for fetch_item in (self._fetch_in_bulk if bulk else self._fetch_by_one)(nums, message_parts, reverse):  # noqa
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 68, in _fetch_by_one
    fetch_result = self.box.fetch(message_num, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 534, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 1196, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.7/imaplib.py", line 1027, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.error: FETCH command error: BAD [b'Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs).']

update

I'm in with OpenSSL now. It's a Dovecote-server and look what I just got on an empty mailbox:

A fetch 1 rfc822.header
A BAD Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs).

conclusion

I need to check if there are new mails before fetching. Dovecote seems to throw an error on an empty set.

[tobwen]

oh come on... this ain't logical :)

#!/usr/bin/env python3

import time
from imap_tools import MailBox
with MailBox('x.x.x.x').login('xyz', 'xyz', 'INBOX') as mailbox:
    while len(mailbox.search('SUBJECT Access')) == 0:
        print('waiting...')
        time.sleep(10)

    print(mailbox.search('SUBJECT Access'))
    print(mailbox.folder.status('INBOX'))
    uidnext = mailbox.folder.status('INBOX')['UIDNEXT']
    for msg in mailbox.fetch('UID ' + str(uidnext - 1)):
        print(msg.subject)
        mailbox.delete(msg.uid)

waiting...
waiting...
waiting...
waiting...
['1']
{'MESSAGES': 1, 'RECENT': 0, 'UIDNEXT': 83, 'UIDVALIDITY': 1592072292, 'UNSEEN': 1}
Traceback (most recent call last):
  File "./get-fortitoken.py", line 18, in <module>
    for msg in mailbox.fetch('UID ' + str(uidnext - 1)):
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 103, in fetch
    for fetch_item in (self._fetch_in_bulk if bulk else self._fetch_by_one)(nums, message_parts, reverse):  # noqa
  File "/usr/local/lib/python3.7/dist-packages/imap_tools/mailbox.py", line 68, in _fetch_by_one
    fetch_result = self.box.fetch(message_num, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 534, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python3.7/imaplib.py", line 1196, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.7/imaplib.py", line 1027, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.error: FETCH command error: BAD [b'Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs).'

[ikvk]

Seems problem in Dovecote , try it with anoter mail server and will understand it

[tobwen]

Sorry, I need to make it work with Dovecot. Closing this unsolved.

[ikvk]

Let me know about results

[tobwen]

I'm going for this now:

import email
import imaplib
import time

# login into mail-server
mail = imaplib.IMAP4_SSL(host=HOST, port=993)
mail.login(ACCOUNT, PASSWORD)
mail.select('INBOX')

# loop until a mail with matching subject appears or t_end is reached
t_end = time.time() + 60 * 2
while time.time() < t_end:
    res, data = mail.uid('SEARCH', None, 'NOT DELETED NOT SEEN SUBJECT AuthCode')
    if res == 'OK':
        i = len(data[0].split())
        if i > 0:
            break
        else:
            time.sleep(10)

# quit when no mail has been found
if i == 0:
    print('0000000')
    quit()

# go through all the messages & collect subjects in a dict (timestamp: subject)
messages = {}
for msg in range(i):
    latest_email_uid   = data[0].split()[msg]
    _, email_data      = mail.uid('FETCH', latest_email_uid, '(BODY.PEEK[HEADER.FIELDS (From Subject Date)] RFC822.SIZE)')
    raw_email_string   = email_data[0][1].decode('utf-8')
    email_message      = email.message_from_string(raw_email_string)
    unixtime           = email.utils.mktime_tz(email.utils.parsedate_tz(email_message['Date']))
    messages[unixtime] = str(email.header.make_header(email.header.decode_header(email_message['Subject'])))
    mail.uid('STORE', data[0].split()[msg], '+FLAGS', '(\Deleted)') 

# get the latest subject by latest timestamp (ugly work-around with tuples)
print(sorted(messages.items())[-1][1].split()[1])

# delete flagged mails and close the connection
mail.expunge()
mail.close()
mail.logout()

[ikvk]

This works for me.

    mailbox.folder.set('INBOX')
    while not len(mailbox.search(A(subject='KEY'))):
        print('waiting...{}'.format(datetime.datetime.now()))
        time.sleep(5)
    for msg in mailbox.fetch(reverse=True, limit=1):
        print(msg.subject)

I think the essence of your error is here: 'UID' + str(uidnext-1)

[ziima]

I encountered a similar issue and fixed it by using bulk=True in fetch.

[mightym]

I'm getting this error when the following happens:

I fetch the uids via

>>> mailbox.uids()
['1', '2', '3', '4']

Then I'm sending a new message to my inbox via my local mail client. So the list should be now ['1', '2', '3', '4', '5']
But when I refetch the uids now I get this error:

>>> mailbox.uids()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/admin/.pyenv/versions/test_env/lib/python3.10/site-packages/imap_tools/mailbox.py", line 123, in uids
    fetch_result = self.client.fetch(','.join(nums), "(UID)")
  File "/Users/admin/.pyenv/versions/3.10.3/lib/python3.10/imaplib.py", line 548, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/Users/admin/.pyenv/versions/3.10.3/lib/python3.10/imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/Users/admin/.pyenv/versions/3.10.3/lib/python3.10/imaplib.py", line 1055, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: FETCH command error: BAD [b'Error in IMAP command FETCH: Invalid messageset (0.001 + 0.000 secs).']

If I now reinitialize my mailbox and do mailbox.uids() I get the expected list of ['1', '2', '3', '4', '5']