[BUG] Rule on mail fetching for Hotmail IMAP gives error

[bobvmierlo]

Description

When trying to fetch my Hotmail email attachments using a rule looking for the subject "Paperless", throws the error below in the logs. I read about charset issues with Outlook, so I tried both 'UTF-8' and 'US-ASCII' in the mail account settings. When using 'US-ASCII' I get the 1st error, so I changed to 'UTF-8' which resulted in 2nd error below.

'US-ASCII' error:

[2023-11-26 22:02:54,719] [ERROR] [paperless_mail] Rule Outlook.Paperless Subject: Error while processing rule: UID command error: BAD [b'Command Argument Error. 11']
Traceback (most recent call last):
  File "/opt/paperless/src/paperless_mail/mail.py", line 501, in handle_mail_account
    total_processed_files += self._handle_mail_rule(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/paperless/src/paperless_mail/mail.py", line 568, in _handle_mail_rule
    for message in messages:
  File "/usr/local/lib/python3.11/dist-packages/imap_tools/mailbox.py", line 160, in fetch
    uids = tuple((reversed if reverse else iter)(self.uids(criteria, charset)))[limit_range]
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/imap_tools/mailbox.py", line 119, in uids
    uid_result = self.client.uid('SEARCH', 'CHARSET', charset, encoded_criteria)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/imaplib.py", line 890, in uid
    typ, dat = self._simple_command(name, command, *args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/imaplib.py", line 1055, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: UID command error: BAD [b'Command Argument Error. 11']

'UTF-8' error:

[2023-11-26 21:56:23,375] [ERROR] [paperless_mail] Rule Outlook.Archive Hotmail: Error while processing rule: Response status "OK" expected, but "NO" received. Data: [b'[BADCHARSET (US-ASCII)] The specified charset is not supported.']

Traceback (most recent call last):

  File "/opt/paperless/src/paperless_mail/mail.py", line 501, in handle_mail_account

    total_processed_files += self._handle_mail_rule(

                             ^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/paperless/src/paperless_mail/mail.py", line 568, in _handle_mail_rule

    for message in messages:

  File "/usr/local/lib/python3.11/dist-packages/imap_tools/mailbox.py", line 160, in fetch

    uids = tuple((reversed if reverse else iter)(self.uids(criteria, charset)))[limit_range]

                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/dist-packages/imap_tools/mailbox.py", line 120, in uids

    check_command_status(uid_result, MailboxUidsError)

  File "/usr/local/lib/python3.11/dist-packages/imap_tools/utils.py", line 46, in check_command_status

    raise exception(command_result=command_result, expected=expected)

imap_tools.errors.MailboxUidsError: Response status "OK" expected, but "NO" received. Data: [b'[BADCHARSET (US-ASCII)] The specified charset is not supported.']

Steps to reproduce

1. Configured Hotmail account using 'app password', because basic auth is deprecated by Microsoft
2. Tested connection, no errors so far
3. Tried importing mail based on filter, errors above happened
4. Tried both charset's, both give an error

Webserver logs

See description

Browser logs

No response

Paperless-ngx version

1.17.4

Host OS

Debian 12 as LXC on Proxmox

Installation method

Other (please describe above)

Browser

No response

Configuration changes

No response

Other

Installed using the LXC script from: https://tteck.github.io/Proxmox/. I believe this is a bare-metal install.

[stumpylog]

Can't say I see this as our issue. We're using imsp_tools, which is where the error originates. But even that's probably not their fault, since it goes deeper into the Python standard library.

Basically, it's probably a configuration problem, either with your account or the server.

[stumpylog]

You're encountering ikvk/imap_tools#181. So it seems you won't be able to use all the options to search. It's just a wonderful quirk of Microsoft I guess.

[tobiaswx]

I understand that the fault does not lie with you (the Paperless-ngx developers). But unfortunately neither this, nor the linked issue from imap_tools, answers what we as users of the Paperless-ngx application should do specifically to work around the problem. In my opinion, as the developer of the application, the least you can do is to inform the users of this or at least explicitly say "Oh, by the way, we don't support provider xyz for certain reasons" in a place that can be found (or when answering an issue).

[stumpylog]

There is no workaround. Hotmail has chosen to not support some portion of IMAP searching.

Feel free to create a wiki page for issue with providers, but there are simply too many providers with too many quirks. I'm not going to take that much work on.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns.