[BUG] Incompatibility with mail server that does not seem to support CHARSET option in SEARCH #6815

rqi14 · 2024-05-23T09:28:32Z

Description

Similar to #4685

I figured out using imaplib and ipython that if not specifying CHARSET in imap.uid command, my mailbox works fine' If CHARSET is included in the command, no matter what it is set to (UTF-8/US-ASCII), it reports error.

{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"OK","url":"https://paperless.example.com/api/mail_accounts/1/","ok":false,"name":"HttpErrorResponse","message":"Http failure response for https://paperless.example.com/api/mail_accounts/1/: 400 OK","error":{"character_set":["该字段不能为空。"]}}

It is not obvious why this charset option is needed in setting up the email function. I tried to remove it but the settngs in paperless won't let me do it.

If I run the following code it works.

import imaplib
import logging

logging.basicConfig(level=logging.DEBUG)

def test_imap_command():
    mail_server = 'imap.qiye.aliyun.com'
    mail_account = 'paperless-consumption@example.com'
    password = 'example'  # Be sure to use a secure way to handle passwords in producti
on

    try:
        imap = imaplib.IMAP4_SSL(mail_server)
        imap.login(mail_account, password)
        imap.select('INBOX')  # Selecting the inbox
        typ, data = imap.uid('SEARCH', 'UNSEEN')
        if typ == 'OK':
            print("UID SEARCH without CHARSET successful:", data)
        else:
            print("UID SEARCH without CHARSET  failed:", data)
    except Exception as e:
        logging.error(f"Error during UID SEARCH without UTF-8: {e}", exc_info=True)
    finally:
        imap.logout()

test_imap_command()

UID SEARCH without CHARSET successful: [b'']

If I run the following, there is an error

import imaplib
import logging

logging.basicConfig(level=logging.DEBUG)

def test_imap_command():
    mail_server = 'imap.qiye.aliyun.com'
    mail_account = 'paperless-consumption@example.com'
    password = 'example' 

    try:
        imap = imaplib.IMAP4_SSL(mail_server)
        imap.login(mail_account, password)
        imap.select('INBOX')  # Selecting the inbox
        typ, data = imap.uid('SEARCH', 'CHARSET', 'UTF-8', 'UNSEEN')
        if typ == 'OK':
            print("UID SEARCH with UTF-8 successful:", data)
        else:
            print("UID SEARCH with UTF-8 failed:", data)
    except Exception as e:
        logging.error(f"Error during UID SEARCH with UTF-8: {e}", exc_info=True)
    finally:
        imap.logout()

test_imap_command()

ERROR:root:Error during UID SEARCH with UTF-8: UID command error: BAD [b'invalid command or parameters']
Traceback (most recent call last):
  File "<ipython-input-22-d16d34dc6a66>", line 15, in test_imap_command
    typ, data = imap.uid('SEARCH', 'CHARSET', 'UTF-8', 'UNSEEN')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\imaplib.py", line 890, in uid
    typ, dat = self._simple_command(name, command, *args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\imaplib.py", line 1055, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: UID command error: BAD [b'invalid command or parameters']

Steps to reproduce

set up mail with aliyun email
set up a mail rule
see error

Webserver logs

[2024-05-23 17:20:00,063] [DEBUG] [paperless_mail] Processing mail account paperless-consumption@example.com

[2024-05-23 17:20:00,200] [DEBUG] [paperless_mail] GMAIL Label Support: False

[2024-05-23 17:20:00,201] [DEBUG] [paperless_mail] AUTH=PLAIN Support: False

[2024-05-23 17:20:00,649] [DEBUG] [paperless_mail] Account paperless-consumption@example.com: Processing 1 rule(s)

[2024-05-23 17:20:00,655] [DEBUG] [paperless_mail] Rule paperless-consumption@example.com.mail inbox receiving attachment: Selecting folder INBOX

[2024-05-23 17:20:00,671] [DEBUG] [paperless_mail] Rule paperless-consumption@example.com.mail inbox receiving attachment: Searching folder with criteria (SINCE 20-May-2024 UNSEEN)

[2024-05-23 17:20:00,687] [ERROR] [paperless_mail] Rule paperless-consumption@example.com.mail inbox receiving attachment: Error while processing rule: UID command error: BAD [b'invalid command or parameters']

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 518, in handle_mail_account

    total_processed_files += self._handle_mail_rule(

                             ^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 586, in _handle_mail_rule

    for message in messages:

  File "/usr/local/lib/python3.11/site-packages/imap_tools/mailbox.py", line 182, in fetch

    uids = tuple((reversed if reverse else iter)(self.uids(criteria, charset, sort)))[limit_range]

                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/imap_tools/mailbox.py", line 126, in uids

    uid_result = self.client.uid('SEARCH', 'CHARSET', charset, encoded_criteria)  # *charset are opt here

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/imaplib.py", line 890, in uid

    typ, dat = self._simple_command(name, command, *args)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/imaplib.py", line 1230, in _simple_command

    return self._command_complete(name, self._command(name, *args))

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/imaplib.py", line 1055, in _command_complete

    raise self.error('%s command error: %s %s' % (name, typ, data))

imaplib.IMAP4.error: UID command error: BAD [b'invalid command or parameters']

Browser logs

No response

Paperless-ngx version

2.8.6

Host OS

ubuntu 23.04

Installation method

Docker - official image

System status

No response

Browser

No response

Configuration changes

No response

Please confirm the following

I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
I have already searched for relevant existing issues and discussions before opening this report.
I have updated the title field above with a concise description.

The text was updated successfully, but these errors were encountered:

stumpylog · 2024-05-23T13:07:51Z

All the imap code is through imap-tools, please report this upstream for any resolution.

rqi14 · 2024-05-23T13:31:34Z

All the imap code is through imap-tools, please report this upstream for any resolution.

Hi. Maybe I didn't express myself clearly. The problem here is paperless forcing using the CHARSET option. This option is not supported by some of the email providers. imap-tools is totally fine if paperless allows sending requests without CHARSET. There is nothing wrong with it.

Paperless does not allow configuring an email account without specifying this CHARSET option. I tried with imap-tools and found that at least for my email provider, this option is completely unnecessary and only causes trouble. To be compatible with a larger range of email providers, maybe paperless should allow setting up email account without this CHARSET option.

stumpylog · 2024-05-23T13:38:33Z

imap-tools always sets a charset:
https://github.com/ikvk/imap_tools/blob/7b29453d39fae768a768619569e7096d154d3ed3/imap_tools/mailbox.py#L158

Support for none would have to start there.

rqi14 · 2024-05-23T13:45:40Z

imap-tools always sets a charset: https://github.com/ikvk/imap_tools/blob/7b29453d39fae768a768619569e7096d154d3ed3/imap_tools/mailbox.py#L158

Support for none would have to start there.

I don't think this is the correct function to look at. The error message from paperless indicates the problem is with the uid function not the fetch function.

I tested this uid function with CHARSET=US-ASCII, which is the default arugment value specified in the function you mentioned. Obviously, the result is different from not specifying CHARSET at all

In [1]: import imaplib
   ...: import logging
   ...:
   ...: logging.basicConfig(level=logging.DEBUG)
   ...:
   ...: def test_imap_command():
   ...:     mail_server = 'imap.qiye.aliyun.com'
   ...:     mail_account = 'paperless-consumption@example.com'
   ...:     password = 'example'
   ...:
   ...:     try:
   ...:         imap = imaplib.IMAP4_SSL(mail_server)
   ...:         imap.login(mail_account, password)
   ...:         imap.select('INBOX')  # Selecting the inbox
   ...:         typ, data = imap.uid('SEARCH', 'CHARSET', 'US-ASCII', 'UNSEEN')
   ...:         if typ == 'OK':
   ...:             print("UID SEARCH with US-ASCII successful:", data)
   ...:         else:
   ...:             print("UID SEARCH with US-ASCII failed:", data)
   ...:     except Exception as e:
   ...:         logging.error(f"Error during UID SEARCH with US-ASCII: {e}", exc_info=True)
   ...:     finally:
   ...:         imap.logout()
   ...:
   ...: test_imap_command()
ERROR:root:Error during UID SEARCH with US-ASCII: UID command error: BAD [b'invalid command or parameters']
Traceback (most recent call last):
  File "<ipython-input-1-2d9d94bfca98>", line 15, in test_imap_command
    typ, data = imap.uid('SEARCH', 'CHARSET', 'US-ASCII', 'UNSEEN')
  File "C:\Program Files\Python310\lib\imaplib.py", line 890, in uid
    typ, dat = self._simple_command(name, command, *args)
  File "C:\Program Files\Python310\lib\imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "C:\Program Files\Python310\lib\imaplib.py", line 1055, in _command_complete
    raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: UID command error: BAD [b'invalid command or parameters']

In [2]: import imaplib
   ...: import logging
   ...:
   ...: logging.basicConfig(level=logging.DEBUG)
   ...:
   ...: def test_imap_command():
   ...:     mail_server = 'imap.qiye.aliyun.com'
   ...:     mail_account = 'paperless-consumption@example.com'
   ...:     password = 'example'
   ...:
   ...:     try:
   ...:         imap = imaplib.IMAP4_SSL(mail_server)
   ...:         imap.login(mail_account, password)
   ...:         imap.select('INBOX')  # Selecting the inbox
   ...:         typ, data = imap.uid('SEARCH', 'UNSEEN')
   ...:         if typ == 'OK':
   ...:             print("UID SEARCH without CHARSET successful:", data)
   ...:         else:
   ...:             print("UID SEARCH without CHARSET failed:", data)
   ...:     except Exception as e:
   ...:         logging.error(f"Error during UID SEARCH without CHARSET: {e}", exc_info=True)
   ...:     finally:
   ...:         imap.logout()
   ...:
   ...: test_imap_command()
UID SEARCH without CHARSET successful: [b'']

rqi14 · 2024-05-23T14:47:40Z

imap-tools always sets a charset: https://github.com/ikvk/imap_tools/blob/7b29453d39fae768a768619569e7096d154d3ed3/imap_tools/mailbox.py#L158

Support for none would have to start there.

You are right. Imap-tools needs to support omitting CHARSET first. Simply changing the UI and function calling in paperless is not sufficient.

rqi14 added bug Bug report or a Bug-fix unconfirmed labels May 23, 2024

stumpylog closed this as not planned Won't fix, can't repro, duplicate, stale May 23, 2024

stumpylog added dependencies Pull requests that update a dependency file not a bug not a bug in paperless-ngx and removed bug Bug report or a Bug-fix unconfirmed labels May 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Incompatibility with mail server that does not seem to support CHARSET option in SEARCH #6815

[BUG] Incompatibility with mail server that does not seem to support CHARSET option in SEARCH #6815

rqi14 commented May 23, 2024 •

edited

stumpylog commented May 23, 2024

rqi14 commented May 23, 2024 •

edited

stumpylog commented May 23, 2024

rqi14 commented May 23, 2024

rqi14 commented May 23, 2024

[BUG] Incompatibility with mail server that does not seem to support CHARSET option in SEARCH #6815

[BUG] Incompatibility with mail server that does not seem to support CHARSET option in SEARCH #6815

Comments

rqi14 commented May 23, 2024 • edited

Description

Steps to reproduce

Webserver logs

Browser logs

Paperless-ngx version

Host OS

Installation method

System status

Browser

Configuration changes

Please confirm the following

stumpylog commented May 23, 2024

rqi14 commented May 23, 2024 • edited

stumpylog commented May 23, 2024

rqi14 commented May 23, 2024

rqi14 commented May 23, 2024

rqi14 commented May 23, 2024 •

edited

rqi14 commented May 23, 2024 •

edited