Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeEncodeError: surrogates not allowed when parsing invalid charset #87489

Open
andersk mannequin opened this issue Feb 25, 2021 · 9 comments
Open

UnicodeEncodeError: surrogates not allowed when parsing invalid charset #87489

andersk mannequin opened this issue Feb 25, 2021 · 9 comments
Labels
3.9 only security fixes 3.10 bug and security fixes 3.11 bug and security fixes expert-email stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@andersk
Copy link
Mannequin

andersk mannequin commented Feb 25, 2021

BPO 43323
Nosy @warsaw, @bitdancer, @andersk, @serhiy-storchaka, @glaubitz
PRs
  • bpo-43323: Fix UnicodeEncodeError in the email module #32137
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-02-25.18:18:08.224>
    labels = ['type-bug', 'expert-email', '3.10', '3.11', 'library', '3.9']
    title = 'UnicodeEncodeError: surrogates not allowed when parsing invalid charset'
    updated_at = <Date 2022-03-28.22:29:32.019>
    user = 'https://github.com/andersk'

    bugs.python.org fields:

    activity = <Date 2022-03-28.22:29:32.019>
    actor = 'glaubitz'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)', 'email']
    creation = <Date 2021-02-25.18:18:08.224>
    creator = 'andersk'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43323
    keywords = ['patch']
    message_count = 9.0
    messages = ['387685', '410426', '416092', '416101', '416107', '416109', '416112', '416113', '416225']
    nosy_count = 6.0
    nosy_names = ['barry', 'r.david.murray', 'andersk', 'serhiy.storchaka', 'mdengler', 'glaubitz']
    pr_nums = ['32137']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue43323'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @andersk
    Copy link
    Mannequin Author

    andersk mannequin commented Feb 25, 2021

    We ran into a UnicodeEncodeError exception using email.parser to parse this email <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February/msg00135.html\>, with full headers available in the raw archive <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February.txt\>. The offending header is hilariously invalid:

    Content-Type: text/plain; charset*=utf-8”''utf-8%E2%80%9D

    but I’m filing an issue since the parser is intended to be robust against invalid input. Minimal reproduction:

    >>> import email, email.policy
    >>> email.message_from_bytes(b"Content-Type: text/plain; charset*=utf-8\xE2\x80\x9D''utf-8%E2%80%9D", policy=email.policy.default)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/lib/python3.10/email/__init__.py", line 46, in message_from_bytes
        return BytesParser(*args, **kws).parsebytes(s)
      File "/usr/local/lib/python3.10/email/parser.py", line 123, in parsebytes
        return self.parser.parsestr(text, headersonly)
      File "/usr/local/lib/python3.10/email/parser.py", line 67, in parsestr
        return self.parse(StringIO(text), headersonly=headersonly)
      File "/usr/local/lib/python3.10/email/parser.py", line 57, in parse
        return feedparser.close()
      File "/usr/local/lib/python3.10/email/feedparser.py", line 187, in close
        self._call_parse()
      File "/usr/local/lib/python3.10/email/feedparser.py", line 180, in _call_parse
        self._parse()
      File "/usr/local/lib/python3.10/email/feedparser.py", line 256, in _parsegen
        if self._cur.get_content_type() == 'message/delivery-status':
      File "/usr/local/lib/python3.10/email/message.py", line 578, in get_content_type
        value = self.get('content-type', missing)
      File "/usr/local/lib/python3.10/email/message.py", line 471, in get
        return self.policy.header_fetch_parse(k, v)
      File "/usr/local/lib/python3.10/email/policy.py", line 163, in header_fetch_parse
        return self.header_factory(name, value)
      File "/usr/local/lib/python3.10/email/headerregistry.py", line 608, in __call__
        return self[name](name, value)
      File "/usr/local/lib/python3.10/email/headerregistry.py", line 196, in __new__
        cls.parse(value, kwds)
      File "/usr/local/lib/python3.10/email/headerregistry.py", line 453, in parse
        kwds['decoded'] = str(parse_tree)
      File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in __str__
        return ''.join(str(x) for x in self)
      File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in <genexpr>
        return ''.join(str(x) for x in self)
      File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 798, in __str__
        for name, value in self.params:
      File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 783, in params
        value = value.decode(charset, 'surrogateescape')
    UnicodeEncodeError: 'utf-8' codec can't encode characters in position 5-7: surrogates not allowed

    @andersk andersk mannequin added 3.7 only security fixes 3.8 only security fixes 3.10 bug and security fixes 3.9 only security fixes expert-email labels Feb 25, 2021
    @terryjreedy terryjreedy added type-bug An unexpected behavior, bug, or error and removed 3.7 only security fixes labels Feb 27, 2021
    @glaubitz
    Copy link
    Mannequin

    glaubitz mannequin commented Jan 12, 2022

    I'm running into exactly this issue when using 'offlineimap' which is written in Python.

    @serhiy-storchaka
    Copy link
    Member

    It is interesting that you get an UnicodeEncodeError when try to decode. Could the charser name contain non-ascii characters?

    @andersk
    Copy link
    Mannequin Author

    andersk mannequin commented Mar 27, 2022

    It could and does, as quoted in my original report.

    Content-Type: text/plain; charset*=utf-8”''utf-8%E2%80%9D

    That’s a U+201D right double quotation mark.

    This is not a valid charset for the charset of course, but it seems like the code was intended to handle an invalid charset value without crashing, so it should also handle an invalid charset charset (despite the absurdity of the entire concept of a charset charset).

    @serhiy-storchaka
    Copy link
    Member

    Sorry, I was puzzled by the exception type and missed details in a long traceback (I have issues with reading large texts). Thank you for your detailed report.

    The simple fix is to add UnicodeEncodeError to "except LookupError". But there may be other places where we can get a similar error. They should be fixed too.

    Alternatively we can do something when we get an invalid charset from the parsed data. I am not the email package expert, so I do not know what would be better in that context.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir 3.11 bug and security fixes and removed 3.8 only security fixes labels Mar 27, 2022
    @glaubitz
    Copy link
    Mannequin

    glaubitz mannequin commented Mar 27, 2022

    Hi Serhiy!

    The simple fix is to add UnicodeEncodeError to "except LookupError". But there may be other places where we can get a similar error. They should be fixed too.

    I would be very interested to test this as this issue currently blocks my use of offlineimap.

    Would you mind creating a proof-of-concept patch for me if it's not too much work?

    Thanks!

    @serhiy-storchaka
    Copy link
    Member

    I fixed all suspicious places for which I found reproducers in PR 32137.

    @glaubitz
    Copy link
    Mannequin

    glaubitz mannequin commented Mar 27, 2022

    Awesome, thanks! I'll give it a try later today or tomorrow.

    @glaubitz
    Copy link
    Mannequin

    glaubitz mannequin commented Mar 28, 2022

    Awesome, thanks! I'll give it a try later today or tomorrow.

    I have applied the patch and the problem seems to have been fixed. \o/

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 bug and security fixes 3.11 bug and security fixes expert-email stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants