Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'UnstructuredTokenList' object has no attribute '_fold_as_ew' #71427

touilleMan mannequin opened this issue Jun 6, 2016 · 11 comments

'UnstructuredTokenList' object has no attribute '_fold_as_ew' #71427

touilleMan mannequin opened this issue Jun 6, 2016 · 11 comments
3.7 (EOL) end of life topic-email type-bug An unexpected behavior, bug, or error


Copy link

touilleMan mannequin commented Jun 6, 2016

BPO 27240
Nosy @warsaw, @bitdancer, @berkerpeksag, @touilleMan, @ArchibaldArt, @rslinckx, @vtemian
  • bpo-27240 Rewrite the email header folding algorithm. #3488
  • [3.6] bpo-27240 Rewrite the email header folding algorithm. (GH-3488) #4693
  • Files
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-12-05.12:21:29.872>
    created_at = <Date 2016-06-06.10:25:59.132>
    labels = ['type-bug', '3.7', 'expert-email']
    title = "'UnstructuredTokenList' object has no attribute '_fold_as_ew'"
    updated_at = <Date 2018-11-17.14:54:32.694>
    user = '' fields:

    activity = <Date 2018-11-17.14:54:32.694>
    actor = 'r.david.murray'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-12-05.12:21:29.872>
    closer = 'r.david.murray'
    components = ['email']
    creation = <Date 2016-06-06.10:25:59.132>
    creator = 'touilleMan'
    dependencies = []
    files = ['43257', '46941']
    hgrepos = []
    issue_num = 27240
    keywords = ['patch']
    message_count = 11.0
    messages = ['267525', '295751', '301456', '301481', '301796', '307528', '307530', '307642', '307659', '330016', '330030']
    nosy_count = 8.0
    nosy_names = ['barry', 'r.david.murray', 'berker.peksag', 'touilleMan', 'archibald', 'Zhang Wenbo', 'rslinckx', 'vtemian']
    pr_nums = ['3488', '4693']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = ''
    versions = ['Python 3.6', 'Python 3.7']

    Copy link
    Mannequin Author

    touilleMan mannequin commented Jun 6, 2016

    The folding code for email headers crashes whenever it hits line given it try to call method _fold_as_ew.

    (venv)➜  vigiechiro-api git:(in2p3) ✗ python  
    Traceback (most recent call last):
      File "", line 36, in <module>
      File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/", line 492, in send
      File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/", line 427, in send
      File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/", line 190, in send
        message.as_bytes() if PY3 else message.as_string(),
      File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/", line 385, in as_bytes
        return self._message().as_bytes()
      File "/usr/lib/python3.5/email/", line 179, in as_bytes
        g.flatten(self, unixfrom=unixfrom)
      File "/usr/lib/python3.5/email/", line 115, in flatten
      File "/usr/lib/python3.5/email/", line 195, in _write
      File "/usr/lib/python3.5/email/", line 422, in _write_headers
        self._fp.write(self.policy.fold_binary(h, v))
      File "/usr/lib/python3.5/email/", line 190, in fold_binary
        folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
      File "/usr/lib/python3.5/email/", line 204, in _fold
        return self.header_factory(name, ''.join(lines)).fold(policy=self)
      File "/usr/lib/python3.5/email/", line 255, in fold
        return header.fold(policy=policy)
      File "/usr/lib/python3.5/email/", line 300, in fold
      File "/usr/lib/python3.5/email/", line 1228, in _fold
      File "/usr/lib/python3.5/email/", line 432, in _fold
    AttributeError: 'UnstructuredTokenList' object has no attribute '_fold_as_ew'
    > /usr/lib/python3.5/email/
    -> part._fold_as_ew(folded)

    Given folding depends a lot on the processed value, this bug only trigger in very specific encoded headers (I've try to reproduce this bug with various values but only the one which originally made me find the bug worked so far).

    I've isolated the bug into a simple test case (see enclosed file).

    It seems this part of the code has been rewritten between Python2 and 3 (the file doesn't exists in Python2). So unsurprisingly the bug doesn't trigger in Python2.

    Replacing part._fold_as_ew(folded) by part._fold(folded) seems to fix the bug, but given my poor knowledge of this code, this fix probably produce unwanted behaviors.

    @touilleMan touilleMan mannequin added the topic-email label Jun 6, 2016
    @SilentGhost SilentGhost mannequin added the type-bug An unexpected behavior, bug, or error label Jun 6, 2016
    Copy link

    ZhangWenbo mannequin commented Jun 12, 2017

    I think I find a case which can reproduct this bug:
    the UnstructuredTokenList has a ValueTerminal which is long enough(its encoded word is longer than policy.maxlen ) and not at the beginning.

    For examples, the following subjects can cause this bug:

    • 'Re: 我只是一个有那么一点点长的主题而已'
    • '回复: 回复:【北京大学-2017年招生信息公告】'

    So, the code can not fold a ValueTerminal correctly.
    Seems the method _fold_as_ew was designed to do this, but it is missing.

    Copy link

    rslinckx mannequin commented Sep 6, 2017

    It happens regularly for emails with long header lines for me too, which makes the python3 email module quite unusable.

    A workaround which seems to help a bit is to set refold_source='none' on the email policy, although it seems to keep happening, less often.

    Is the fix proposed by Emmanuel a proper fix ? I'm not knowledgeable enough on this subject...

    Copy link

    I started rewriting the header folder in response to this bug, because the root problem is a bit deeper than just _fold_as_ew being missing. I will probably work on it some more this week, but if I don't think I'm going to get it finished I'll look at putting in a fix that doesn't address the other problems but does address this one.

    @bitdancer bitdancer added the 3.7 (EOL) end of life label Sep 6, 2017
    Copy link

    I managed to finish the rewrite, and it does fix this issue. I'll probably post the PR tomorrow.

    Copy link

    New changeset 85d5c18 by R. David Murray in branch 'master':
    bpo-27240 Rewrite the email header folding algorithm. (bpo-3488)

    Copy link

    New changeset a87ba60 by R. David Murray (Miss Islington (bot)) in branch '3.6':
    bpo-27240 Rewrite the email header folding algorithm. (GH-3488) (bpo-4693)

    Copy link

    All three examples in correctly folded (doesn't hit the ``except Exception as e:`` branch anymore) in current master. Can this issue be closed now?

    Copy link

    Huh, I thought I had closed it.

    Copy link

    vtemian mannequin commented Nov 17, 2018

    Is there any chance that this fix is gonna be ported to 3.5 as well?

    Copy link

    No. 3.5 is in security-fix-only mode.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    3.7 (EOL) end of life topic-email type-bug An unexpected behavior, bug, or error
    None yet

    No branches or pull requests

    2 participants