Duplicated Content-Transfer-Encoding header when applying email.encoders

[cancel]

BPO	15115
Nosy	@warsaw, @bitdancer, @csabella, @miss-islington
PRs	bpo-15115: Document deprecation of email.encoders in Python 3 #5354 [3.7] bpo-15115: Document deprecation of email.encoders in Python 3 (GH-5354) #13709

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-31.20:20:22.885>
created_at = <Date 2012-06-20.10:38:31.118>
labels = ['3.8', 'type-bug', '3.7', 'docs']
title = 'Duplicated Content-Transfer-Encoding header when applying email.encoders'
updated_at = <Date 2019-05-31.20:26:09.907>
user = 'https://bugs.python.org/cancel'

bugs.python.org fields:

activity = <Date 2019-05-31.20:26:09.907>
actor = 'miss-islington'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-05-31.20:20:22.885>
closer = 'cheryl.sabella'
components = ['Documentation']
creation = <Date 2012-06-20.10:38:31.118>
creator = 'cancel'
dependencies = []
files = []
hgrepos = []
issue_num = 15115
keywords = ['patch']
message_count = 11.0
messages = ['163266', '163275', '163276', '163278', '163279', '163284', '309937', '310013', '310816', '344117', '344119']
nosy_count = 6.0
nosy_names = ['barry', 'r.david.murray', 'docs@python', 'cancel', 'cheryl.sabella', 'miss-islington']
pr_nums = ['5354', '13709']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15115'
versions = ['Python 3.7', 'Python 3.8']

[cancel]

Here is the test script:

--------------

from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email import encoders

msg = MIMEMultipart()
msg['Subject'] = 'Bug test'

text_part = MIMEText('actual content doesnt matter')
text_part.set_charset('utf-8')
msg.attach(text_part)

xml_part = MIMEText(b'<xml>aaa</xml>')
xml_part.set_type('text/xml')
xml_part.set_charset('utf-8')
encoders.encode_base64(xml_part)
msg.attach(xml_part)

print(msg.as_string())

---

It prints the following:
--------------------------
Content-Type: multipart/mixed; boundary="===============2584752675366770986=="
MIME-Version: 1.0
Subject: Bug test

--===============2584752675366770986==
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"

actual content doesnt matter
--===============2584752675366770986==
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/xml; charset="utf-8"
Content-Transfer-Encoding: base64

PHhtbD5hYWE8L3htbD4=

--===============2584752675366770986==--
--------------------------

And that's incorrect: the header "Content-Transfer-Encoding" set twice. As workaround you can use:

del xml_part['Content-Transfer-Encoding']

[bitdancer]

First of all, I presume you are running this in python2.7, since it doesn't work in python3. In Python3 MIMEText takes a string as input, not a bytes object, unless you pass it a _charset parameter. If you do that, the encoding is done when the object is created, and the explicit call to encodings that you do fails. If, on the other hand, you pass in a string, the explicit call to encodings fails.

In 2.7, the encoder call does not fail, but as you report this results in an extra header. This is because MIMEText sets a content type and CTE header using the default values when it is first created.

The explicit call to encoders is not needed What you want to do in stead is to pass the charset and type when you make the MIMEText call:

  MIMEText('<xml>aaa</xml>', 'xml', 'utf-8')

This is the correct way to do it, and the portable way. So, you get the right output by using the API the way it was designed.

That leaves the question of whether or not we should add some documentation (such as: *never* call the functions from the encoders module directly :).

Note that I don't *like* that the current API is that calling set_charset does the body encode if and only if there are no existing headers, but that is the way it has always worked, and there are programs out there that depend on it.

In theory we could fix the encoders functions to check for existing headers and do the right thing in that case, but it is not something that would rate very high on my priority list. I'll happily look at a patch if someone wants to propose one, but since the right way to do this exists, I'm going to treat this issue as documentation-only.

If someone wants to propose a patch for this, please open a new issue.

[bitdancer]

Barry: I think we should documentationally deprecate the encoders module. I can't see any utility in a new program calling those functions explicitly, especially if the program ever wants to port to Python3. Or maybe the Python2 docs would say deprecated in Python3.

What do you think?

[warsaw]

On Jun 20, 2012, at 01:58 PM, R. David Murray wrote:

Barry: I think we should documentationally deprecate the encoders module. I
can't see any utility in a new program calling those functions explicitly,
especially if the program ever wants to port to Python3. Or maybe the
Python2 docs would say deprecated in Python3.

I agree that we should document them as deprecated, as long as we include text
explaining why, or providing alternatives (e.g. "you think you need this but
you don't because...")

I think it does make sense to include text in the Py2 docs about this.

[warsaw]

On Jun 20, 2012, at 01:51 PM, R. David Murray wrote:

Note that I don't *like* that the current API is that calling set_charset
does the body encode if and only if there are no existing headers, but that
is the way it has always worked, and there are programs out there that depend
on it.

Can we nuke that for email6?

[bitdancer]

I think so, yes. When we have the mimeregistry equivalent of the headerregistry, the new mime Message classes can have a set_charset with a different implementation. I'll want to talk about the API details on email-sig before I do anything, though.

[csabella]

Barry/David,

Is this still a change you wanted to include in the documentation?

Thanks!

[bitdancer]

I believe so. For python3 I think it should only apply to the legacy API docs (you would use set_content (directly or indirectly) in python3, not set_payload). I've updated the versions.

[csabella]

Hi David,

I've made a pull request for the way I think you wanted this documented. Please take a look and let me know if it's even close to what you were suggesting. Thanks! :-)