Imap lib implicit conversion from bytes to string

[surprising42]

BPO	6734
Nosy	@pitrou, @vstinner, @bitdancer
Superseder	bpo-4471: IMAP4 missing support for starttls
Files	imaplib.py: updated imap.py imaplib-login.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-06-20.03:03:26.058>
created_at = <Date 2009-08-19.13:17:59.077>
labels = ['type-bug', 'library']
title = 'Imap lib implicit conversion from bytes to string'
updated_at = <Date 2011-06-20.03:03:26.056>
user = 'https://bugs.python.org/surprising42'

bugs.python.org fields:

activity = <Date 2011-06-20.03:03:26.056>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2011-06-20.03:03:26.058>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2009-08-19.13:17:59.077>
creator = 'surprising42'
dependencies = []
files = ['14743', '14752']
hgrepos = []
issue_num = 6734
keywords = ['patch']
message_count = 9.0
messages = ['91729', '91733', '91748', '91765', '91767', '91768', '91775', '91815', '138685']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'marcin.bachry', 'r.david.murray', 'surprising42']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '4471'
type = 'behavior'
url = 'https://bugs.python.org/issue6734'
versions = ['Python 3.1', 'Python 3.2']

[surprising42]

using imaplib IMAP4_SSL, would fail at login due to TypeError, implicit
conversion from bytes object to string around line 1068 involving
function "_quote".

My fix:
def _quote(self, arg):
#Could not implicitly convert to bytes object to string
arg = arg.encode('utf-8') #added this line to solve problem
arg = arg.replace(b'\\', b'\\\\')
arg = arg.replace(b'"', b'\\"')

    return b'"' + arg + b'"'

[bitdancer]

See bpo-1210 for background on the imaplib bytes/string issues.

A quick glance at the imaplib code leaves me confused. _checkquote,
for example, appears to be mixing string comparisons and byte
comparisons. bpo-1210 says imaplib _command should only quote
strings, but it does not appear at a quick glance to do any
quoting (the call to _checkquote is commented out).

It looks like the fix that would be consonant with the rest of
the code would be to convert the password to bytes using
the ASCII codec in the login method before calling _quote.

I'm adding Victor as nosy since he did the 1210 patch.

[vstinner]

IMAP4 protocol uses bytes, not characters. There is no "standard
charset", so you have to encode (manually) your login and password as
bytes. login method prototype:
IMAP4.login(login: bytes, password: bytes)

You server may use UTF-8, but another server may use ISO-8859-1 or any
other charset.

The documentation should explain why the Python library uses bytes and
not "simply" characters.

[surprising42]

I checked the latest documentation for 3.1.1
(http://docs.python.org/3.1/library/imaplib.html), but I can't find any
reference to needing to encode information myself for the login
procedure. Is there some other documentation you are referring to?

In any case, the error wasn't returned by the server, but by imaplib. If
the arg needs to be encoded for the whole process, then the arg should
be checked for the appropriate type (I think the doc even says "plain
text password"), or an optional encoding argument for the relevant
functions (or a catchall used when connecting to the server?) with a
default encoding attempted.

[vstinner]

I can't find any reference to needing to encode information
myself for the login procedure. Is there some other
documentation you are referring to?

Exactly, the Python imaplib documentation should be fixed (the doc, not
the code).

[marcinbachry]

It seems most IMAP4 methods accept str as arguments right now (I
checked: list, lsub, myrights, select, status, search, fetch) and
login() is a sole exception. I know the protocol is mostly ascii only,
but still having possibility of using str in the API feels convenient
and reasonable to me. Can't we convert str parameters to login() to
ascii too? It already done in other methods (the _command() method
iterates over args and converts them).

[vstinner]

It seems most IMAP4 methods accept str as arguments right now (I
checked: list, lsub, myrights, select, status, search, fetch) and
login() is a sole exception. I know the protocol is mostly ascii only,
but still having possibility of using str in the API feels convenient
and reasonable to me.

Ok, that sounds like a good compromise. Can you write a patch?

[marcinbachry]

Ok, it think it's good to leave internal _quote() function operating on
bytes and convert password argument in login(). The method now works
with both str and bytes as arguments.

[bitdancer]

This was fixed in bpo-4471 by Antoine when he added some tests that call login. He fixed it by changing _quote to work with strings. Per the discussion here I'm not sure this is the best fix, but until someone reports a bug with it it we may as well let it stand.