Cannot use remotepasseval with Python3

[darnir]

General informations

• system/distribution (with version): Arch Linux
• offlineimap version (offlineimap -V): offlineimap v7.0.5, imaplib2 v2.55 (bundled)
• Python version: Python 3.5.2
• server name or domain:
• CLI options: -o

Configuration file offlineimaprc

remotepasseval = get_pass('UdS')

pythonfile (if any)

#! /usr/bin/env python
from subprocess import check_output


def get_pass(account):
    return check_output(["pass", "Mail/" + account]).splitlines()[0]

Logs, error

OfflineIMAP 7.0.5
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
*** Processing account UdS
Establishing connection to a.remote.server (UdS-remote)
ERROR: While attempting to sync account 'UdS'
  'bytes' object has no attribute 'encode'
*** Finished account 'UdS' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'UdS'
  'bytes' object has no attribute 'encode'

Traceback:
  File "/usr/lib/python3.5/site-packages/offlineimap/accounts.py", line 271, in syncrunner
    self.__sync()
  File "/usr/lib/python3.5/site-packages/offlineimap/accounts.py", line 334, in __sync
    remoterepos.getfolders()
  File "/usr/lib/python3.5/site-packages/offlineimap/repository/IMAP.py", line 444, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/lib/python3.5/site-packages/offlineimap/imapserver.py", line 519, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/lib/python3.5/site-packages/offlineimap/imapserver.py", line 422, in __authn_helper
    if func(imapobj):
  File "/usr/lib/python3.5/site-packages/offlineimap/imapserver.py", line 351, in __authn_plain
    imapobj.authenticate('PLAIN', self.__plainhandler)
  File "/usr/lib/python3.5/site-packages/offlineimap/bundled_imaplib2.py", line 705, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File "/usr/lib/python3.5/site-packages/offlineimap/bundled_imaplib2.py", line 1692, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3.5/site-packages/offlineimap/bundled_imaplib2.py", line 1418, in _command
    literal = literator(data, rqb)
  File "/usr/lib/python3.5/site-packages/offlineimap/bundled_imaplib2.py", line 2283, in process
    ret = self.mech(self.decode(data))
  File "/usr/lib/python3.5/site-packages/offlineimap/imapserver.py", line 216, in __plainhandler
    passwd = self.__getpassword()
  File "/usr/lib/python3.5/site-packages/offlineimap/imapserver.py", line 169, in __getpassword
    self.password = self.repos.getpassword() or \
  File "/usr/lib/python3.5/site-packages/offlineimap/repository/IMAP.py", line 384, in getpassword
    return self.localeval.eval(passwd).encode('UTF-8')

Steps to reproduce the error

[darnir]

Setting the pythonfile to be:

def get_pass(account):
    return check_output(["pass", "Mail/" + account]).splitlines()[0].decode()

does not work since offlineimap expects it to be a byte object, not a string. But Python3 byte objects do not have an encode attribute

[drnedm]

I had the same issue and found the following fixes it. However, using python3, offlineimap tries to authenticate using method=PLAIN first, which fails. Server log:

python3:
Sep 18 17:34:49 mail imapd-ssl[2327]: Connection, ip=[1.2.3.4]
Sep 18 17:34:50 mail imapd-ssl[2327]: LOGIN FAILED, method=PLAIN, ip=[1.2.3.4]
Sep 18 17:34:55 mail imapd-ssl[2327]: LOGIN, user=myUsername, ip=[1.2.3.4], port=[35990], protocol=IMAP

python2:
Sep 18 17:36:15 mail imapd-ssl[2327]: Connection, ip=[1.2.3.4]
Sep 18 17:36:16 mail imapd-ssl[2327]: LOGIN, user=myUsername, ip=[1.2.3.4], port=[35998], protocol=IMAP

No idea why that happens, but after an initial
PLAIN authentication failed: Login failed.

offlineimap works nicely with python2 and python3.

Hope that helps,

Cheers,
B

offlineimapStrByteFix.patch.txt

[nicolas33]

@BeerSerc Could send your signoff so I can merge? The chunks on imaplib2 must be sent to the imaplib2 project.

[drnedm]

As I said, it still triggers the behaviour that OI now tries this PLAIN auth to the server, so I think it might be good if somebody else could have a look at it. Not a python3 expert myself ...

And as I'm new to github, how would I sign off an attached patch like this? Or do I need to fork and send a pull request? Or do you just want a signed-off line with an email address?

[dolohow]

And as I'm new to github, how would I sign off an attached patch like this?

When doing git commit add switch --signoff.

Or do I need to fork and send a pull request? Or do you just want a signed-off line with an email address?

@nicolas33 accepts both, but the commits or email patches need to be signed off.

[nicolas33]

I'll remove the chunk about UTF-8 encoding, too. This is the wrong place to do this. In IMAP.py the value of IMAPRepository.getuser() must be UTF-8 encoded, already..

[drnedm]

This is all about python2 str vs python3 bytes. So if you remove the encode/decode stuff, we are back to zero where offlineimap does not work with python3 if you use remotepasseval.

[nicolas33]

I was talking the encode call in your patch. It's wrong to have two chained UTF-8 encode calls. With the patch they are for some case. If fixing one use case breaks other use cases, the patch is wrong.

[drnedm]

Sorry for the delay. What I wanted to say is: My patch is nothing but encodes and decodes, because for some reason in 7.0.6/HEAD this is in some places mixed, what results in the problems the original reporter (and me) had.
If you remove all encodes/decodes, there is literally no patch left.
I don't know, where precisely this has to be fixed, as it's the first time I am looking at offlineimap code. And it's also why I didn't create a pull-request for it or signed it off, as I am not sure the patch is sane, just saying it fixes the problem and thus might help somebody with more knowledge than me to find the correct place to fix the string/bytes handling.

[nicolas33]

@BeerSerc Got it! Thanks!

[drnedm]

I traced it down a little further. I am pretty convinced that the problem is the _quote() function in bundled_imaplib2.py +1648.

it reads

def _quote(self, arg):
return '"%s"' % arg.replace('', '\').replace('"', '"')

which fails if arg is your password as a byte string.
Unfortunately, if I change it to work with bytestrings, I get no backtrace anymore, but

ERROR: All authentication types failed:
PLAIN: Login failed.
LOGIN: Error in IMAP command received by server.

[thekix]

This bug should be closed.

If you are running Python3, then use offlineimap3.

Regards,
kix