Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

offlineimap 7.0.0 breaks remotepasseval #354

Closed
deadc0de6 opened this issue Jul 23, 2016 · 10 comments
Assignees
Labels

Comments

@deadc0de6
Copy link

@deadc0de6 deadc0de6 commented Jul 23, 2016

While using remotepasseval with below configs was working using offlineimap version 6.6.1, it fails when using version 7.0.0 with my current configs.

Here is the change on the call in offlineimap/repository/IMAP.py between those two versions:

326c356
<             return self.localeval.eval(passwd)

---
>             return self.localeval.eval(passwd).encode('UTF-8')

I don't pretend it's a bug. It seems when providing a str instead of a byte as a return of the funciton called by remotepasseval fails further down (in offlineimap/bundled_imaplib2.py.

General informations

  • offlineimap version 7.0.0:
  • Python version: 3.4.2

Configuration file offlineimaprc

...
remotepasseval = getpass()
...

pythonfile (if any)

...
def getpass():
  path = ...
  return subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip()
...

Logs, error

Error occurring when using above config:

ERROR: While attempting to sync account 'XXX'
  'bytes' object has no attribute 'encode'
 *** Finished account 'XXX' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'XXX'
  'bytes' object has no attribute 'encode'

Traceback:
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 271, in syncrunner
    self.__sync()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 334, in __sync
    remoterepos.getfolders()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/repository/IMAP.py", line 416, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 521, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 422, in __authn_helper
    if func(imapobj):
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 361, in __authn_login
    self.__loginauth(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 203, in __loginauth
    imapobj.login(self.username, self.__getpassword())
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 164, in __getpassword
    self.password = self.repos.getpassword() or \
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/repository/IMAP.py", line 356, in getpassword
    return self.localeval.eval(passwd).encode('UTF-8')

It seems it is thus expecting a str instead of bytes. Changing the function to something like that:

...
def getpass():
  path = ...
  return subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip().decode()
...

Using .decode() and providing a string in getpass() fails further down:

ERROR: While attempting to sync account 'XXX'
  expected bytes, bytearray or buffer compatible object
 *** Finished account 'XXX' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'XXX'
  expected bytes, bytearray or buffer compatible object

Traceback:
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 271, in syncrunner
    self.__sync()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 334, in __sync
    remoterepos.getfolders()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/repository/IMAP.py", line 416, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 521, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 422, in __authn_helper
    if func(imapobj):
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 361, in __authn_login
    self.__loginauth(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 203, in __loginauth
    imapobj.login(self.username, self.__getpassword())
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/bundled_imaplib2.py", line 906, in login
    typ, dat = self._simple_command('LOGIN', user, self._quote(password))
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/bundled_imaplib2.py", line 1646, in _quote
    return '"%s"' % arg.replace('\\', '\\\\').replace('"', '\\"')

Steps to reproduce the error

  • use above configs and run offlineimap -o
@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 23, 2016

It seems it is thus expecting a str instead of bytes. Changing the function to something like that:

Right. I've talked about that in the announce but this might require more explanations.

While on the road to py3 we need to think larger and Unicode is the way to proceed. This includes password encodings.

You're correct about the change of type (str). The expected encoding changes to Unicode. Adding .decode() in getpass() might or might not be the right thing.
It might be required to check both the encodings and types for subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip(). It could be possible it is already unicode. I don't know.
If adding .decode() is correct I'd be explicit about the encoding and use .decode('UTF-8').

By chance, this should be easy to debug. We must have the exact same type and encoding after eval().encode('UTF-8') than before the change. All of this can be tested in interactive shell.

@nicolas33 nicolas33 added the question label Jul 23, 2016
@deadc0de6

This comment has been minimized.

Copy link
Author

@deadc0de6 deadc0de6 commented Jul 27, 2016

check_output returned value is bytes

>>> val = subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip()
>>> type(val)
<class 'bytes'>

however even by changing it explicitely to UTF-8 with

>>> type(val.decode('utf-8'))
<class 'str'>

it still fails.

Here the entry in my python script:

...
def getpass():
  path = ...
  return subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip().decode('utf-8')
...

and here's the error

 ERROR: While attempting to sync account 'XXX'
  expected bytes, bytearray or buffer compatible object
 *** Finished account 'XXX' in 0:02
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'XXX'
  expected bytes, bytearray or buffer compatible object

Traceback:
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 271, in syncrunner
    self.__sync()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 334, in __sync
    remoterepos.getfolders()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/repository/IMAP.py", line 416, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 526, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 427, in __authn_helper
    if func(imapobj):
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 366, in __authn_login
    self.__loginauth(imapobj)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/imapserver.py", line 208, in __loginauth
    imapobj.login(self.username, self.__getpassword())
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/bundled_imaplib2.py", line 906, in login
    typ, dat = self._simple_command('LOGIN', user, self._quote(password))
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/bundled_imaplib2.py", line 1646, in _quote
    return '"%s"' % arg.replace('\\', '\\\\').replace('"', '\\"')

If it's not unicode it fails in offlineimap/repository/IMAP.py but if it's unicode, it then fails in offlineimap/bundled_imaplib2.py.

@nicolas33 nicolas33 added bug Py3 and removed question labels Jul 27, 2016
@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 27, 2016

Ok. Looks like there are issues with Python3 in this area.

Did you try to check the output and type of return self.localeval.eval(passwd).encode('UTF-8') in offlineimap/repository/IMAP.py?

@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 27, 2016

Does it work with Python2?

@deadc0de6

This comment has been minimized.

Copy link
Author

@deadc0de6 deadc0de6 commented Jul 27, 2016

interesting ... actually return self.localeval.eval(passwd).encode('UTF-8') returns bytes ... adding an encode('utf-8') solves that very issue with the password.

It then however raises further issues due to encoding in other parts:

...
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 591, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 1089, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 917, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 817, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 577, in savemessage
    content)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 346, in __generate_randomheader
    headervalue  = str( binascii.crc32(content) & 0xffffffff ) + '-'
'str' does not support the buffer interface
...
Traceback:
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 817, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 577, in savemessage
    content)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 346, in __generate_randomheader
    headervalue  = str( binascii.crc32(content) & 0xffffffff ) + '-'
...
Traceback:
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 1089, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 917, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 817, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 577, in savemessage
    content)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 346, in __generate_randomheader
    headervalue  = str( binascii.crc32(content) & 0xffffffff ) + '-'
...
 Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/accounts.py", line 591, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 1089, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 917, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/Base.py", line 817, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 577, in savemessage
    content)
  File "/usr/local/lib/python3.4/dist-packages/offlineimap/folder/IMAP.py", line 346, in __generate_randomheader
    headervalue  = str( binascii.crc32(content) & 0xffffffff ) + '-'
TypeError: 'str' does not support the buffer interface
....

Sounds like that road to py3 and Unicode encoding involves some changes in the format of the config files / pythonfile ;-)

It does work with python2 and offllineimap 6.7.0-rc1

@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 27, 2016

interesting ... actually return self.localeval.eval(passwd).encode('UTF-8') returns bytes ... adding an encode('utf-8') solves that very issue with the password.

So, there is something wrong in this area. I still have no idea why it breaks, though.

Sounds like that road to py3 and Unicode encoding involves some changes in the format of the config files / pythonfile ;-)

Yes. I've already tried to open the config file in UTF-8 mode but this breaks py2 versions.

It does work with python2 and offllineimap 6.7.0-rc1

Does py2 and v7.0.2 work?

@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 27, 2016

I can't reproduce your issue. It looks to work as expected here:

Python 3.4.3 (default, Jan 15 2016, 11:12:39) [GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def func(): return subprocess.check_output('/home/nicolas/blah.sh').strip().decode('utf-8')
>>> import subprocess
>>> eval("func()", {'func': func})
'é'
>>> eval("func()", {'func': func}).encode('utf-8')
b'\xc3\xa9'

How does this goes with python 3.4.2?

@deadc0de6

This comment has been minimized.

Copy link
Author

@deadc0de6 deadc0de6 commented Jul 28, 2016

The original issue was the following one:

Python 3.4.2 (default, Oct  8 2014, 10:45:20)
>>> def func(): return subprocess.check_output('/tmp/x.sh').strip()
... 
>>> eval("func()", {'func': func})
b'\xc3\xa9'
>>> eval("func()", {'func': func}).encode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute 'encode'

adding a .decode('utf-8') in the pythonfile to ensure the password is then unicode helps solve it in offlineimap/repository/IMAP.py:

>>> def func(): return subprocess.check_output('/tmp/x.sh').strip().decode('utf-8')
... 
>>> eval("func()", {'func': func}).encode('utf-8')
b'\xc3\xa9'

The problem then was in offlineimap/bundled_imaplib2.py complaining it expected bytes in some functions (see above for the traceback).

Then changing IMAP.py line from return self.localeval.eval(passwd).encode('UTF-8') to return self.localeval.eval(passwd).encode('UTF-8').decode('utf-8') solved that issue.

With offlineimap 7.0.2 in py2.7, it works flawlessly.

@nicolas33

This comment has been minimized.

Copy link
Member

@nicolas33 nicolas33 commented Jul 28, 2016

Then changing IMAP.py line from return self.localeval.eval(passwd).encode('UTF-8') to return self.localeval.eval(passwd).encode('UTF-8').decode('utf-8') solved that issue.

Ok, I get it. There is a function expecting unicode while we provide bytes. We do this because with Py2, imaplib2 expects encoded strings. The current imaplib2 was not meant to work with Py3.

With offlineimap 7.0.2 in py2.7, it works flawlessly.

Ok, thanks.

@myrjola

This comment has been minimized.

Copy link

@myrjola myrjola commented Aug 2, 2016

With offlineimap 7.0.2 in py2.7, it works flawlessly.

Works wonderfully in 7.0.3 with python 3.5.1 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.