Python 3 on Windows (pyreadline) - expected an object with the buffer interface #842

Closed
pfmoore opened this Issue Oct 7, 2011 · 10 comments

Projects

None yet

3 participants

@pfmoore
Contributor
pfmoore commented Oct 7, 2011

This is on Python 3 on Windows, using IPython trunk with the development version of pyreadline from lp:pyreadline (which has Python 3 fixes).

The ipython3.exe console version fails with the following error:

Error in sys.excepthook:
TypeError: expected an object with the buffer interface

Original exception was:
Traceback (most recent call last):
  File "D:\Apps\Python32\Scripts\ipython3-script.py", line 9, in 
    load_entry_point('ipython==0.12.dev', 'console_scripts', 'ipython3')()
  File "D:\Apps\Python32\lib\site-packages\IPython\frontend\terminal\ipapp.py", line 390, in launch_new_instance
    app.initialize()
  File "D:\Apps\Python32\lib\site-packages\IPython\frontend\terminal\ipapp.py", line 301, in initialize
    self.init_shell()
  File "D:\Apps\Python32\lib\site-packages\IPython\frontend\terminal\ipapp.py", line 321, in init_shell
    ipython_dir=self.ipython_dir)
  File "D:\Apps\Python32\lib\site-packages\IPython\config\configurable.py", line 295, in instance
    inst = cls(*args, **kwargs)
  File "D:\Apps\Python32\lib\site-packages\IPython\frontend\terminal\interactiveshell.py", line 113, in __init__
    user_global_ns=user_global_ns, custom_exceptions=custom_exceptions
  File "D:\Apps\Python32\lib\site-packages\IPython\core\interactiveshell.py", line 428, in __init__
    self.init_readline()
  File "D:\Apps\Python32\lib\site-packages\IPython\core\interactiveshell.py", line 1706, in init_readline
    delims = delims.replace(d, "")
TypeError: expected an object with the buffer interface

Looking at the code, it looks like it's a problem of mixing bytes and strings:

            # Remove some chars from the delimiters list.  If we encounter
            # unicode chars, discard them.
            delims = readline.get_completer_delims()
            if not py3compat.PY3:
                delims = delims.encode("ascii", "ignore")
            for d in self.readline_remove_delims:
                delims = delims.replace(d, "")
            delims = delims.replace(ESC_MAGIC, '')
            readline.set_completer_delims(delims)
            # otherwise we end up with a monster history after a while:
            readline.set_history_length(self.history_length)

delims = delims.encode("ascii", "ignore") converts delims from string to bytes, and yet it is only run conditionally. So after this point, the type of delims differs between Python 2 and Python 3. I suspect this is where the problem lies (the "expected...buffer interface" error confirms this, as strings in Python 3 don't expose the buffer interface).

Encoding d and ESC_MAGIC, and replacing the '' literals with b'', works partially but clearly is not the correct fix.

@takluyver
Member

Hmmm, I think readline.set_completer_delims expected Unicode on Python 3. This might be an incompatibility between GNU readline and pyreadline that we need to work out.

@jstenar
Member
jstenar commented Nov 16, 2011

I have pushed a change to pyreadline master that makes get_completer_delims return a unicode string on python3.

ps. pyreadline is now on github if you want to test it

@takluyver
Member

@jstenar: Will it also accept a unicode string for set_completer_delims?

@jstenar
Member
jstenar commented Nov 16, 2011

Thomas skrev 2011-11-16 20:53:

@jstenar: Will it also accept a unicode string for set_completer_delims?


Reply to this email directly or view it on GitHub:
#842 (comment)
I don't know, will look into it tomorrow.

@jstenar
Member
jstenar commented Nov 17, 2011

Thomas skrev 2011-11-16 20:53:

@jstenar: Will it also accept a unicode string for set_completer_delims?


Reply to this email directly or view it on GitHub:
#842 (comment)
In my tests I get no exceptions and a string I set with
set_completer_delims I get by calling get_completer_delims afterwards.

/Jörgen

@takluyver
Member

Great, thanks.

@pfmoore: Any chance you could test it again with a fresh checkout of pyreadline?

@pfmoore
Contributor
pfmoore commented Nov 17, 2011

On 17 November 2011 21:57, Thomas
reply@reply.github.com
wrote:

Great, thanks.

@pfmoore: Any chance you could test it again with a fresh checkout of pyreadline?

Not easily right now. Looks like my Python 3.2 install has gone weird:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp65001

I'll see what I can do in due course...
Paul.

@pfmoore
Contributor
pfmoore commented Nov 17, 2011

On 17 November 2011 22:48, Paul Moore p.f.moore@gmail.com wrote:

Not easily right now. Looks like my Python 3.2 install has gone weird:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp65001

I'll see what I can do in due course...

Never mind, it was a shallow bug. (Although I don't know why it
suddenly appeared :-()

The problem with ipython3.exe appears to be fixed, thanks!
Paul

@takluyver
Member

It appears that for various complex reasons, Python doesn't like a Windows terminal using code page 65001. Either change the code page, or, according to this bug report: "The current workaround is to set PYTHONIOENCODING environment variable to utf-8,".

I think it will be fixed for Python 3.3. Anyway, thanks for testing.

@takluyver takluyver closed this Nov 17, 2011
@pfmoore
Contributor
pfmoore commented Nov 17, 2011

On 17 November 2011 23:01, Thomas
reply@reply.github.com
wrote:

It appears that for various complex reasons, Python doesn't like a Windows terminal using code page 65001. Either change the code page,
or, according to this bug report: "The current workaround is to set PYTHONIOENCODING environment
variable to utf-8,".

The odd thing is that I don't use code page 65001 (although I do use
Powershell, which is probably the thing here). But that's not changed

  • I've used Powershell for ages. Anyway, not the real issue here, but
    thanks for the pointer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment