Got "TypeError: must be convertible to a buffer, not bytearray" in PyPy

[ethe]

PyPy version: 4.0.1 with GCC 4.8.4 on linux2
System: Ubuntu 15.04 x86 64bit
Error line: ./site-packages/redis/connection.py line 359 self._reader.feed(self._buffer, 0, bufflen)

Traceback (most recent call last):
  File "/home/zeno/amap/env/pypy/site-packages/redis/client.py", line 572, in execute_command
    connection.send_command(*args)
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 564, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 539, in send_packed_command
    self.connect()
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 447, in connect
    self.on_connect()
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 515, in on_connect
    if nativestr(self.read_response()) != 'OK':
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 578, in read_response
    response = self._parser.read_response()
  File "/home/zeno/amap/env/pypy/site-packages/redis/connection.py", line 360, in read_response
    self._reader.feed(self._buffer, 0, bufflen)
TypeError: must be convertible to a buffer, not bytearray

Now I just move hiredis out, redis clients works no problem.

[ethe]

I checked hiredis-py's usage, method feed expects a string instead of bytearray. I think above pull request can solve this and does not cause any other problem.

[ethe]

This error was catched so can't be throwed out, only use single-step debug can find it.

[andymccurdy]

I'm not sure about this. The whole point of using a bytearray is to prevent memory allocations. Support for bytearray was specifically added to hiredis-py here: redis/hiredis-py#22

Casting the bytearray to bytes creates an additional memory allocation every call. I'd really like to avoid that.

Does PyPy treat bytearray objects differently than CPython does? From the Python docs, the two should be completely interchangeable:

"Both bytes and bytearray objects support the common sequence operations. They interoperate not just with operands of the same type, but with any bytes-like object. Due to this flexibility, they can be freely mixed in operations without causing errors."
https://docs.python.org/3/library/stdtypes.html#bytes-methods

[ethe]

Yeah, I'm sure that the behavior of bytearray is different on pypy 4.0.1. Maybe I provide a simple way to reproduce it? I think could just check if interpreter is pypy then use bytes method?

[ethe]

Can simply reproduce by below program.

import hiredis


test = bytearray()

reader = hiredis.Reader()
reader.feed(test)

Python 2.7.10 run it no problem, but use PyPy 4.0.1 got below error.

 ➜  pypy reproduce.py              
Traceback (most recent call last):
  File "reproduce.py", line 7, in <module>
    reader.feed(test)
TypeError: must be convertible to a buffer, not bytearray

[andymccurdy]

It looks like this is a hiredis-py error. The bytearray test in the hiredis-py suite (https://github.com/redis/hiredis-py/blob/master/test/reader.py#L168) fails when running against PyPy 4.0.1 raising the error you're seeing.

Also, these two tests (https://github.com/redis/hiredis-py/blob/master/test/reader.py#L133-L139) cause the test runner to segfault against PyPy 4.0.1. Not sure why, but this is likely another problem you'll run into trying to run hiredis-py on PyPy. I commented them out temporarily to get the rest of the suite to run and see the bytearray test error.

I'm not very familiar with PyPy (certainly not at the C extension level), so I'm not sure where to really start.

[ethe]

Well, I think the temporary solution to solve this problem is disable hiredis-py when using PyPy, I think pure Python code run on PyPy have an acceptable performance.

[andymccurdy]

I created an issue on the hiredis-py repo. Hopefully someone more familiar with PyPy than I can take a look.

[jbrockmendel]

FWIW I just got the same error with PyPy 5.1.2 (Python 2.7.10, Ubuntu 16.04):

  File "[...]/site-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "[...]/site-packages/redis/connection.py", line 577, in read_response
    response = self._parser.read_response()
  File "[...]/site-packages/redis/connection.py", line 359, in read_response
    self._reader.feed(self._buffer, 0, bufflen)
TypeError: must be string or buffer, not bytearray

[andymccurdy]

PyPy fixed their ByteArray support in version 5.3. hiredis-py (and redis-py with hiredis-py enabled) works with PyPy 5.3 or later.