Broken GZIP + pickle on Python 3.3 #307

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
4 participants

When enabling pickle as the result serializer and enabling gzip as the message compression, I get a consistent UnicodeDecodeError on this line since it is trying to decode bytes into UTF-8, which cannot be done. Removing this line makes everything work like a charm. It also works fine with JSON and JSON+GZIP.

Why was it inserted? What did it fix? Was there a change in python 3.3 that maybe makes it no longer necessary?

Broken GZIP/pickle on Python 3.3
When enabling pickle as the result serializer and enabling gzip as the message compression, I get a consistent UnicodeDecodeError on this line since it is trying to decode bytes into UTF-8, which cannot be done.  Removing this line makes everything work like a charm.  It also works fine with JSON and JSON+GZIP.

Why was it inserted?  What did it fix?  Was there a change in python 3.3 that maybe makes it no longer necessary?
Owner

ask commented Jan 28, 2014

Not sure why you are changing the core bytes_to_str function? It's most certainly needed by both celery and kombu when running on Python 3.

Why cannot bytes be converted to utf-8?

I'll post the error i'm getting. —
George Sibble
Sent from a mobile device

On Tue, Jan 28, 2014 at 6:27 AM, Ask Solem Hoel notifications@github.com
wrote:

Not sure why you are changing the core bytes_to_str function? It's most certainly needed by both celery and kombu when running on Python 3.

Why cannot bytes be converted to utf-8?

Reply to this email directly or view it on GitHub:
#307 (comment)

The error (which goes away with the above fix):

And again, as I said, the above fix does not negatively affect any other combination of result encoding and compression. Maybe pull it into a side branch and run unit tests?

Traceback (most recent call last):
  File "/apps/positronics/api/src/positronics/clry/connection.py", line 21, in send_task
    raw_response = request.get(timeout=20)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/result.py", line 151, in get
    interval=interval)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 153, in wait_for
    meta = self.consume(task_id, timeout=timeout)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 225, in consume
    return wait(conn, consumer, timeout)[task_id]
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 211, in drain_events
    wait(timeout=timeout)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/connection.py", line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 90, in drain_events
    return connection.drain_events(**kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/amqp/connection.py", line 322, in drain_events
    return amqp_method(channel, args, content)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/amqp/channel.py", line 1908, in _basic_deliver
    fun(msg)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/messaging.py", line 580, in _receive_callback
    message = m2p(message)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 54, in message_to_python
    return self.Message(self, raw_message)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 33, in __init__
    **kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/message.py", line 42, in __init__
    body = decompress(body, self.headers['compression'])
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/compression.py", line 70, in decompress
    return bytes_to_str(get_decoder(content_type)(body))
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/utils/encoding.py", line 54, in bytes_to_str
    return s.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/positronics/api/src/positronics/resource/repo_based_list.py", line 29, in do_get
    if not self._repo.find(self.model_class, models, **find_args):
  File "/apps/positronics/api/src/positronics/repository/repository.py", line 18, in find
    'order': order,
  File "/apps/positronics/api/src/positronics/repository/repository.py", line 44, in _chain
    result = method(**kwargs)
  File "/apps/positronics/api/src/positronics/repository/base.py", line 17, in find
    self._do_find(results, start, limit, where, sort, order)
  File "/apps/positronics/api/src/positronics/repository/celery.py", line 22, in _do_find
    response = self._send_task(self._find_task(),args=self._find_args(where),queue=self._queue(location_id))
  File "/apps/positronics/api/src/positronics/repository/celery.py", line 104, in _send_task
    return self._celery.send_task(*args, **kwargs)
  File "/apps/positronics/api/src/positronics/clry/connection.py", line 27, in send_task
    raise GenericException(e)
positronics.repository.exception.GenericException: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Traceback (most recent call last):
  File "/apps/positronics/api/src/positronics/clry/connection.py", line 21, in send_task
    raw_response = request.get(timeout=20)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/result.py", line 151, in get
    interval=interval)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 153, in wait_for
    meta = self.consume(task_id, timeout=timeout)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 225, in consume
    return wait(conn, consumer, timeout)[task_id]
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/celery/backends/amqp.py", line 211, in drain_events
    wait(timeout=timeout)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/connection.py", line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 90, in drain_events
    return connection.drain_events(**kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/amqp/connection.py", line 322, in drain_events
    return amqp_method(channel, args, content)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/amqp/channel.py", line 1908, in _basic_deliver
    fun(msg)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/messaging.py", line 580, in _receive_callback
    message = m2p(message)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 54, in message_to_python
    return self.Message(self, raw_message)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/transport/pyamqp.py", line 33, in __init__
    **kwargs)
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/message.py", line 42, in __init__
    body = decompress(body, self.headers['compression'])
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/compression.py", line 70, in decompress
    return bytes_to_str(get_decoder(content_type)(body))
  File "/apps/positronics/api/env_positronics_api/lib/python3.3/site-packages/kombu/utils/encoding.py", line 54, in bytes_to_str
    return s.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/positronics/api/src/positronics/resource/repo_based_list.py", line 29, in do_get
    if not self._repo.find(self.model_class, models, **find_args):
  File "/apps/positronics/api/src/positronics/repository/repository.py", line 18, in find
    'order': order,
  File "/apps/positronics/api/src/positronics/repository/repository.py", line 44, in _chain
    result = method(**kwargs)
  File "/apps/positronics/api/src/positronics/repository/base.py", line 17, in find
    self._do_find(results, start, limit, where, sort, order)
  File "/apps/positronics/api/src/positronics/repository/celery.py", line 22, in _do_find
    response = self._send_task(self._find_task(),args=self._find_args(where),queue=self._queue(location_id))
  File "/apps/positronics/api/src/positronics/repository/celery.py", line 104, in _send_task
    return self._celery.send_task(*args, **kwargs)
  File "/apps/positronics/api/src/positronics/clry/connection.py", line 27, in send_task
    raise GenericException(e)
positronics.repository.exception.GenericException: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/positronics/api/src/positronics/resource/base.py", line 32, in __call__
    return attr(request)
  File "/apps/positronics/api/src/positronics/resource/base.py", line 65, in get
    data = self.do_get(request)
  File "/apps/positronics/api/src/positronics/resource/repo_based_list.py", line 33, in do_get
    raise Error(500, str(e))
positronics.http.error.Error: (500, "'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte")

sibblegp commented Feb 4, 2014

Can we take a closer look at this? It's still breaking this for us.

milagre commented Feb 4, 2014

I'm also having this problem.

Owner

ask commented Feb 9, 2014

Are you saying that string and bytes are interchangeable in Python 3 now? That was not the case before where many functions do not support the bytes type. Removing the bytes_to_str function will mean that every function that before required a string type will be given a bytes type instead, which almost certainly would blow up in previous versions of Python 3.

sibblegp commented Feb 9, 2014

All I know is that this function breaks gzip in 3.3. I don't know kombu well enough to make a determination as to why but I would like to help get it fixed.—
George Sibble
Sent from a mobile device

On Sun, Feb 9, 2014 at 8:17 AM, Ask Solem Hoel notifications@github.com
wrote:

Are you saying that string and bytes are interchangeable in Python 3 now? That was not the case before where many functions do not support the bytes type. Removing the bytes_to_str function will mean that every function that before required a string type will be given a bytes type instead, which almost certainly would blow up in previous versions of Python 3.

Reply to this email directly or view it on GitHub:
#307 (comment)

@ask ask added this to the v4.1 milestone Jul 9, 2016

Contributor

thedrow commented Jul 16, 2017

This has become stale and #747 replaces it. Let's continue the discussion there.

@thedrow thedrow closed this Jul 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment