Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kombu decode error #3315

Closed
vatsalshah1990 opened this issue Jul 14, 2016 · 6 comments
Closed

kombu decode error #3315

vatsalshah1990 opened this issue Jul 14, 2016 · 6 comments
Labels
Status: Has Testcase ✔ Status: Needs Verification ✘

Comments

@vatsalshah1990
Copy link

I'm using pickle as serializer. Using celery 3.1.23. Only getting it in this particular worker where I'm using a couple of tasks with retry turned on using bind. Not sure if that has something to do with this.

Can't decode message body: DecodeError(TypeError("__init__() missing 1 required positional argument: 'offset'",),) [type:'application/x-python-serialize' encoding:'binary' headers:{}]
return decode(data)
decoded = None if on_m else message.decode()
File "/var/app/lib/python3.4/site-packages/kombu/message.py", line 142, in decode
return decode(data)
return decode(data)
kombu.exceptions.DecodeError: __init__() missing 1 required positional argument: 'offset'
@ask
Copy link
Contributor

ask commented Jul 19, 2016

This could seem like you are simply trying to pass an unpickleable object. What type of objects are you passing to the task?

@vatsalshah1990
Copy link
Author

I'm not getting this error when it runs the first time. Celery is able to unpickle the first time but fails to decode on retry. I'm passing a django model object and string as parameters to the function. Here is one of the functions where I'm getting this error:

@app.task(bind=True, queue="notifications")
def send_push_notification(self, shipment, status):
    try:
        logger.info("Started task send_push_notification")
    ...
    except Exception as exc:
        raise self.retry(exc=exc)

@ask
Copy link
Contributor

ask commented Aug 9, 2016

Where is offset coming from? That it works the first time doesn't mean it will work the second time.

@alzeih
Copy link
Contributor

alzeih commented Sep 29, 2016

I ran across this issue and worked around it by changing

raise self.retry(exc=exc)

to

raise Retry(exc=exc)

@auvipy
Copy link
Member

auvipy commented Dec 19, 2017

reopen if it is still an issue on master

@auvipy auvipy closed this as completed Dec 19, 2017
@ocervell
Copy link

ocervell commented Apr 23, 2024 •
edited

@auvipy this bit me today while trying to access results using result.children or result.info: it is definitely still an issue when using pickle as a serializer and the data is quite large.

Here is the tracebacks:

Traceback (most recent call last):
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 265, in loads
    return decode(data)
           ^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 337, in unpickle
    return pickle_loads(str_to_bytes(s))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 50, in pickle_loads
    return load(BytesIO(s))
           ^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/osboxes/Workspace/secator/.venv/bin/secator", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/rich_click/rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/secator/decorators.py", line 311, in func
    runner.run()
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 217, in run
    return list(self.__iter__())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 226, in __iter__
    for item in self.yielder():
  File "/home/osboxes/Workspace/secator/secator/runners/scan.py", line 63, in yielder
    yield from workflow
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 226, in __iter__
    for item in self.yielder():
  File "/home/osboxes/Workspace/secator/secator/runners/workflow.py", line 61, in yielder
    yield from results
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 644, in process_live_tasks
    for data in Runner.get_live_results(result):
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 535, in get_live_results
    yield from Runner.get_celery_results(result)
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 559, in get_celery_results
    data = get_task_data(task_id)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/secator/runners/_helpers.py", line 126, in get_task_data
    if res.info and not isinstance(res.info, list):
                                   ^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/result.py", line 466, in result
    return self._get_task_meta()['result']
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/result.py", line 442, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 608, in get_task_meta
    meta = self._get_task_meta_for(task_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 998, in _get_task_meta_for
    return self.decode_result(meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 425, in decode_result
    return self.meta_from_decoded(self.decode(payload))
                                  ^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 431, in decode
    return loads(payload,
           ^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 264, in loads
    with _reraise_errors(DecodeError):
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 45, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/exceptions.py", line 34, in reraise
    raise value.with_traceback(tb)
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 265, in loads
    return decode(data)
           ^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 337, in unpickle
    return pickle_loads(str_to_bytes(s))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 50, in pickle_loads
    return load(BytesIO(s))
           ^^^^^^^^^^^^^^^^
kombu.exceptions.DecodeError: pickle data was truncated

and:

Traceback (most recent call last):
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 265, in loads
    return decode(data)
           ^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 337, in unpickle
    return pickle_loads(str_to_bytes(s))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 50, in pickle_loads
    return load(BytesIO(s))
           ^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/osboxes/Workspace/secator/.venv/bin/secator", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/rich_click/rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/secator/decorators.py", line 311, in func
    runner.run()
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 217, in run
    return list(self.__iter__())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 226, in __iter__
    for item in self.yielder():
  File "/home/osboxes/Workspace/secator/secator/runners/scan.py", line 63, in yielder
    yield from workflow
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 226, in __iter__
    for item in self.yielder():
  File "/home/osboxes/Workspace/secator/secator/runners/workflow.py", line 61, in yielder
    yield from results
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 644, in process_live_tasks
    for data in Runner.get_live_results(result):
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 535, in get_live_results
    yield from Runner.get_celery_results(result)
  File "/home/osboxes/Workspace/secator/secator/runners/_base.py", line 556, in get_celery_results
    get_task_ids(result, ids=task_ids)
  File "/home/osboxes/Workspace/secator/secator/runners/_helpers.py", line 98, in get_task_ids
    get_task_ids(result.parent, ids=ids)
  File "/home/osboxes/Workspace/secator/secator/runners/_helpers.py", line 98, in get_task_ids
    get_task_ids(result.parent, ids=ids)
  File "/home/osboxes/Workspace/secator/secator/runners/_helpers.py", line 94, in get_task_ids
    get_task_ids(child, ids=ids)
  File "/home/osboxes/Workspace/secator/secator/runners/_helpers.py", line 92, in get_task_ids
    if hasattr(result, 'children') and result.children:
                                       ^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/result.py", line 429, in children
    return self._get_task_meta().get('children')
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/result.py", line 442, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 608, in get_task_meta
    meta = self._get_task_meta_for(task_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 998, in _get_task_meta_for
    return self.decode_result(meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 425, in decode_result
    return self.meta_from_decoded(self.decode(payload))
                                  ^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 431, in decode
    return loads(payload,
           ^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 264, in loads
    with _reraise_errors(DecodeError):
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 45, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/exceptions.py", line 34, in reraise
    raise value.with_traceback(tb)
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 265, in loads
    return decode(data)
           ^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 337, in unpickle
    return pickle_loads(str_to_bytes(s))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/osboxes/Workspace/secator/.venv/lib/python3.11/site-packages/kombu/serialization.py", line 50, in pickle_loads
    return load(BytesIO(s))
           ^^^^^^^^^^^^^^^^
kombu.exceptions.DecodeError: pickle data was truncated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Has Testcase ✔ Status: Needs Verification ✘
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ask @thedrow @alzeih @vatsalshah1990 @auvipy @ocervell