struct.error: pack_into requires a buffer of at least 130101 bytes

[grillazz]

Hi,

Have below error msg when testing celery==4.0.0rc4 with XML files

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/code/opta/management/commands/reload_opta_feed.py", line 63, in handle
    refresh_document.delay(feed, xml_body, options['feed_type'].capitalize())
  File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 395, in delay
    return self.apply_async(args, kwargs)
  File "/usr/local/lib/python3.5/site-packages/celery/app/task.py", line 510, in apply_async
    **options
  File "/usr/local/lib/python3.5/site-packages/celery/app/base.py", line 665, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/usr/local/lib/python3.5/site-packages/celery/app/amqp.py", line 547, in send_task_message
    **properties
  File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 176, in publish
    exchange_name, declare,
  File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 488, in _ensured
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/kombu/messaging.py", line 198, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/usr/local/lib/python3.5/site-packages/amqp/channel.py", line 1770, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/usr/local/lib/python3.5/site-packages/amqp/abstract_channel.py", line 62, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/usr/local/lib/python3.5/site-packages/amqp/method_framing.py", line 160, in write_frame
    3, channel, framelen, str_to_bytes(body), 0xce)
struct.error: pack_into requires a buffer of at least 130101 bytes

[ask]

Can reproduce using

from amqp import Connection, Message

queue = 'buffertest'

with Connection() as conn:
    with conn.channel() as channel:
        hugebody = 'X' * (conn.frame_max - 8)
        channel.exchange_declare(queue, 'direct')
        channel.queue_declare(queue)
        channel.queue_bind(queue, queue, queue)

        channel.basic_publish(Message(hugebody))

[ask]

conn.frame_max - 7 works as then the message is split into multiple frames, but there is some overhead not accounted for so anything between conn.frame_max - 60, and conn.frame_max - 8 crashes.

[ask]

Tested by sending messages in all sizes from 1 up to frame_size + 1000 🎉

[galuszkak]

Awesome! Thanks for help @ask !

[grillazz]

Thanks @ask, tested and now working well.

[zxh13]

I also encountered the same problem, how do you solve it? @grillazz