ValueError: Too many packets in payload after update to v3.10.0

[vgonisanz]

I am sending a file using fixed size chunks (binary data, with size=1024 in example) using socket.io from a client to a flask_socketio server emit:

sio.emit("chunk", data=content, namespace="audio")

The audio source is just a wave file, the snippet of the generator is:

chunk_size = 1024
delay = 0.0

while data != b'':
    yield data
    data = wave_file.readframes(chunk_size)
    time.sleep(delay)
    break

Until now, the data arrived without problem. After update python-engineio to v3.10.0, if delay = 0.0, sometime server print the following error:

2019-10-24T07:47:42.708635Z [error    ] post request handler error     event_id=146cb3c88 host=vgonisanz-linux source=engineio.server user=vgonisanz version=dl:0.1.0
Traceback (most recent call last):
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/server.py", line 394, in handle_request
    socket.handle_post_request(environ)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/socket.py", line 123, in handle_post_request
    p = payload.Payload(encoded_payload=body)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/payload.py", line 15, in __init__
    self.decode(encoded_payload)
  File "/home/vgonisanz/miniconda3/envs/foo3.6/lib/python3.6/site-packages/engineio/payload.py", line 61, in decode
    raise ValueError('Too many packets in payload')
ValueError: Too many packets in payload

This error didn't appear before updating. Also, If I set up delay as 0.01, it don't appear. It seems the change comes from commit c8407ae, and the issue is related with the time between requests.

• Why is this happening now?
• It is possible to continue use delay = 0 (for testing purpose I want to run my tests faster as I can)?

Thanks.

[miguelgrinberg]

This change that I've made is for security purposes. Decoding packets from payloads is somewhat expensive, and I have rarely seen payloads with lots of small packets, except when someone is trying to get the server to break. If you want to bypass this measure, you can do something like this:

from engineio.payload import Payload

Payload.max_decode_packets = 50

You can raise the 50 to the number that you need. The value that I selected prevent high CPU usage is 16, which I thought was more than enough, but I'm open to review it and raise if a lot of people get tripped by this.

[vgonisanz]

It would be possible to make this parameter configurable? If this value is provided as parameter when Engine.IO client is created (or configurable in other way), it will allow users to fit to their needs.

In my use case, I'm using flask_socketio to create a communication to process audio and I would prefer to no modify dependency source code. I already config ping timeout and ping interval when SocketIO object is created:

from flask_socketio import SocketIO
socketio = SocketIO(async_mode='gevent', ping_timeout=cfg.service.PING_TIMEOUT, ping_interval=cfg.service.PING_INTERVAL)

[miguelgrinberg]

Oh no, sorry I did not make myself clear on this. This was not about modifying the dependency. Just add that to your application before your create your socketio object. Something like this:

from flask_socketio import SocketIO
from engineio.payload import Payload

Payload.max_decode_packets = cfg.service.ENGINEIO_MAX_DECODE_PACKETS
socketio = SocketIO(async_mode='gevent', ping_timeout=cfg.service.PING_TIMEOUT, ping_interval=cfg.service.PING_INTERVAL)

[vgonisanz]

This solution works perfectly. In my use case require ENGINEIO_MAX_DECODE_PACKETS = 500 to work with a 0 delay. Thanks.

[VeNoMouS]

Came here for the same issue, w00t for resolve :)

[IsaacHub]

@miguelgrinberg Could you give the code for "cfg" and the import statement?
cfg.service.ENGINEIO_MAX_DECODE_PACKETS

[vgonisanz]

That cfg is not from flask_socketio, it just a pydantic struct I use in the project to manage all configuration. It works in the same way with this snippet:

from engineio.payload import Payload

Payload.max_decode_packets = 500

[TiankaiHang]

Hello,
I want send video frame via socket-io.
I get the binary code of the frame as a string and send it to server. However, i increase ENGINEIO_MAX_DECODE_PACKETS to 50000, it still doesn't work ... After a long time, i could get one frame...

How to set the parameter ENGINEIO_MAX_DECODE_PACKETS for frame / image data?

Best.

[miguelgrinberg]

This is how you have to do it:

from engineio.payload import Payload

Payload.max_decode_packets = 50

[TiankaiHang]

This is how you have to do it:

from engineio.payload import Payload

Payload.max_decode_packets = 50

Yep, i have tried that. I will still meet the error. Too many ...

When i increase it to 500 or 5000, it can work, but the speed is so slow (send a string of picture to server and send it back to client)

[miguelgrinberg]

@TiankaiHang it's probably slow because you are sending hundreds/thousands of little packets.

[TiankaiHang]

@miguelgrinberg Hi, yes, i want to transfer image data from browser to server and process it on server, and send it back to browser(client). If i send one frame by one frame, it would be very slow, and some data may be blocked. Any ideas about that?

If I use local server (127.0.0.1), it my be good. However, it could be slow when I use 0.0.0.0.

Based on flask and webrtc.

Thanks.

[miguelgrinberg]

@TiankaiHang Have you considered using WebSocket? I can't really give you any more details, you may be stressing the network beyond what it can do for you. This isn't a problem with this package.

[TiankaiHang]

@miguelgrinberg Thanks a lot! I have tried websocket. Thanks again for your reply!

[seandepagnier]

I hit this "too many packets in payload" as well. An chance the error could say how many packets? eg:
raise ValueError('Too many packets in payload ' + str(len(encoded_packets)))