Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
RPC chunks with streaming SSZ decoding, snappy frames, and stricter DOS limits where possible. #1606
This is a proposal to focus on length-encoding SSZ contents, enable streaming of chunk contents, and put stricter DOS limits in place.
So far I am aware that Prysm has non-streaming Snappy compression in place as option, and Lighthouse has a PR open that made a start on snappy support, but blocked by questions/ongoing network changes by Adrian. In other clients, such as Lodestar, Artemis and Nimbus, I could find the Snappy dependency, but no implementation of
The motivation here is that it is relatively quick to calculate the encoded length of an SSZ value, since fixed-length values are easily computed from the type, and list values often from multiplying a fixed-length element size with a list length.
Given a SSZ length, a SSZ decoder can directly read contents from the stream, avoiding the need for a temporary buffer. Additionally, snappy-frames avoid the need for more than a single frame of buffered bytes to decompress data (worst case
Additionally, this puts better DOS protection in place, as we can calculate the maximum size of a SSZ object, and should use it to be stricter about inputs. And given the worst-case snappy encoded length, we can account for compressed bytes even if we don't know the exact number of compressed bytes in advance. We don't need to know the exact number, as we can just verify the decompressed bytes instead, while checking the bytes read from the stream against the limits we should be checking regardless.
I think it's reasonable to use the framed format of snappy over the wire. It may not be super important for us now given current objects but could be more useful for larger objects in the future.
In principle, we could have a