feat: compress state witness

[pugachAG]

This PR adds state witness compression as well as metrics around it. See #10780 for the big picture analysis.

We explicitly limit the size of the decompressed state witness to 64MB to handle Zip bomb attack. This is implemented by using BufMut and Limit along with zstd::stream::copy_decode, so it fails when attempting to write data beyond the limit.

Compression reduces state witness size particularly well for large state witnesses (containing many ContractCode values), which makes it worthwhile. For cases when compression doesn't yield much improvements the latency overhead is not significant.

In practice shadow validation was used to verify the statements above with the current mainnet traffic:

• overall traffic reduction (dashboard) is about 16% which is not great. That makes sense considering that after resharding to 6 shards the size of state witnesses per shard dropped significantly which makes compression less effective.
• max state witness size (uncompressed and compressed dashboards) reduced a lot for large state witnesses. We didn't observe any witnesses larger than 2.5MB in compressed state while uncompressed ones were as big as 6.6MB.
• additional latency (avg and distribution dashboards) is mostly comes from encoding and is in a 10-20ms range for the most busy shard 2.

This PR also includes the following changes:

• reduce state witness size histogram step
• bump actix-http version to avoid bringing multiple version of zstd dependency (zulip thread)

[codecov]

Codecov Report

Attention: Patch coverage is 77.09924% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 71.08%. Comparing base (4c0aa98) to head (2b40576).
Report is 33 commits behind head on master.

Files	Patch %	Lines
...client/src/stateless_validation/shadow_validate.rs	0.00%	22 Missing ⚠️
core/primitives/src/stateless_validation.rs	90.90%	0 Missing and 4 partials ⚠️
...nt/src/stateless_validation/chunk_validator/mod.rs	83.33%	0 Missing and 2 partials ⚠️
...src/stateless_validation/state_witness_producer.rs	91.30%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10715      +/-   ##
==========================================
- Coverage   71.33%   71.08%   -0.26%     
==========================================
  Files         760      761       +1     
  Lines      152288   153042     +754     
  Branches   152288   153042     +754     
==========================================
+ Hits       108640   108790     +150     
- Misses      39176    39814     +638     
+ Partials     4472     4438      -34     

Flag	Coverage Δ
backward-compatibility	0.24% <0.00%> (-0.01%)	⬇️
db-migration	0.24% <0.00%> (-0.01%)	⬇️
genesis-check	1.43% <0.00%> (-0.01%)	⬇️
integration-tests	36.79% <54.19%> (-0.18%)	⬇️
linux	69.52% <26.71%> (-0.29%)	⬇️
linux-nightly	70.56% <77.09%> (-0.24%)	⬇️
macos	54.26% <26.71%> (-0.04%)	⬇️
pytests	1.66% <0.00%> (-0.01%)	⬇️
sanity-checks	1.44% <0.00%> (-0.01%)	⬇️
unittests	66.79% <62.59%> (-0.19%)	⬇️
upgradability	0.29% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shreyan-gupta]

Looks good! 😄

[shreyan-gupta]

qq: why do we need two versions of zstd, 0.12.3 and 1.5.2? Is that how I read it?

[pugachAG]

Not idea, Cargo.lock is automatically generated 🤷‍♂️
@nagisa can you please help here

[shreyan-gupta]

Don't think it matters too much. We good!

[nagisa]

The one version is the Rust crate, the other is the bundled libzstd. And this now shows up because we have duplicate dependency (one of other crates that we depend on still depends on the older version.)

[pugachAG]

@shreyan-gupta also see this zulip thread

[jancionear]

Is it possible to construct Zip bombs with zstd? I'm worried that someone might be able to send something that would decompress to multiple terabytes in size, which would crash the node.

[shreyan-gupta]

Interesting.... I'm working on dividing witness into parts and sending over network and I would probably require some sort of header + body type setup. We can consider using the header for basic validation and only then using the body for unzip + full validation.

[pugachAG]

@jancionear this is a very good point, thanks a lot for highlighting that! I will look into this.

[pugachAG]

addressed in 9ad0851

[jancionear]

LGTM 👍

[jancionear]

It's a bit risky to limit state witness size to 64MB, what if there's some corner case that would blow up the size above that? I have flashbacks from #10615 x.x

AFAIK we have a 500MB limit for incoming network messages, maybe that'd be a bit safer? But at the end of the day we have to choose some value, so maybe 64MB is okay.

[pugachAG]

Good point, I will change that to match the network message size limit.

[nagisa]

If the network messages are limited to 500M, aren't these witnesses already limited to that amount?

[pugachAG]

Compressed state witness is indeed limited by the network message size, but at this point we haven't verified that data yet, so it can be malicious which could result in large memory usage when decompressed and crash the node. That is why we also need to explicitly limit the size of decompressed data here.

[pugachAG]

@jancionear the limit was updated in 94a406b

[nagisa]

Our past use of plain type aliases has proven to result in a not-great experience: as the codebase expanded, there have been cases of type mixups that had led to bugs.

On the other hand, this is currently used as a return type only where an usize would otherwise be perfectly adequate, so I think this is probably alright to leave as-is.

[nagisa]

How large are these witnesses? If they are of appreciable size, I would recommend streaming this data instead (https://docs.rs/borsh/latest/borsh/fn.to_writer.html and https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder.html seem relevant.)

[pugachAG]

It can be as large as max network message size which is 500MB :(
Streaming is a great idea, I've tried implementing it and counting written bytes appeared to somewhat non-trivial.
I think it is not critical for MVP, so I'm merging it as it is for now as agreed here. Also I've created #11064 to address this in the future.

[nagisa]

If the network messages are limited to 500M, aren't these witnesses already limited to that amount?

[nagisa]

Similarly, I would opt for a reader interface here, to avoid having 3 full copies of the data (although one of them is compressed, but still) in memory at the same time. You can still limit the amount of data decompressed by having a layer of Read implementation that counts the bytes and passes the data along to the outer reader up to the limit.

[pugachAG]

Same as this.