Parallel compressed snapshot experiment

[ripatel-fd]

Introduces a new compression scheme for Solana snapshots that greatly speed up compression and decompression at virtually no compression ratio loss.

This compression scheme is backwards-compatible and can be decompressed by Agave nodes or any other decompression library, that is not aware of this format.

Several other nice properties for both compression and decompression:

• Requires no seek/skip ability (supports streaming-only input/output backends like pipes or HTTP servers without Range requests)
• Compresses at least twice as fast as Zstandard's CLI on the same box for a testnet snapshot
• Both compression and decompression is linearly scalable to line rate (10GB/s realistic)
• Zstandard frames are aligned to tar stream entries

The format is identified by a single skippable type 0 zstandard frame with the 8 byte user data 0xf212f209fd944ba2 (little endian u64)

Receipts:

# one fread() thread, one fwrite() thread, and 14 Zstandard compression threads
$ fd_snapmk_para --tile-cpus 1-16 --page-cnt 32 --in /data/r/dl2/snapshot-366945294-CDVQk4WDVHJ78bombiqBgDAasPoXofgRzurtvxvpRic2.tar --out /data/r/test.tar.zst --frame-sz 2000000 --depth 512
Compressed 31.126 GiB to 4.505 GiB in 7.866 s (4.249 GB/s, ratio 6.91)

$ zstd -l -v /data/r/test.tar.zst 
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
/data/r/test.tar.zst 
# Zstandard Frames: 8498
# Skippable Frames: 1
DictID: 0
Window Size: 2.00 MiB (2097152 B)
Compressed Size: 4.50 GiB (4836798741 B)
Check: None

$ time zstdmt -T16 -3 -o /data/r/test2.tar.zst /data/r/dl2/snapshot-366945294-CDVQk4WDVHJ78bombiqBgDAasPoXofgRzurtvxvpRic2.tar
/data/r/dl2/snapshot-366945294-CDVQk4WDVHJ78bombiqBgDAasPoXofgRzurtvxvpRic2.tar : 14.35%   (  31.1 GiB =>   4.47 GiB, /data/r/test2.tar.zst) 

real    0m16.533s
user    1m8.334s
sys     0m19.170s

$ zstd -l -v /data/r/test2.tar.zst
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
/data/r/test2.tar.zst 
# Zstandard Frames: 1
DictID: 0
Window Size: 2.00 MiB (2097152 B)
Compressed Size: 4.47 GiB (4796439520 B)
Decompressed Size: 31.1 GiB (33421735936 B)
Ratio: 6.9680
Check: XXH64 fdbdbdcb

$ zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

[github-actions]

Performance Measurements ⏳

Suite	Baseline	New	Change
backtest mainnet-368528500-perf per slot	0.056094 s	0.051048 s	-8.996% ✅
backtest mainnet-368528500-perf snapshot load	2.6 s	2.198 s	-15.462% ✅
backtest mainnet-368528500-perf total elapsed	56.09404 s	51.048495 s	-8.995% ✅
firedancer mem usage with mainnet.toml	1001.1 GiB	1001.1 GiB	0.000% ✅

[ripatel-fd]

Moving to #7406

[github-actions]

Performance Measurements ⏳

Suite	Baseline	New	Change
backtest mainnet-368528500-perf per slot	0.074174 s	0.073921 s	-0.341% ✅
backtest mainnet-368528500-perf snapshot load	3.792 s	3.259 s	-14.056% ✅
backtest mainnet-368528500-perf total elapsed	74.174239 s	73.921326 s	-0.341% ✅
firedancer mem usage with mainnet.toml	1001.23 GiB	1001.23 GiB	0.000% ✅