Snapshot files are not valid zstd archives

[lemmih]

Describe the bug

To Reproduce Steps to reproduce the behavior:

1. Download a Forest snapshot: forest-cli --chain calibnet snapshot fetch
2. Verify its integrity: zstd --test forest_snapshot_calibnet_2023-06-19_height_660943.car.zst (this fails)
3. Forest thinks everything is okay: forest-cli --chain calibnet snapshot validate forest_snapshot_calibnet_2023-06-19_height_660943.car.zst

Log output

Log Output

$ zstd --test forest_snapshot_calibnet_2023-06-19_height_660943.car.zst 
eight_660943.car.zst : 2190 MB...     eight_660943.car.zst : Read error (39) : premature end

$ forest-cli --chain calibnet snapshot validate forest_snapshot_calibnet_2023-06-19_height_660943.car.zst 
✔ This will result in using approximately 1386 MB of data. Proceed? · yes
2023-06-19T06:56:25.927370Z  INFO forest_filecoin::genesis: Initialized genesis: BlockHeader: Cid(bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4)    
2023-06-19T06:56:25.929039Z  INFO forest_filecoin::utils::net::download: Loading forest_snapshot_calibnet_2023-06-19_height_660943.car.zst, is_zstd_compressed: true    
280.72 MB/s 0s 2023-06-19T06:56:34.867571Z  INFO forest_filecoin::utils::db: Buffered write completed: total entries: 2577964, total size: 2.58GB, took: 8s    
Snapshot is valid

Expected behaviour

The zst snapshot files should not prematurely end.

Screenshots

Environment (please complete the following information):

• OS: Linux
• Rust version(e.g. rustc --version): default toolchain
• Branch/commit: latest main

Other information and links

[ruseinov]

@lemmih As far as I understand this issue entails:

1. Making sure that the downloaded snapshots are correct.
2. Snapshot validation should fail given a corrupt snapshot.

[lemmih]

@lemmih As far as I understand this issue entails:

1. Making sure that the downloaded snapshots are correct.

2. Snapshot validation should fail given a corrupt snapshot.

The snapshots are not corrupted during upload or download. We generate them incorrectly (ie. if you generate a fresh calibnet snapshot, zstd --test will tell you it is invalid). Older snapshots appear to be fine so this might be a bug that was introduced recently. Ideally, a bug fix would include updates to forest-cli snapshot validate to not accept broken files but it isn't strictly necessary.

• Fix bug in snapshot generation code.
• Add CI check for broken zst files. That is, add a zstd --test step after we generate a snapshot.
• (Optional) Update forest-cli snapshot validate to detect incorrectly compressed files.