Snapshots compression for import/export/download

[LesnyRumcajs]

Issue summary

We could speed up things by downloading compressed snapshots. Mainnet compressed snapshots are significantly smaller (70G vs 130G), and it would make a huge difference for bandwidth-constrained users.

Task summary

We should handle both importing and exporting compressed snapshots. Compression could be on by default. This must be configurable via configuration and/or flags.

• add zstd decompression functionality for importing snapshots. A potential crate to use https://crates.io/crates/zstd
• add a configuration option to use compressed snapshots for downloading (both the forest and the forest-cli snapshot fetch) and importing,
• add zstd compression functionality for exporting snapshots.
• update the snapshot service to upload both compressed and uncompressed files.
• Modify forest and forest-cli to prefer compressed files #2873
• Directly use the stable snapshot links rather than scanning S3 #2874
• add a configuration option to use compression when exporting,

Acceptance Criteria

• Limit disk space usage. That is to say; it should be possible to write directly to the zstd stream and not create a car file, compress it and then delete it.
• Forest can export compressed snapshots,
• Forest can import compressed snapshots,
• Forest can download compressed snapshots,
• forest-cli can download compressed snapshots (right now it's downloading uncompressed ones only)

Other information and links

[elmattic]

I like it.

[lemmih]

@hanabi1224 Is this done?

[hanabi1224]

@lemmih updated the checkboxes, modify forest and forest-cli to prefer compressed files one is not done

[lemmih]

Done.