Incorrect progress bar when importing compressed snapshots

[lemmih]

Describe the bug

To Reproduce Steps to reproduce the behavior:

1. Remove any mainnet data: forest-cli db clean
2. Import a snapshot: forest --encrypt-keystore false --halt-after-import --auto-download-snapshot
3. Notice how forest appears to freeze after the progress bar hits 100%.

(Note, the same can be observed using a calibnet snapshot, but the effect is less noticeable.)

Expected behaviour

The progress bar is configured such that 100% is the size of the compressed snapshot but, when running, it counts the decompressed bytes. Since our compression factor is roughly 2x, when the progress bar hits 100%, only about half of the compressed snapshot has been consumed.

The progress bar should show how much of the snapshot file we've consumed.

Environment (please complete the following information):

• OS: Linux
• Rust version(e.g. rustc --version): rustc 1.70.0 (90c541806 2023-05-31)
• Branch/commit: main/3e06cd98709ff2ef6ec11d9e943482acc686edf8

Other information and links

[aatifsyed]

Probably introduced in #2893

[aatifsyed]

David, am I correct in remembering you wanted to remove progress bars altogether?

[ruseinov]

@lemmih Am I correct to assume that we actually want to decompress it first and then show progress based on decompressed size?

[lemmih]

Nope, that would take up a lot of space. We want to process the data immediately as it is being uncompressed. The progress bar can display how much of the compressed file we've consumed.

[lemmih]

David, am I correct in remembering you wanted to remove progress bars altogether?

They should be changed to use the logging framework instead of printing terminal codes. Let's fix this bug first.

[ruseinov]

2. forest --encrypt-keystore false --halt-after-import --auto-download-snapshot

I did not ask the question I meant to ask, the question here is: how do we come by the uncompressed file size? I'll figure this out.

[ruseinov]

Looks like the best way is to parse zstd frame header in hopes that it stored the original file size, which is optional.

[ruseinov]

Yeah, unfortunately this is not re-exported by async-compression, but we can add a zstd-safe dependency manually as it won't grow the bin size.