Verify param files that are downloaded via ipfs http gateway

[hanabi1224]

Issue summary

Currently, a forest daemon fetches param files via ipfs http gateway without content verification, while IPFS http gateway now offers a trustless mode (SPEC) in which content is downloaded in car format so that a client can verify the integrity of the downloaded content.

And here is a web demo.

Related forest code: https://github.com/ChainSafe/forest/blob/main/utils/paramfetch/src/lib.rs#L20

Other information and links

https://github.com/ipfs/specs/blob/main/http-gateways/TRUSTLESS_GATEWAY.md
https://github.com/2color/verified-ipfs-retrieval

[dapplion]

Just published rs-car as first building block for this issue https://crates.io/crates/rs-car

I found on crates that there's a brand new implementation for CAR format in https://github.com/blocklessnetwork/rust-car will look into it too

Such an interesting coincidence that both packages are back to back in the docs queue 🤣 (see items 5 and 6 in the build queue)

[dapplion]

@hanabi1224 What's the usage pattern of paramfetch crate?

• How often are files downloaded (on start-up, on network event, on interval)
• How many files are downloaded?
• What's the size distribution of those files?

[hanabi1224]

@dapplion

How often are files downloaded (on start-up, on network event, on interval)

only for the first run

How many files are downloaded?

23 files (as of writing)

What's the size distribution of those files?

1 file at ~293MB, 2 at ~2MB, others at <1MB

[dapplion]

Ok so it would be ok to buffer in memory? I assume the eventual memory required by the node after startup is more than ~500MB

[hanabi1224]

We need ~8GB memory in total so ~500MB is not a big deal. The largest file (Qmdq44DjcQnFfU3PJcdX7J49GCqcUYszr1TxMbHtAkvQ3g) is defined here if u want to do some manual testing

[LesnyRumcajs]

Re-opening. This caused hash mismatches that unfortunately passed the CI so I had to revert 2b5515d

https://github.com/ChainSafe/forest/actions/runs/4374624518/jobs/7654288564

2023-03-09T12:57:28.367395Z ERROR forest_paramfetch: Error fetching param file v28-empty-sector-update-poseidon-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-empty-sector-update-poseidon-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.vk". (786a02f742015903c6c6fd852552d272 != 1ac05784f304129f74c5184190c1ec78)    
2023-03-09T12:57:28.368362Z ERROR forest_paramfetch: Error fetching param file v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk". (786a02f742015903c6c6fd852552d272 != dc1ade9929ade1708238f155343044ac)    
2023-03-09T12:57:28.369708Z ERROR forest_paramfetch: Error fetching param file v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-61fa69f38b9cc771ba27b670124714b4ea77fbeae05e377fb859c4a43b73a30c.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-61fa69f38b9cc771ba27b670124714b4ea77fbeae05e377fb859c4a43b73a30c.vk". (786a02f742015903c6c6fd852552d272 != 994c5b7d450ca9da348c910689f2dc7f)    
2023-03-09T12:57:28.375[40](https://github.com/ChainSafe/forest/actions/runs/4374624518/jobs/7654288564#step:6:41)2Z ERROR forest_paramfetch: Error fetching param file v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk". (786a02f7[42](https://github.com/ChainSafe/forest/actions/runs/4374624518/jobs/7654288564#step:6:43)015903c6c6fd852552d272 != b687beb9adbd9dabe265a7e3620813e4)    
2023-03-09T12:57:28.375880Z ERROR forest_paramfetch: Error fetching param file v28-empty-sector-update-merkletree-poseidon_hasher-8-8-2-102e1444a7e9a97ebf1e3d6855dcc77e66c011ea66f936d9b2c508f87f2f83a7.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-2-102e1444a7e9a97ebf1e3d6855dcc77e66c011ea66f936d9b2c508f87f2f83a7.vk". (786a02f742015903c6c6fd852552d272 != 80e366df2f1011953c2d01c7b7c9ee8e)    
2023-03-09T12:57:28.376960Z ERROR forest_paramfetch: Error fetching param file v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk". (786a02f742015903c6c6fd852552d272 != 065179da19fbe515507267677f02823e)    
2023-03-09T12:57:45.342201Z ERROR forest_paramfetch: Error fetching param file v28-fil-inner-product-v1.srs: Checksum mismatch in param file "/home/ubuntu/.local/share/forest/filecoin-proof-parameters/v28-fil-inner-product-v1.srs". (861219731e04cf1a3[43](https://github.com/ChainSafe/forest/actions/runs/4374624518/jobs/7654288564#step:6:44)766be8e66f471 != ae20310138f5ba81[45](https://github.com/ChainSafe/forest/actions/runs/4374624518/jobs/7654288564#step:6:46)1d723f858e3797)

[dapplion]

Rethinking about this issue, if you already have a hash to check the integrity of the file, the statement from the original is not true

Currently, a forest daemon fetches param files via ipfs http gateway without content verification

As long as the hash is vetted by you and distributed in the binary you should be fine. Is that the case?

[LesnyRumcajs]

@dapplion yeah, they are in the binary.

❯ strings forest | grep 1ac05784f304129f74c5184190c1ec78
        "digest": "1ac05784f304129f74c5184190c1ec78",
    "digest": "1ac05784f304129f74c5184190c1ec78",

@hanabi1224 Could you please rephrase what you had in mind? Is the verification that we have in place now enough?

[hanabi1224]

@LesnyRumcajs @dapplion IMHO downloading from IPFS in trustless mode is in general a more robust and simple solution than doing verifications on our own, e.g. we might want to download state migration bundles from IPFS as well in the future. If the lib can be fixed in a way that the reconstructed file matches original file byte-to-byte, we could remove our own verification code, does that make sense?

[hanabi1224]

@LesnyRumcajs @dapplion I did some investigation and found that the issue only happens to very small files (<5KiB) when BufWriter is used, see https://github.com/ChainSafe/forest/pull/3244/files#diff-734f459e0440e878447df521e861ef2ae155d2ced12b3ac144bffd60cb3989d4R44

[LesnyRumcajs]

Hmm, it seems like an issue with the crate itself, right?

[dapplion]

Thanks for looking into this @hanabi1224 whenever I have bandwidth I can add tests for this usage pattern upstream. Can you confirm that now all config files big and small match the expected digest?

[hanabi1224]

@dapplion Yes, it works now and has been merged: #3244 After some investigation, I think it's rather a misuse of BufWriter than an issue with rs-car-ipfs crate