How about: Instead of special-casing the manifest file (and having to deal with large vs small manifests), recursively treat the manifest as a downloadable artifact: If the manifest is small (1<MB), send the whole manifest in the response, otherwise send the manifest of the manifest.

I believe this would make public gateways way more useful by removing MITM risk (cc @gozala @autonome) ```suggestion 4. File identified using hash of the full content enables [validation of HTTP gateway responses](https://github.com/ipfs/in-web-browsers/issues/128) without running full IPFS stack, which allows for: - user agents such as web browsers: display integrity indicator when HTTP response matched the CID from the request - IoT devices: downloading firmware updates over HTTPS without the need for trusting a gateway or a CA ```

Decent deduplication requires using a consistent hash and chunking algorithm, and to deduplicate arbitrarily aligned data, the chunking algorithm must be content based with variably sized blocks of any byte-length between a min and max value. Every different way of chunking/hashing the data (and creating the merkle-tree) results in another copy of the data that cannot be deduplicated. To do deduplication efficiently you want IPFS to have a fixed chunking/hashing algorithm and merkle-tree under the hood, and then support alternative "views" on top of this, that can present the underlying data as if it was chunked/hashed differently. I don't know how much demand there is for alternative tree views, but certainly a common use-case is the "one big file with this hash" view. This could be implemented as just an alternative DHT entry similar to an IPNS entry that is keyed by the whole-file hash and points to a list of CID's (each different hash/chunker/tree options result in a different CID. ideally there is only one) for that file. These could be signed by the node that did the upload for verification purposes, but you would still need to download the whole file to verify the whole-file hash. I don't know how much demand there is for alternative tree views of the data, but this could be implemented using an alternative merkle tree containing the desired hash-type for each node, and where the "raw" leaves are actually ranged references into the underlying native merkle tree nodes. I'm not sure exactly how validation of these alerternative-view merkle nodes would work, but you would probably have to download the data-segment (by downloading the underlying merkle-tree-fragment) for each -node to validate the hash. There might be ways to include signatures by the uploading peer-node, but you probably want to do this in a way that the same alternative-view-uploaded by different peers can share the same data. Perhaps an IPNS entry pointing at the alternative-view-root-merkle node is the best way that peers can sign that the've uploaded/verified that alternative-view.

Note that breaking blocks on hash function chunk boundaries means breaking rabin (or any other content based) chunking, which is essential for efficient content deduplication. For this to support arbitrary content boundaries for IPFS blocks, the hash state will need to include more than just the accumulated hash value; it will also need the trailing partial hash chunk data.

sha 1 is deprecated / not-recommended at this point. It seems unclear it's valuable or safe to support it. why do we want to?

cc @ribasushi I feel like you may have some thoughts on this 😄

I'd like some 👀 on this ideally from people who are more practiced with this type of cryptanalysis than I am.