From e7027d1fa2f95a432368aa60052d2eba5fbd8275 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Wed, 11 Oct 2023 06:18:01 +0200 Subject: [PATCH] ipip(0445): add skip-leaves --- src/http-gateways/trustless-gateway.md | 22 +++++- src/ipips/ipip-0445.md | 105 +++++++++++++++++++++++++ 2 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 src/ipips/ipip-0445.md diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index 949e2b0bf..cf7528743 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -214,7 +214,7 @@ The Body hash MUST match the Multihash from the requested CID. A CAR stream for the requested [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) -content type (with optional `order` and `dups` params), path and optional +content type (with optional `order`, `dups` and `skip-leaves` params), path and optional `dag-scope` and `entity-bytes` URL parameters. ## CAR version @@ -301,6 +301,26 @@ of their presence in the DAG or the value assigned to the "dups" parameter, as the raw data is already present in the parent block that links to the identity CID. +## CAR `skip-leaves` (content type parameter) + +The `skip-leaves` parameter specifies whether blocks with the multicodec `raw` +`0x55` must be sent. + +It accepts two values: +- `y`: Blocks with `raw` multicodec MUST NOT be sent. +- `n`, or unspecified: Blocks with `raw` multicodec MUST be sent. + +A gateway MUST NOT assume this field is `y` if unspecified. +When not specified it always MUST be understood as `n`. + +:::note Notes for implementers + +A request which is rooted at a `raw` block and has `skip-leaves=y` does not +make sense and SHOULD NOT be sent by clients, it is fair for servers to +error in this situation. + +::: + ## CAR format parameters and determinism The default header and block order in a CAR format is not specified by IPLD specifications. diff --git a/src/ipips/ipip-0445.md b/src/ipips/ipip-0445.md new file mode 100644 index 000000000..ce97d6d28 --- /dev/null +++ b/src/ipips/ipip-0445.md @@ -0,0 +1,105 @@ +--- +title: "IPIP-0445: trustless gateway skip-leaves option" +date: 2023-10-09 +ipip: open +editors: + - name: Hugo VALTIER + github: Jorropo + url: https://jorropo.net/ + affiliation: + name: Protocol Labs + url: https://protocol.ai/ +relatedIssues: + - https://github.com/ipfs/specs/issues/444 +order: 445 +tags: ['ipips'] +--- + +## Summary + +Introduce `skip-leaves` flag for the :cite[trustless-gateway]. + +## Motivation + +Allow clients to read a stream which only contain proofs in a bottom heavy +graph using `raw` codec for it's leaves. + +Usefull with unixfs for features like webseeds [#444](https://github.com/ipfs/specs/issues/444). + +## Detailed design + +The `skip-leaves` CAR Content-Type parameter on :cite[trustless-gateway] +allows clients to download an entity except blocks with the multicodec +`raw` (`0x55`). + +- When set to `y`, the parameter instructs the gateway not to transmit + blocks tagged with the `raw` multicodec. +- If set to `n`, or left unspecified, the gateway MUST transmit `raw` + multicodec blocks. + +Importantly, unless explicitly specified as `y`, the default operational +mode of the gateway MUST assume the value of `skip-leaves` to be `n`. + +## Design rationale + +### User Benefit + +Implementing the `skip-leaves` parameter offers several benefits to users: + +1. **Verification Flexibility:** Clients can verify out-of-band (OOB) received + files in their deserialized form without necessitating the transmission of + raw blocks from the gateway. +2. **Incremental Download:** Clients can incrementally download files in + deserialized forms from non-IPFS servers. Allowing applications to share + distribution for IPFS and non IPFS clients. +3. **Efficient Block Discovery:** With the `skip-leaves` option enabled, + clients can quickly discover numerous candidate blocks without being + bottlenecked by the gateway's transmission of raw blocks. + +### Compatibility + +Setting the default value of the `skip-leaves` parameter to `n` ensures +backward compatibility with existing clients and systems that are unaware +of this new flag. + +### Prevention of Amplification Attacks and Efficient Server Operation + +By utilizing the `raw` (`0x55`) codec servers can trivially determine whether +to fetch or skip a block without having to learn any new information. +Although more limited and not able to handle unixfs file using dag-pb for their +leaves, it allows both the client and server to trivially verify a block +must not be fetched. Preventing issues of Amplification where a server could +need to fetch multiple orders more data than the client when executing the +request. + +### Why not `dag-scope=skip-leaves` ? + +The `dag-scope` parameter determines the overall range of blocks to retrieve, +while `skip-leaves` selectively filters specific blocks within that range. +Combining them under one parameter would restrict their combined utility. + +For example: +- A client is streaming a video from a webseed and the user seeked through the + video, then the client would send `dag-scope=entity&entity-bytes=42:1337` + with `skip-leaves=y` to download the proofs for the required section of the + video. +- A client is verifying an OOB transfered directory, then `dag-scope=all` with + `skip-leaves=y` makes sense. + +### Alternatives + +An alternative approach would be to request blocks individually. +However it adds extra round trips and more per HTTP request overhead +and thus is undesireable. + +## Security + +None. + +## Test fixtures + +TODO + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).