Skip to content

Commit

Permalink
ipip(0445): add skip-leaves
Browse files Browse the repository at this point in the history
  • Loading branch information
Jorropo committed Oct 11, 2023
1 parent 24861cd commit e7027d1
Show file tree
Hide file tree
Showing 2 changed files with 126 additions and 1 deletion.
22 changes: 21 additions & 1 deletion src/http-gateways/trustless-gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ The Body hash MUST match the Multihash from the requested CID.

A CAR stream for the requested
[application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
content type (with optional `order` and `dups` params), path and optional
content type (with optional `order`, `dups` and `skip-leaves` params), path and optional
`dag-scope` and `entity-bytes` URL parameters.

## CAR version
Expand Down Expand Up @@ -301,6 +301,26 @@ of their presence in the DAG or the value assigned to the "dups" parameter, as
the raw data is already present in the parent block that links to the identity
CID.

## CAR `skip-leaves` (content type parameter)

The `skip-leaves` parameter specifies whether blocks with the multicodec `raw`
`0x55` must be sent.

It accepts two values:
- `y`: Blocks with `raw` multicodec MUST NOT be sent.
- `n`, or unspecified: Blocks with `raw` multicodec MUST be sent.

A gateway MUST NOT assume this field is `y` if unspecified.
When not specified it always MUST be understood as `n`.

:::note Notes for implementers

A request which is rooted at a `raw` block and has `skip-leaves=y` does not
make sense and SHOULD NOT be sent by clients, it is fair for servers to
error in this situation.

:::

## CAR format parameters and determinism

The default header and block order in a CAR format is not specified by IPLD specifications.
Expand Down
105 changes: 105 additions & 0 deletions src/ipips/ipip-0445.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
---
title: "IPIP-0445: trustless gateway skip-leaves option"
date: 2023-10-09
ipip: open
editors:
- name: Hugo VALTIER
github: Jorropo
url: https://jorropo.net/
affiliation:
name: Protocol Labs
url: https://protocol.ai/
relatedIssues:
- https://github.com/ipfs/specs/issues/444
order: 445
tags: ['ipips']
---

## Summary

Introduce `skip-leaves` flag for the :cite[trustless-gateway].

## Motivation

Allow clients to read a stream which only contain proofs in a bottom heavy
graph using `raw` codec for it's leaves.

Usefull with unixfs for features like webseeds [#444](https://github.com/ipfs/specs/issues/444).

## Detailed design

The `skip-leaves` CAR Content-Type parameter on :cite[trustless-gateway]
allows clients to download an entity except blocks with the multicodec
`raw` (`0x55`).

- When set to `y`, the parameter instructs the gateway not to transmit
blocks tagged with the `raw` multicodec.
- If set to `n`, or left unspecified, the gateway MUST transmit `raw`
multicodec blocks.

Importantly, unless explicitly specified as `y`, the default operational
mode of the gateway MUST assume the value of `skip-leaves` to be `n`.

## Design rationale

### User Benefit

Implementing the `skip-leaves` parameter offers several benefits to users:

1. **Verification Flexibility:** Clients can verify out-of-band (OOB) received
files in their deserialized form without necessitating the transmission of
raw blocks from the gateway.
2. **Incremental Download:** Clients can incrementally download files in
deserialized forms from non-IPFS servers. Allowing applications to share
distribution for IPFS and non IPFS clients.
3. **Efficient Block Discovery:** With the `skip-leaves` option enabled,
clients can quickly discover numerous candidate blocks without being
bottlenecked by the gateway's transmission of raw blocks.

### Compatibility

Setting the default value of the `skip-leaves` parameter to `n` ensures
backward compatibility with existing clients and systems that are unaware
of this new flag.

### Prevention of Amplification Attacks and Efficient Server Operation

By utilizing the `raw` (`0x55`) codec servers can trivially determine whether
to fetch or skip a block without having to learn any new information.
Although more limited and not able to handle unixfs file using dag-pb for their
leaves, it allows both the client and server to trivially verify a block
must not be fetched. Preventing issues of Amplification where a server could
need to fetch multiple orders more data than the client when executing the
request.

### Why not `dag-scope=skip-leaves` ?

The `dag-scope` parameter determines the overall range of blocks to retrieve,
while `skip-leaves` selectively filters specific blocks within that range.
Combining them under one parameter would restrict their combined utility.

For example:
- A client is streaming a video from a webseed and the user seeked through the
video, then the client would send `dag-scope=entity&entity-bytes=42:1337`
with `skip-leaves=y` to download the proofs for the required section of the
video.
- A client is verifying an OOB transfered directory, then `dag-scope=all` with
`skip-leaves=y` makes sense.

### Alternatives

An alternative approach would be to request blocks individually.
However it adds extra round trips and more per HTTP request overhead
and thus is undesireable.

## Security

None.

## Test fixtures

TODO

### Copyright

Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

0 comments on commit e7027d1

Please sign in to comment.