FRC: Piece Gateway

[willscott]

We have had a protocol that has been offered by Boost for a while, and has been used by several of the FIL+ deal making programs. This is for a piece gateway - an HTTP endpoint exposed on SPs allowing retrieval of a full piece of data.

The full semantics of this gateway protocol have not been fully specified. It seems useful to codify them at a specification level, rather than point to a single implementation.

To that end, I'm proposing an FRC defining what we mean by a piece gateway.
I put together a first pass as #709 . It parallels in many ways the block gateway used by IPFS in structure, and expected HTTP semantics.

[willscott]

From subsequent conversations, I'm planning to extend the semantics here to also specify transfer of sealed pieces.

Currently,
/piece/{piece cid} is used for unsealed transfers. My inclination is that to keep similar semantics extending the piece gateway to support sealed data transfer, we would specify that one of:

1. /sealed/{piece cid}
2. /piece/{piece cid}/sealed
3. /piece/{commR}

Would be interested in what others would prefer here

[ribasushi]

As discussed in slack - only 3 is viable. A single CommP can (and does) have multiple replicas within the same SP. When downloading a sealed replica you must pull a specific CommR so you can find the correct seed to regenerate its SDR counterpart.

TLDR: all sealed replicas are unique, thus need to be addressed by a unique CommR

[willscott]

I've updated the language of the proposed FRC to include reference to supporting the fetch of CommR

[xinaxu]

Should we include unpadded piece retrieval in this FRC. This would help reducing number of bytes needed to be downloaded to verify the integrity as the downloader can can do the padding locally when checking commp.
A similar function like https://boost.filecoin.io/http-retrieval#retrieving-a-car-file but under /piece route

[willscott]

I've noted that transport compression may be used - are you asking for a definition in the specification that server implementations must support specific forms of http transport compression?

[ribasushi]

@willscott given how common significant runs-of-\x00 will be in this context, we probably could throw a SHOULD in the spec... Not feeling strongly either way though

[xinaxu]

Reiterating my point, correct me if I'm wrong, there is no client supporting gzip transfer encoding, i.e. aria2c only supports gzip content compression, but not transfer compression.
How about a custom response header that indicates known range of non zeroes? So users can HEAD first and then do range request.

[Kubuxu]

@willscott It should be Content-Encoding (so content compression). Transfer compression is widely unsupported, as @xinaxu mentions.

We could go as far as requiring that compliant servers MUST support at least gzip content compression (and identity for no compression), and clients SHOULD support and use gzip in their Accept-Encoding header.

[xinaxu]

Thanks, using gzip content compression is good enough to some extent, though I'd complain that it won't work with multithreading - because it can't compress from the middle.
Not a blocker but in pratice when I distribute CAR files to my partner SPs, all of them are using multithreaded downloading clients which is almost a requirement for transfer cross continents.