From bacd7d4700c931596b3eb8f688d63239ed80f59c Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 7 Oct 2022 13:15:13 +0200 Subject: [PATCH 01/46] wip: cbor and json ipip --- .../0000-gateway-json-cbor-response-format.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 IPIP/0000-gateway-json-cbor-response-format.md diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md new file mode 100644 index 00000000..96e5fb00 --- /dev/null +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -0,0 +1,47 @@ +# IPIP 0000: Gateway JSON and TAR Response Formats + +- Start Date: (format: 2022-10-10) +- Related Issues: + - [ipfs/specs/pull/288](https://github.com/ipfs/specs/pull/288) + - [ipfs/kubo/issues/8823](https://github.com/ipfs/kubo/issues/8823) + - [ipfs/kubo/pull/9335](https://github.com/ipfs/kubo/pull/9335) + +## Summary + +Add JSON and CBOR response formats to the [HTTP Gateway](../http-gateways/). + +## Motivation + +TODO + +## Detailed design + +TODO + +## Test fixtures + +TODO + +## Design rationale + +TODO + +### User benefit + +TODO + +### Compatibility + +This RFC is backwards compatible. + +### Security + +TODO + +### Alternatives + +TODO + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file From 72b61e6e2124cff7dd17b1787d19af6d7f1520f9 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 7 Oct 2022 13:15:50 +0200 Subject: [PATCH 02/46] wip: update related issue --- IPIP/0000-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 96e5fb00..5a5f6329 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -2,7 +2,7 @@ - Start Date: (format: 2022-10-10) - Related Issues: - - [ipfs/specs/pull/288](https://github.com/ipfs/specs/pull/288) + - [ipfs/specs/pull/328](https://github.com/ipfs/specs/pull/328) - [ipfs/kubo/issues/8823](https://github.com/ipfs/kubo/issues/8823) - [ipfs/kubo/pull/9335](https://github.com/ipfs/kubo/pull/9335) From 9c2aa8eb0e047f83ed8774b7d504f4970d1f709e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 11:11:47 +0200 Subject: [PATCH 03/46] add new json and cbor response formats --- http-gateways/PATH_GATEWAY.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 35ad96dd..132f4e86 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -1,6 +1,6 @@ # Path Gateway Specification -![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) +![Status: Work In Progress](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) **Authors**: @@ -181,10 +181,10 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned - +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/) +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/) +- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, data will be serialized as JSON +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, data will be serialized as CBOR ### `Range` (request header) @@ -246,9 +246,13 @@ parameter, if present) Optional, `format=` can be used to request specific response format. -This is a URL-friendly alternative to sending -`Accept: application/vnd.ipld.` header, see [`Accept`](#accept-request-header) -for more details. +This is a URL-friendly alternative to sending an [`Accept`](#accept-request-header) header. These are the equivalents: +- `format=raw` → `Accept: application/vnd.ipld.raw` +- `format=car` → `Accept: application/vnd.ipld.car` +- `format=dag-json` → `Accept: application/vnd.ipld.dag-json` +- `format=dag-cbor` → `Accept: application/vnd.ipld.dag-cbor` +- `format=json` → `Accept: application/json` +- `format=cbor` → `Accept: application/cbor` - -# Appendix: notes for implementers ## Content resolution From 2936f85f9be3e3fd479d953b3e856e2afd2619ea Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 11:13:00 +0200 Subject: [PATCH 04/46] add removed header --- http-gateways/PATH_GATEWAY.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 132f4e86..cb1f639d 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -594,6 +594,8 @@ Data sent with HTTP response depends on the type of requested IPFS resource: - CAR - CAR file or stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) +# Appendix: notes for implementers + ## Content resolution Content resolution is a process of turning an HTTP request into an IPFS content From bf97587187248ed017a6889ab06428d55f1479e4 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 11:38:30 +0200 Subject: [PATCH 05/46] add summary and motivation --- .../0000-gateway-json-cbor-response-format.md | 40 +++++++++++++++---- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 5a5f6329..533bd4a1 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -1,18 +1,32 @@ # IPIP 0000: Gateway JSON and TAR Response Formats -- Start Date: (format: 2022-10-10) +- Start Date: 2022-10-07 - Related Issues: - - [ipfs/specs/pull/328](https://github.com/ipfs/specs/pull/328) - - [ipfs/kubo/issues/8823](https://github.com/ipfs/kubo/issues/8823) - - [ipfs/kubo/pull/9335](https://github.com/ipfs/kubo/pull/9335) + - [ipfs/in-web-browsers/issues/182] + - [ipfs/specs/pull/328] + - [ipfs/kubo/issues/8823] + - [ipfs/kubo/pull/9335] + - [ipfs/go-ipfs/issues/7552] ## Summary -Add JSON and CBOR response formats to the [HTTP Gateway](../http-gateways/). +Add [DAG-JSON], [DAG-CBOR], JSON and CBOR response formats to the [HTTP Gateway](../http-gateways/). ## Motivation -TODO +Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] +formats. However, it should be possible to download everything over the gateway +in order to unlock the potentials of the [IPLD Data Model][ipld-data-model]. + +For example, DAG-PB has a [logical format][dag-pb-format] which makes it possible +to represent a DAG-PB directory as a [DAG-JSON] document. This means that, if we +support DAG-JSON in the gateway, then we would support +[JSON responses for directory listings][ipfs/go-ipfs/issues/7552], which has been +requested by our users in the past. + +In addition, this functionality is already present on the current Kubo CLI. By +bringing it to the gateways, we provide users with more power when it comes +to traversing IPLD data. ## Detailed design @@ -44,4 +58,16 @@ TODO ### Copyright -Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + +[DAG-PB]: https://ipld.io/docs/codecs/known/dag-pb/ +[dag-pb-format]: https://ipld.io/specs/codecs/dag-pb/spec/#logical-format +[DAG-JSON]: https://ipld.io/docs/codecs/known/dag-json/ +[DAG-CBOR]: https://ipld.io/docs/codecs/known/dag-json/ +[CAR]: https://ipld.io/specs/transport/car/ +[ipld-data-model]: https://ipld.io/docs/data-model/ +[ipfs/in-web-browsers/issues/182]: https://github.com/ipfs/in-web-browsers/issues/182 +[ipfs/specs/pull/328]: https://github.com/ipfs/specs/pull/328 +[ipfs/kubo/issues/8823]: https://github.com/ipfs/kubo/issues/8823 +[ipfs/kubo/pull/9335]: https://github.com/ipfs/kubo/pull/9335 +[ipfs/go-ipfs/issues/7552]: https://github.com/ipfs/go-ipfs/issues/7552 \ No newline at end of file From c442b6ac78c5294d247c5b137d991d3e60605c4c Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 11:44:02 +0200 Subject: [PATCH 06/46] add detailed design --- IPIP/0000-gateway-json-cbor-response-format.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 533bd4a1..e8921824 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -30,7 +30,9 @@ to traversing IPLD data. ## Detailed design -TODO +The solution is to allow the Gateway to support serializing an IPLD model +representation as [DAG-JSON], [DAG-CBOR], JSON and CBOR by requesting them +using either the `Accept` HTTP header or the `format` URL query. ## Test fixtures From f0ca426bdd5aece2bae83ccba8cc0488b91219d0 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 11:45:00 +0200 Subject: [PATCH 07/46] add design rationality and compatibility --- IPIP/0000-gateway-json-cbor-response-format.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index e8921824..be5bdfc2 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -40,7 +40,9 @@ TODO ## Design rationale -TODO +The current gateway already supports different response formats via the +`Accept` HTTP header and the `format` URL query. This IPIP proposes adding +one more supported format to that list. ### User benefit @@ -48,7 +50,7 @@ TODO ### Compatibility -This RFC is backwards compatible. +This IPIP is backwards compatible. ### Security From bed55154f67d5f0c644e84c0814fb2fd7f2df640 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 18 Oct 2022 12:01:47 +0200 Subject: [PATCH 08/46] add some user benefit --- IPIP/0000-gateway-json-cbor-response-format.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index be5bdfc2..67276f4b 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -46,7 +46,9 @@ one more supported format to that list. ### User benefit -TODO +The user benefits from this change as they will now be able to retrieve +content encoded in the traversable DAG-JSON and DAG-CBOR formats. This is +something that has been [requested before][ipfs/go-ipfs/issues/7552]. ### Compatibility From be230bd31fe0e59c4a4508c6a2eda8b0d5af0c03 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 11:32:53 +0200 Subject: [PATCH 09/46] Update IPIP/0000-gateway-json-cbor-response-format.md Co-authored-by: Marcin Rataj --- IPIP/0000-gateway-json-cbor-response-format.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 67276f4b..ae646609 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -15,8 +15,8 @@ Add [DAG-JSON], [DAG-CBOR], JSON and CBOR response formats to the [HTTP Gateway] ## Motivation Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] -formats. However, it should be possible to download everything over the gateway -in order to unlock the potentials of the [IPLD Data Model][ipld-data-model]. +formats. However, it should be possible to download deserialized version of data other than UnixFS +in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. For example, DAG-PB has a [logical format][dag-pb-format] which makes it possible to represent a DAG-PB directory as a [DAG-JSON] document. This means that, if we From d7ff20445e5e48172ac867cedd38be4d6b3020e9 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 11:33:02 +0200 Subject: [PATCH 10/46] Update IPIP/0000-gateway-json-cbor-response-format.md Co-authored-by: Marcin Rataj --- IPIP/0000-gateway-json-cbor-response-format.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index ae646609..d4af05be 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -18,7 +18,14 @@ Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] formats. However, it should be possible to download deserialized version of data other than UnixFS in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. -For example, DAG-PB has a [logical format][dag-pb-format] which makes it possible + +The main functional gap in IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP Gateways. +Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), +but they are unable to load deserialized DAG-CBOR documents from local gateway, +which is severely decreasing the utility of non-UnixFS DAGs. + +Adding new responses types will also benefit UnixFS. +DAG-PB has a [logical format][dag-pb-format] which makes it possible to represent a DAG-PB directory as a [DAG-JSON] document. This means that, if we support DAG-JSON in the gateway, then we would support [JSON responses for directory listings][ipfs/go-ipfs/issues/7552], which has been From 4c8e3ca546f8c8799e0efb4ee6c880715a7e579e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 12:43:29 +0200 Subject: [PATCH 11/46] update application/json and /cbor --- http-gateways/PATH_GATEWAY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index cb1f639d..03f5e776 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -183,8 +183,8 @@ For example: - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned - [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/) - [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/) -- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, data will be serialized as JSON -- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, data will be serialized as CBOR +- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned ### `Range` (request header) From 4563155ba9a63555735defc279ff94e46312b54c Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 12:44:54 +0200 Subject: [PATCH 12/46] Update IPIP/0000-gateway-json-cbor-response-format.md Co-authored-by: Marcin Rataj --- IPIP/0000-gateway-json-cbor-response-format.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index d4af05be..b9452b9c 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -59,7 +59,8 @@ something that has been [requested before][ipfs/go-ipfs/issues/7552]. ### Compatibility -This IPIP is backwards compatible. +This IPIP adds a new response types and does not modify existing ones, +making it a backwards-compatible change. ### Security From d2f4a617b926a30abd8248eaeccf61628227a6f5 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 12:45:22 +0200 Subject: [PATCH 13/46] Update IPIP/0000-gateway-json-cbor-response-format.md Co-authored-by: Marcin Rataj --- IPIP/0000-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index b9452b9c..20529911 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -77,7 +77,7 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public [DAG-PB]: https://ipld.io/docs/codecs/known/dag-pb/ [dag-pb-format]: https://ipld.io/specs/codecs/dag-pb/spec/#logical-format [DAG-JSON]: https://ipld.io/docs/codecs/known/dag-json/ -[DAG-CBOR]: https://ipld.io/docs/codecs/known/dag-json/ +[DAG-CBOR]: https://ipld.io/docs/codecs/known/dag-cbor/ [CAR]: https://ipld.io/specs/transport/car/ [ipld-data-model]: https://ipld.io/docs/data-model/ [ipfs/in-web-browsers/issues/182]: https://github.com/ipfs/in-web-browsers/issues/182 From 948829b2056cde403ac3cc69d51d6cb5ba67a5db Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 13:36:12 +0200 Subject: [PATCH 14/46] "IPLD model" -> "IPLD Data Model" --- IPIP/0000-gateway-json-cbor-response-format.md | 2 +- http-gateways/PATH_GATEWAY.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 20529911..efb977f8 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -37,7 +37,7 @@ to traversing IPLD data. ## Detailed design -The solution is to allow the Gateway to support serializing an IPLD model +The solution is to allow the Gateway to support serializing an IPLD Data Model representation as [DAG-JSON], [DAG-CBOR], JSON and CBOR by requesting them using either the `Accept` HTTP header or the `format` URL query. diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 03f5e776..d7e3e12e 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -181,8 +181,8 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned -- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/) -- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/) +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/) +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/) - [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned - [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned From 08c315833cfc2583f24b8c13387f3861eadeccf5 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 14:24:00 +0200 Subject: [PATCH 15/46] updated response payload --- http-gateways/PATH_GATEWAY.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index d7e3e12e..168ccbd4 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -593,6 +593,15 @@ Data sent with HTTP response depends on the type of requested IPFS resource: - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) - CAR - CAR file or stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) +- JSON + - Bytes representing a JSON file, see [application/json](https://www.iana.org/assignments/media-types/application/json) +- DAG-JSON + - Bytes representing a DAG-JSON file, see [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) +- CBOR + - Bytes representing a CBOR file, see [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) +- DAG-CBOR + - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) + # Appendix: notes for implementers From c0923c1706d4ab389d753f47c5ad00cb33c8ef39 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 16:33:34 +0200 Subject: [PATCH 16/46] add some test fixtures --- IPIP/0000-gateway-json-cbor-response-format.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index efb977f8..8ca1c342 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -43,7 +43,17 @@ using either the `Accept` HTTP header or the `format` URL query. ## Test fixtures -TODO +- `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` is a DAG-PB directory. +- `bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm` is the aforementioned + DAG-PB directory's Logical DAG-JSON representation that is expected to be returned + when using `?format=dag-json`. +- Traversal Test Fixtures: the following test fixtures contain two levels of nested + documents of their encoding. Accessing `/ipfs/$CID/foo/bar` should return the JSON + equivalent of `{"hello":"this is not a link"}`. + - DAG-CBOR: `bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu` + - DAG-JSON: `baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta` +- `TODO` is a valid JSON but not a valid DAG-JSON +- `TODO` is a valid CBOR but not a valid DAG-CBOR ## Design rationale From 2cfe769a15ca84960d3bfa1fdaf17c69d14ac69e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 19 Oct 2022 16:40:54 +0200 Subject: [PATCH 17/46] add links --- .../0000-gateway-json-cbor-response-format.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 8ca1c342..406c659c 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -43,15 +43,16 @@ using either the `Accept` HTTP header or the `format` URL query. ## Test fixtures -- `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` is a DAG-PB directory. -- `bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm` is the aforementioned - DAG-PB directory's Logical DAG-JSON representation that is expected to be returned - when using `?format=dag-json`. +- [`bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke`][dag-pb] is a + DAG-PB directory. +- [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][dag-pb-json] + is the aforementioned DAG-PB directory's Logical DAG-JSON representation that + is expected to be returned when using `?format=dag-json`. - Traversal Test Fixtures: the following test fixtures contain two levels of nested documents of their encoding. Accessing `/ipfs/$CID/foo/bar` should return the JSON equivalent of `{"hello":"this is not a link"}`. - - DAG-CBOR: `bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu` - - DAG-JSON: `baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta` + - DAG-CBOR: [`bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu`][dag-cbor-traversal] + - DAG-JSON: [`baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta`][dag-json-traversal] - `TODO` is a valid JSON but not a valid DAG-JSON - `TODO` is a valid CBOR but not a valid DAG-CBOR @@ -94,4 +95,8 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public [ipfs/specs/pull/328]: https://github.com/ipfs/specs/pull/328 [ipfs/kubo/issues/8823]: https://github.com/ipfs/kubo/issues/8823 [ipfs/kubo/pull/9335]: https://github.com/ipfs/kubo/pull/9335 -[ipfs/go-ipfs/issues/7552]: https://github.com/ipfs/go-ipfs/issues/7552 \ No newline at end of file +[ipfs/go-ipfs/issues/7552]: https://github.com/ipfs/go-ipfs/issues/7552 +[dag-pb]: https://dweb.link/ipfs/bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke +[dag-pb-json]: https://dweb.link/ipfs/bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm +[dag-cbor-traversal]: https://dweb.link/ipfs/bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu +[dag-json-traversal]: https://dweb.link/ipfs/baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta From c4f9fa555cee557f43fce038dce185b5bd8f1978 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 20 Oct 2022 11:45:35 +0200 Subject: [PATCH 18/46] add security section --- .../0000-gateway-json-cbor-response-format.md | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index 406c659c..b988dfa7 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -20,8 +20,8 @@ in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyon The main functional gap in IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP Gateways. -Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), -but they are unable to load deserialized DAG-CBOR documents from local gateway, +Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), +but they are unable to load deserialized DAG-CBOR documents from local gateway, which is severely decreasing the utility of non-UnixFS DAGs. Adding new responses types will also benefit UnixFS. @@ -75,7 +75,19 @@ making it a backwards-compatible change. ### Security -TODO +Serializers and deserializers for the JSON and CBOR must follow the security +considerations of the original specifications, found in: + +- [RFC 8259 (JSON), Section 12][rfc8259-sec12] +- [RFC 8949 (CBOR), Section 10][rfc8949-sec10] + +DAG-JSON and DAG-CBOR follow the same security considerations as JSON and CBOR. +Note that DAG-JSON and DAG-CBOR are stricter variants of JSON and CBOR, respectively. +Therefore they must follow their IPLD specification and error if the payload +is not strict enough: + +- [DAG-JSON Spec][dag-json-spec] +- [DAG-CBOR Spec][dag-cbor-spec] ### Alternatives @@ -100,3 +112,7 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public [dag-pb-json]: https://dweb.link/ipfs/bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm [dag-cbor-traversal]: https://dweb.link/ipfs/bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu [dag-json-traversal]: https://dweb.link/ipfs/baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta +[rfc8259-sec12]: https://datatracker.ietf.org/doc/html/rfc8259#section-12 +[rfc8949-sec10]: https://datatracker.ietf.org/doc/html/rfc8949#section-10 +[dag-json-spec]: https://ipld.io/specs/codecs/dag-json/spec/ +[dag-cbor-spec]: https://ipld.io/specs/codecs/dag-cbor/spec/ From a0c86bb78aa995cb799b691c34225713122bcf6a Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 20 Oct 2022 12:03:55 +0200 Subject: [PATCH 19/46] write alternatives section --- IPIP/0000-gateway-json-cbor-response-format.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index b988dfa7..dbbed665 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -91,7 +91,17 @@ is not strict enough: ### Alternatives -TODO +Introducing DAG-JSON, DAG-CBOR, JSON and CBOR in the HTTP Gateway allows for +a broader usage of the IPLD Data Model. If we do not introduce more IPLD +response formats in the gateway, the usage of IPFS is constricted to files +and directories represented by UnixFS (DAG-PB) codec. Therefore, it would keep +the IPLD potential locked due to an artificial barrier created by the gateways. + +In addition, we could introduce only DAG-JSON and DAG-CBOR. However, not +supporting the generic variants, JSON and CBOR, would lead to poor UX. The +ability to retrieve DAG-JSON as `application/json` is an important step +for the interoperability of the HTTP Gateway with web browsers and other tools +that expect specific Content Types. ### Copyright From 628775a26ee0b8220046a443b463c50d1ed825f0 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 10 Nov 2022 12:04:31 +0100 Subject: [PATCH 20/46] fix: duplicate links --- IPIP/0000-gateway-json-cbor-response-format.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0000-gateway-json-cbor-response-format.md index dbbed665..64adb018 100644 --- a/IPIP/0000-gateway-json-cbor-response-format.md +++ b/IPIP/0000-gateway-json-cbor-response-format.md @@ -18,7 +18,6 @@ Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] formats. However, it should be possible to download deserialized version of data other than UnixFS in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. - The main functional gap in IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP Gateways. Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), but they are unable to load deserialized DAG-CBOR documents from local gateway, @@ -43,16 +42,16 @@ using either the `Accept` HTTP header or the `format` URL query. ## Test fixtures -- [`bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke`][dag-pb] is a +- [`bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke`][f-dag-pb] is a DAG-PB directory. -- [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][dag-pb-json] +- [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][f-dag-pb-json] is the aforementioned DAG-PB directory's Logical DAG-JSON representation that is expected to be returned when using `?format=dag-json`. - Traversal Test Fixtures: the following test fixtures contain two levels of nested documents of their encoding. Accessing `/ipfs/$CID/foo/bar` should return the JSON equivalent of `{"hello":"this is not a link"}`. - - DAG-CBOR: [`bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu`][dag-cbor-traversal] - - DAG-JSON: [`baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta`][dag-json-traversal] + - DAG-CBOR: [`bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu`][f-dag-cbor-traversal] + - DAG-JSON: [`baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta`][f-dag-json-traversal] - `TODO` is a valid JSON but not a valid DAG-JSON - `TODO` is a valid CBOR but not a valid DAG-CBOR @@ -118,10 +117,10 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public [ipfs/kubo/issues/8823]: https://github.com/ipfs/kubo/issues/8823 [ipfs/kubo/pull/9335]: https://github.com/ipfs/kubo/pull/9335 [ipfs/go-ipfs/issues/7552]: https://github.com/ipfs/go-ipfs/issues/7552 -[dag-pb]: https://dweb.link/ipfs/bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke -[dag-pb-json]: https://dweb.link/ipfs/bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm -[dag-cbor-traversal]: https://dweb.link/ipfs/bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu -[dag-json-traversal]: https://dweb.link/ipfs/baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta +[f-dag-pb]: https://dweb.link/ipfs/bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke +[f-dag-pb-json]: https://dweb.link/ipfs/bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm +[f-dag-cbor-traversal]: https://dweb.link/ipfs/bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu +[f-dag-json-traversal]: https://dweb.link/ipfs/baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta [rfc8259-sec12]: https://datatracker.ietf.org/doc/html/rfc8259#section-12 [rfc8949-sec10]: https://datatracker.ietf.org/doc/html/rfc8949#section-10 [dag-json-spec]: https://ipld.io/specs/codecs/dag-json/spec/ From 9581ac1e211e2a01ad3ea5bfe97cb9afb028a73e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Thu, 10 Nov 2022 12:04:51 +0100 Subject: [PATCH 21/46] refactor: rename ipip --- ...sponse-format.md => 0328-gateway-json-cbor-response-format.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename IPIP/{0000-gateway-json-cbor-response-format.md => 0328-gateway-json-cbor-response-format.md} (100%) diff --git a/IPIP/0000-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md similarity index 100% rename from IPIP/0000-gateway-json-cbor-response-format.md rename to IPIP/0328-gateway-json-cbor-response-format.md From 33b4bea5f1feafb4661765f8d5001afe35109030 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 11 Nov 2022 13:23:12 +0100 Subject: [PATCH 22/46] ipip: clarify no paths --- .../0328-gateway-json-cbor-response-format.md | 25 +++++++++++++------ 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 64adb018..99094b13 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -47,11 +47,6 @@ using either the `Accept` HTTP header or the `format` URL query. - [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][f-dag-pb-json] is the aforementioned DAG-PB directory's Logical DAG-JSON representation that is expected to be returned when using `?format=dag-json`. -- Traversal Test Fixtures: the following test fixtures contain two levels of nested - documents of their encoding. Accessing `/ipfs/$CID/foo/bar` should return the JSON - equivalent of `{"hello":"this is not a link"}`. - - DAG-CBOR: [`bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu`][f-dag-cbor-traversal] - - DAG-JSON: [`baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta`][f-dag-json-traversal] - `TODO` is a valid JSON but not a valid DAG-JSON - `TODO` is a valid CBOR but not a valid DAG-CBOR @@ -102,6 +97,24 @@ ability to retrieve DAG-JSON as `application/json` is an important step for the interoperability of the HTTP Gateway with web browsers and other tools that expect specific Content Types. +Finally, we considered supporting pathing for both DAG and non-DAG variants of +the JSON and CBOR codecs. However, supporting pathing raises questions whose +answers are not clearly defined or agreed upon yet: + +1. Pathing is an important element of DAG-JSON and DAG-CBOR as they can materialize +CIDs. That would allow users to traverse an IPLD document directly by using the paths +in the gateway. However, if we support paths that materialize CIDs, we would also +need to support pathing to extract nested fields in the document. That raises questions +regarding caching and Etags. +2. Pathing could be done on CBOR documents, as the tag 42 indicates that there is a CID. +However, that would raise the same question as in the previous point: if we materialize +CIDs, do we support extracting all nested fields? If so, why? Then, how would caching +work? + +Therefore, we deferred this decision for a future IPIP. Giving users the possibility +to retrieve JSON, CBOR, DAG-JSON AND DAG-CBOR documents through the gateway is, in +itself, a progress and will open the doors for new tools and explorations. + ### Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). @@ -119,8 +132,6 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public [ipfs/go-ipfs/issues/7552]: https://github.com/ipfs/go-ipfs/issues/7552 [f-dag-pb]: https://dweb.link/ipfs/bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke [f-dag-pb-json]: https://dweb.link/ipfs/bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm -[f-dag-cbor-traversal]: https://dweb.link/ipfs/bafyreiehxu373cu3v5gyxyxfsfjryscs7sq6fh3unqcqgqhdfn3n43vrgu -[f-dag-json-traversal]: https://dweb.link/ipfs/baguqeeraoaeabj5hdfcmpkzfeiwtfwb3qbvfwzbiknqn7itcwsb2fdtu7eta [rfc8259-sec12]: https://datatracker.ietf.org/doc/html/rfc8259#section-12 [rfc8949-sec10]: https://datatracker.ietf.org/doc/html/rfc8949#section-10 [dag-json-spec]: https://ipld.io/specs/codecs/dag-json/spec/ From 378c5b9ab360de8393bea5206c6e51930c8d32e3 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 15 Nov 2022 15:14:18 +0100 Subject: [PATCH 23/46] info validation dag-json and dag-cbor Co-authored-by: Marcin Rataj --- http-gateways/PATH_GATEWAY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index eb866acf..8ba8d482 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -182,8 +182,8 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a root item which name is the same as the requested CID. Produces 400 Bad Request for content that is not UnixFS. -- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/) -- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/) +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID has `dag-json` codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500. +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID has `dag-cbor` codec, data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBON produces HTTP Error 500. - [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned - [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned From 61f43ce214560ef64142afb2d95229487aaa0f27 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 15 Nov 2022 15:48:31 +0100 Subject: [PATCH 24/46] ipip: update path gateway sec --- http-gateways/PATH_GATEWAY.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index eb866acf..99fd2c84 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -588,7 +588,7 @@ A good practice is to always return it with HTTP error [status codes](#response- ## Response Payload -Data sent with HTTP response depends on the type of requested IPFS resource: +Data sent with HTTP response depends on the type of the requested IPFS resource, and the requested response type. The following response formats are selected according to the codec of the resolved CID: - UnixFS (implicit default) - File @@ -596,21 +596,26 @@ Data sent with HTTP response depends on the type of requested IPFS resource: - Directory - Generated HTML with directory index (see [additional notes here](#generated-html-with-directory-index)) - When `index.html` is present, gateway can skip generating directory index and return it instead -- Raw block - - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) -- CAR - - Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) -- TAR - - Deserialized UnixFS files and directories as a TAR file or a stream, see [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) - JSON - - Bytes representing a JSON file, see [application/json](https://www.iana.org/assignments/media-types/application/json) -- DAG-JSON - - Bytes representing a DAG-JSON file, see [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) + - Bytes representing a JSON file, see [application/json](https://www.iana.org/assignments/media-types/application/json). - CBOR - Bytes representing a CBOR file, see [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) +- DAG-JSON + - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-JSON as-is, or converted to DAG-CBOR. + - Bytes representing a DAG-JSON file, see [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) - DAG-CBOR + - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON. - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) +The following response types can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: + +- Raw Block + - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw). +- CAR + - Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car). +- TAR + - Deserialized UnixFS files and directories as a TAR file or a stream, see [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)). + # Appendix: notes for implementers ## Content resolution From 566dabd544b1ae5a7ea2607e0b45173c0b093c4f Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 15 Nov 2022 17:14:22 +0100 Subject: [PATCH 25/46] Update IPIP/0328-gateway-json-cbor-response-format.md Co-authored-by: Steve Loeppky --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 99094b13..3f794bd4 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -54,7 +54,7 @@ using either the `Accept` HTTP header or the `format` URL query. The current gateway already supports different response formats via the `Accept` HTTP header and the `format` URL query. This IPIP proposes adding -one more supported format to that list. +more supported formats to that list. ### User benefit From 239b17b1035ac50f148ce9b6f71999ae2480d531 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:41:12 +0100 Subject: [PATCH 26/46] Update IPIP/0328-gateway-json-cbor-response-format.md Co-authored-by: Gus Eggert --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 3f794bd4..197fda5e 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -15,7 +15,7 @@ Add [DAG-JSON], [DAG-CBOR], JSON and CBOR response formats to the [HTTP Gateway] ## Motivation Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] -formats. However, it should be possible to download deserialized version of data other than UnixFS +formats. However, it should be possible to download deserialized versions of data other than UnixFS in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. The main functional gap in IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP Gateways. From 79674a081ac336df00fabc548e937fb74d3f9026 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:41:19 +0100 Subject: [PATCH 27/46] Update IPIP/0328-gateway-json-cbor-response-format.md Co-authored-by: Gus Eggert --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 197fda5e..bdb22ba7 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -18,7 +18,7 @@ Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] formats. However, it should be possible to download deserialized versions of data other than UnixFS in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. -The main functional gap in IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP Gateways. +The main functional gap in the IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP gateways. Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), but they are unable to load deserialized DAG-CBOR documents from local gateway, which is severely decreasing the utility of non-UnixFS DAGs. From ea486b6d7e5c8d2cdfac64a92a89b82cc139e4f9 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:41:24 +0100 Subject: [PATCH 28/46] Update IPIP/0328-gateway-json-cbor-response-format.md Co-authored-by: Gus Eggert --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index bdb22ba7..4bb08a71 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -20,7 +20,7 @@ in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyon The main functional gap in the IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP gateways. Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), -but they are unable to load deserialized DAG-CBOR documents from local gateway, +but they are unable to load deserialized DAG-CBOR documents from a local gateway, which is severely decreasing the utility of non-UnixFS DAGs. Adding new responses types will also benefit UnixFS. From 542315a5667095969c054595fe04bc6c1fa528d6 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:41:30 +0100 Subject: [PATCH 29/46] Update IPIP/0328-gateway-json-cbor-response-format.md Co-authored-by: Gus Eggert --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 4bb08a71..4bf29e04 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -64,7 +64,7 @@ something that has been [requested before][ipfs/go-ipfs/issues/7552]. ### Compatibility -This IPIP adds a new response types and does not modify existing ones, +This IPIP adds new response types and does not modify existing ones, making it a backwards-compatible change. ### Security From 3a84f2880a5778710f516ddb289cf9bd3ef23aef Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:54:30 +0100 Subject: [PATCH 30/46] document current traversal behaviour --- http-gateways/PATH_GATEWAY.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index e010464e..eebd28bd 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -641,12 +641,22 @@ and [DNSLINK_GATEWAY.md](./DNSLINK_GATEWAY.md)). ### Traversing remaining path UnixFS pathing over files and directories is the implicit default used for -resolving content paths that start with `/ipfs/` and `/ipns/`. It allows for -traversal based on link names, which provides a better user experience than -low level logical pathing from IPLD: +resolving content paths that start with `/ipfs/` and `/ipns/`. It is an abstraction +over the low level [logical pathing][dag-pb-format] from IPLD, providing a +better user experience: - Example of UnixFS pathing: `/ipfs/cid/dir-name/file-name.txt` +### Traversing through DAG-JSON and DAG-CBOR + +Traversing through [DAG-JSON][dag-json] and [DAG-CBOR][dag-json] is possible +through fields that encode a link: + +- DAG-JSON: link are represented as a base encoded CID under the `/` reserved +namespace, see [specification](https://ipld.io/specs/codecs/dag-json/spec/#links). +- DAG-CBOR: links are tagged with CBOR tag 42, indicating that they encode a CID, +see [specification](https://ipld.io/specs/codecs/dag-cbor/spec/#links). + ### Handling traversal errors Gateway MUST respond with HTTP error when it is not possible to traverse the requested content path: @@ -706,9 +716,8 @@ It should be always fast, even when a directory has 10k of items. The usual optimizations involve: - Skipping size and type resolution for child UnixFS items, and using `Tsize` - from [logical format](https://ipld.io/specs/codecs/dag-pb/spec/#logical-format) - instead, allows gateway to respond much faster, as it no longer need to fetch - root nodes of child items. + from [logical format][dag-pb-format] instead, allows gateway to respond much + faster, as it no longer need to fetch root nodes of child items. - Additional information about child nodes can be fetched lazily with JS, but only for items in the browser's viewport. @@ -718,3 +727,7 @@ The usual optimizations involve: limiting the cost of a single page load. - The downside of this approach is that it will always be slower than skipping child block resolution. + +[dag-pb-format]: https://ipld.io/specs/codecs/dag-pb/spec/#logical-format +[dag-json]: https://ipld.io/specs/codecs/dag-json/spec/ +[dag-cbor]: https://ipld.io/specs/codecs/dag-cbor/spec/ From 5720d6a68ee9bbdc2fc0e69ea6fde61d98a7a721 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 09:55:35 +0100 Subject: [PATCH 31/46] add toc link --- http-gateways/PATH_GATEWAY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index eebd28bd..e26ed75d 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -79,6 +79,7 @@ where client prefers to perform all validation locally. - [Content resolution](#content-resolution) - [Finding the content root](#finding-the-content-root) - [Traversing remaining path](#traversing-remaining-path) + - [Traversing through DAG-JSON and DAG-CBOR](#traversing-through-dag-json-and-dag-cbor) - [Handling traversal errors](#handling-traversal-errors) - [Best practices for HTTP caching](#best-practices-for-http-caching) - [Denylists](#denylists) From 79636c953e032ce1d93a7dd7bc9d42b93c6ff754 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 10:10:13 +0100 Subject: [PATCH 32/46] improve ipip and clarify things --- .../0328-gateway-json-cbor-response-format.md | 100 ++++++++++++------ http-gateways/PATH_GATEWAY.md | 2 +- 2 files changed, 68 insertions(+), 34 deletions(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 4bf29e04..3f4b23f7 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -1,4 +1,4 @@ -# IPIP 0000: Gateway JSON and TAR Response Formats +# IPIP 0328: Gateway JSON and TAR Response Formats - Start Date: 2022-10-07 - Related Issues: @@ -10,23 +10,28 @@ ## Summary -Add [DAG-JSON], [DAG-CBOR], JSON and CBOR response formats to the [HTTP Gateway](../http-gateways/). +Add support for the [DAG-JSON], [DAG-CBOR], JSON and CBOR response formats in +the [HTTP Gateway](../http-gateways/). ## Motivation -Currently, the gateway supports requesting data in the [DAG-PB], RAW, and [CAR] -formats. However, it should be possible to download deserialized versions of data other than UnixFS -in order to unlock the potential of the [IPLD Data Model][ipld-data-model] beyond files and directories. - -The main functional gap in the IPFS ecosystem is the lack of support for non-UnixFS DAGs on HTTP gateways. -Users are able to create custom DAGs based on traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs](https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859), -but they are unable to load deserialized DAG-CBOR documents from a local gateway, -which is severely decreasing the utility of non-UnixFS DAGs. - -Adding new responses types will also benefit UnixFS. -DAG-PB has a [logical format][dag-pb-format] which makes it possible -to represent a DAG-PB directory as a [DAG-JSON] document. This means that, if we -support DAG-JSON in the gateway, then we would support +Currently, the gateway supports requesting data in the [DAG-PB], RAW, [CAR] and +TAR formats. In addition, it allows for traversing of IPLD links encoded in +DAG-JSON and DAG-CBOR, as long as they are intermediate links, and not the final +document. However, it should be possible to download deserialized versions +of data other than UnixFS in order to unlock the potential of the +[IPLD Data Model][ipld-data-model] beyond files and directories. + +The main functional gap in the IPFS ecosystem is the lack of support for +non-UnixFS DAGs on HTTP gateways. Users are able to create custom DAGs based on +traversable DAG-CBOR thanks to [CBOR tag 42 being reserved for CIDs][cbor-42] +and DAG-JSON documents, but they are unable to load deserialized documents from +a local gateway, which is severely decreasing the utility of non-UnixFS DAGs. + +Adding new responses types will also benefit UnixFS. DAG-PB has a +[logical format][dag-pb-format] which makes it possible to represent a DAG-PB +directory as a [DAG-JSON] document. This means that, if we support DAG-JSON in +the gateway, then we would support [JSON responses for directory listings][ipfs/go-ipfs/issues/7552], which has been requested by our users in the past. @@ -38,7 +43,9 @@ to traversing IPLD data. The solution is to allow the Gateway to support serializing an IPLD Data Model representation as [DAG-JSON], [DAG-CBOR], JSON and CBOR by requesting them -using either the `Accept` HTTP header or the `format` URL query. +using either the `Accept` HTTP header or the `format` URL query. In addition, +if the resolved CID is of one of the aforementioned types, the gateway should +be able to resolve them instead of failing with `node type unknown`. ## Test fixtures @@ -56,6 +63,11 @@ The current gateway already supports different response formats via the `Accept` HTTP header and the `format` URL query. This IPIP proposes adding more supported formats to that list. +In addition, the current gateway already supports traversing through DAG-CBOR +and DAG-JSON links if they are intermediary documents. With this IPIP, we aim +to be able to download the DAG-CBOR, DAG-JSON, JSON and CBOR documents +themselves. + ### User benefit The user benefits from this change as they will now be able to retrieve @@ -97,28 +109,50 @@ ability to retrieve DAG-JSON as `application/json` is an important step for the interoperability of the HTTP Gateway with web browsers and other tools that expect specific Content Types. -Finally, we considered supporting pathing for both DAG and non-DAG variants of -the JSON and CBOR codecs. However, supporting pathing raises questions whose -answers are not clearly defined or agreed upon yet: - -1. Pathing is an important element of DAG-JSON and DAG-CBOR as they can materialize -CIDs. That would allow users to traverse an IPLD document directly by using the paths -in the gateway. However, if we support paths that materialize CIDs, we would also -need to support pathing to extract nested fields in the document. That raises questions -regarding caching and Etags. -2. Pathing could be done on CBOR documents, as the tag 42 indicates that there is a CID. -However, that would raise the same question as in the previous point: if we materialize -CIDs, do we support extracting all nested fields? If so, why? Then, how would caching -work? - -Therefore, we deferred this decision for a future IPIP. Giving users the possibility -to retrieve JSON, CBOR, DAG-JSON AND DAG-CBOR documents through the gateway is, in -itself, a progress and will open the doors for new tools and explorations. +Finally, we considered supporting pathing within both DAG and non-DAG variants +of the JSON and CBOR codecs. Pathing within these documents could lead to replies +with extracts from the document. For example, if we have the document: + +```json +{ + "link" { + "to": { + "some": { + "cid2": + } + } + } +} +``` + +With CID `bafy`, and we navigate to `/ipfs/bafy/link/to`, we would be able to +retrieve an extract from the document. + +```json +{ + "some": { + "cid2": + } +} +``` + +However, supporting this raises questions whose answers are not clearly defined +or agreed upon yet. The main question this raises is caching and Etags. Therefore, +this IPIP only supports pathing within DAG-JSON and DAG-CBOR if they resolve +a CID, as it is the current behavior. In addition, it adds support to retrieve +the documents in those formats. Pathing within JSON and CBOR documents is not +supported. + +The decision to support document path extractions id deferred to a future IPIP. +Giving users the possibility to retrieve JSON, CBOR, DAG-JSON AND DAG-CBOR +documents through the gateway is, in itself, a progress and will open the doors +for new tools and explorations. ### Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). +[cbor-42]: https://github.com/core-wg/yang-cbor/issues/13#issuecomment-524378859 [DAG-PB]: https://ipld.io/docs/codecs/known/dag-pb/ [dag-pb-format]: https://ipld.io/specs/codecs/dag-pb/spec/#logical-format [DAG-JSON]: https://ipld.io/docs/codecs/known/dag-json/ diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index e26ed75d..f4dee074 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -650,7 +650,7 @@ better user experience: ### Traversing through DAG-JSON and DAG-CBOR -Traversing through [DAG-JSON][dag-json] and [DAG-CBOR][dag-json] is possible +Traversing through [DAG-JSON][dag-json] and [DAG-CBOR][dag-cbor] is possible through fields that encode a link: - DAG-JSON: link are represented as a base encoded CID under the `/` reserved From ba0920642e4a4ed0214582d8510bfb2ccccde87e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 10:46:03 +0100 Subject: [PATCH 33/46] update path gateway --- http-gateways/PATH_GATEWAY.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index f4dee074..24f5168f 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -185,8 +185,8 @@ For example: - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a root item which name is the same as the requested CID. Produces 400 Bad Request for content that is not UnixFS. - [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID has `dag-json` codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500. - [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID has `dag-cbor` codec, data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBON produces HTTP Error 500. -- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned -- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned +- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned. +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned. ### `Range` (request header) @@ -589,9 +589,10 @@ A good practice is to always return it with HTTP error [status codes](#response- ## Response Payload -Data sent with HTTP response depends on the type of the requested IPFS resource, and the requested response type. The following response formats are selected according to the codec of the resolved CID: +Data sent with HTTP response depends on the type of the requested IPFS resource, and the requested response type. +The following response formats are selected according to the codec of the resolved CID: -- UnixFS (implicit default) +- UnixFS - File - Bytes representing file contents - Directory From 6d7f8f02716677cf5b5a5aca096bedf1075174bd Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 10:47:45 +0100 Subject: [PATCH 34/46] add 400 --- http-gateways/PATH_GATEWAY.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 24f5168f..8efe4790 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -609,6 +609,9 @@ The following response formats are selected according to the codec of the resolv - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON. - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) +Pathing through JSON, CBOR, DAG-JSON and DAG-CBOR returns an HTTP error 400 Bad +Request unless it resolves to a CID. + The following response types can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: - Raw Block From d8310e25dec5bd622d92e7576a3cfab05acf0531 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 16 Nov 2022 10:49:11 +0100 Subject: [PATCH 35/46] revert prev commit as it is already mentioned in traversal errors --- http-gateways/PATH_GATEWAY.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 8efe4790..24f5168f 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -609,9 +609,6 @@ The following response formats are selected according to the codec of the resolv - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON. - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) -Pathing through JSON, CBOR, DAG-JSON and DAG-CBOR returns an HTTP error 400 Bad -Request unless it resolves to a CID. - The following response types can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: - Raw Block From a50536eca2a606f7f6b72095fac5af1e2d2df92f Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 26 Nov 2022 00:40:55 +0100 Subject: [PATCH 36/46] ipip-328: code review based on working code https://github.com/ipfs/kubo/pull/9335 --- .../0328-gateway-json-cbor-response-format.md | 7 +++--- http-gateways/PATH_GATEWAY.md | 24 ++++++++++--------- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index 3f4b23f7..d8f9269e 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -52,7 +52,7 @@ be able to resolve them instead of failing with `node type unknown`. - [`bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke`][f-dag-pb] is a DAG-PB directory. - [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][f-dag-pb-json] - is the aforementioned DAG-PB directory's Logical DAG-JSON representation that + is the aforementioned DAG-PB directory's [Logical DAG-JSON representation][dag-pb-format] that is expected to be returned when using `?format=dag-json`. - `TODO` is a valid JSON but not a valid DAG-JSON - `TODO` is a valid CBOR but not a valid DAG-CBOR @@ -107,10 +107,11 @@ In addition, we could introduce only DAG-JSON and DAG-CBOR. However, not supporting the generic variants, JSON and CBOR, would lead to poor UX. The ability to retrieve DAG-JSON as `application/json` is an important step for the interoperability of the HTTP Gateway with web browsers and other tools -that expect specific Content Types. +that expect specific Content Types. Namely, `Content-Type: application/json` with +`Content-Disposition: inline` allows for JSON preview to be rendered in a web browser. Finally, we considered supporting pathing within both DAG and non-DAG variants -of the JSON and CBOR codecs. Pathing within these documents could lead to replies +of the JSON and CBOR codecs. Pathing within these documents could lead to responses with extracts from the document. For example, if we have the document: ```json diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 24f5168f..6f95059e 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -183,10 +183,10 @@ For example: - [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable raw [block](https://docs.ipfs.io/concepts/glossary/#block) to be returned - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) – disables [IPLD codec deserialization](https://ipld.io/docs/codecs/), requests a verifiable [CAR](https://docs.ipfs.io/concepts/glossary/#car) stream to be returned - [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)) – returns UnixFS tree (files and directories) as a [TAR](https://en.wikipedia.org/wiki/Tar_(computing)) stream. Returned tree starts at a root item which name is the same as the requested CID. Produces 400 Bad Request for content that is not UnixFS. -- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID has `dag-json` codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500. -- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID has `dag-cbor` codec, data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBON produces HTTP Error 500. -- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec is JSON. Then, the raw JSON block can be returned. -- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec is CBOR. Then, the raw CBOR block can be returned. +- [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-JSON format](https://ipld.io/docs/codecs/known/dag-json/). If the requested CID already has `dag-json` (0x0129) codec, data is validated as DAG-JSON before being returned as-is. Invalid DAG-JSON produces HTTP Error 500. +- [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) – requests [IPLD Data Model](https://ipld.io/docs/data-model/) representation serialized into [DAG-CBOR format](https://ipld.io/docs/codecs/known/dag-cbor/). If the requested CID already has `dag-cbor` (0x71) codec, data is validated as DAG-CBOR before being returned as-is. Invalid DAG-CBON produces HTTP Error 500. +- [application/json](https://www.iana.org/assignments/media-types/application/json) – same as `application/vnd.ipld.dag-json`, unless the CID's codec already is `json` (0x0200). Then, the raw JSON block can be returned. +- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – same as `application/vnd.ipld.dag-cbor`, unless the CID's codec already is `cbor` (0x51). Then, the raw CBOR block can be returned. ### `Range` (request header) @@ -592,31 +592,31 @@ A good practice is to always return it with HTTP error [status codes](#response- Data sent with HTTP response depends on the type of the requested IPFS resource, and the requested response type. The following response formats are selected according to the codec of the resolved CID: -- UnixFS +- UnixFS, either `dag-pb` (0x70) or `raw` (0x55) - File - Bytes representing file contents - Directory - Generated HTML with directory index (see [additional notes here](#generated-html-with-directory-index)) - When `index.html` is present, gateway can skip generating directory index and return it instead -- JSON +- JSON (0x0200) - Bytes representing a JSON file, see [application/json](https://www.iana.org/assignments/media-types/application/json). -- CBOR +- CBOR (0x51) - Bytes representing a CBOR file, see [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) -- DAG-JSON +- DAG-JSON (0x0129) - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-JSON as-is, or converted to DAG-CBOR. - Bytes representing a DAG-JSON file, see [application/vnd.ipld.dag-json](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) -- DAG-CBOR +- DAG-CBOR (0x71) - If the `Accept` header includes `text/html`: generated HTML with options to download DAG-CBOR as-is, or converted to DAG-JSON. - Bytes representing a DAG-CBOR file, see [application/vnd.ipld.dag-cbor](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) -The following response types can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: +The following response types are opt-in, can only be requested with [`format`](#format-request-query-parameter) query parameter or [`Accept`](#accept-request-header) header: - Raw Block - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw). - CAR - Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car). - TAR - - Deserialized UnixFS files and directories as a TAR file or a stream, see [application/x-tar](https://en.wikipedia.org/wiki/Tar_(computing)). + - Deserialized UnixFS files and directories as a TAR file or a stream, see [IPIP-288](https://github.com/ipfs/specs/pull/288) # Appendix: notes for implementers @@ -659,6 +659,8 @@ namespace, see [specification](https://ipld.io/specs/codecs/dag-json/spec/#links - DAG-CBOR: links are tagged with CBOR tag 42, indicating that they encode a CID, see [specification](https://ipld.io/specs/codecs/dag-cbor/spec/#links). +Note: pathing into [IPLD Kind](https://ipld.io/docs/data-model/kinds/) other than Link (CID) is not supported at the moment. Implementations should return HTTP 501 Not Implemented when fully resolved content path has any remainder left. This feature may be specified in a future IPIP that introduces data onboarding and [IPLD Patch](https://ipld.io/specs/patch/) semantics. + ### Handling traversal errors Gateway MUST respond with HTTP error when it is not possible to traverse the requested content path: From 52a2caf4496a8fe1d721afdf4e5021e38978fa0e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 28 Nov 2022 11:32:39 +0100 Subject: [PATCH 37/46] clarify alternatives section --- .../0328-gateway-json-cbor-response-format.md | 33 ++++++++++--------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index d8f9269e..c0c094dd 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -115,14 +115,14 @@ of the JSON and CBOR codecs. Pathing within these documents could lead to respon with extracts from the document. For example, if we have the document: ```json -{ +{ "link" { "to": { "some": { "cid2": - } - } - } + } + } + } } ``` @@ -133,21 +133,22 @@ retrieve an extract from the document. { "some": { "cid2": - } -} + } +} ``` However, supporting this raises questions whose answers are not clearly defined -or agreed upon yet. The main question this raises is caching and Etags. Therefore, -this IPIP only supports pathing within DAG-JSON and DAG-CBOR if they resolve -a CID, as it is the current behavior. In addition, it adds support to retrieve -the documents in those formats. Pathing within JSON and CBOR documents is not -supported. - -The decision to support document path extractions id deferred to a future IPIP. -Giving users the possibility to retrieve JSON, CBOR, DAG-JSON AND DAG-CBOR -documents through the gateway is, in itself, a progress and will open the doors -for new tools and explorations. +or agreed upon yet. Right now, pathing is only supported over CID-based Links, +such as Tag 42 in CBOR. In addition, some HTTP headers regarding caching are based +on the CID, and adding pathing for other IPLD Kinds would require additional +refactor. Adding those changes prematurely, before adding support to +[IPLD Patch](https://ipld.io/specs/patch/), may lead to confusion for both +the developers and users. + +We want to support full IPLD pathing support after introducing IPLD Patch, +which will be in a future, separate IPIP. Giving users the possibility to retrieve +JSON, CBOR, DAG-JSON AND DAG-CBOR documents through the gateway is, in itself, +a progress and will open the doors for new tools and explorations. ### Copyright From 51f47628f2aae694a6285d82d5cbe939e390e083 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 28 Nov 2022 11:41:29 +0100 Subject: [PATCH 38/46] remove test fixtures todos --- IPIP/0328-gateway-json-cbor-response-format.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index c0c094dd..c05b3553 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -54,8 +54,6 @@ be able to resolve them instead of failing with `node type unknown`. - [`bafkreidmwhhm6myajxlpu7kofe3aqwf4ezxxn46cp5fko7mb6x74g4k5nm`][f-dag-pb-json] is the aforementioned DAG-PB directory's [Logical DAG-JSON representation][dag-pb-format] that is expected to be returned when using `?format=dag-json`. -- `TODO` is a valid JSON but not a valid DAG-JSON -- `TODO` is a valid CBOR but not a valid DAG-CBOR ## Design rationale From 8007584e1eb8d43db16fb892e65506a1e8346317 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 28 Nov 2022 14:56:07 +0100 Subject: [PATCH 39/46] add UX and DX note --- IPIP/0328-gateway-json-cbor-response-format.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index c05b3553..acd9b462 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -72,6 +72,10 @@ The user benefits from this change as they will now be able to retrieve content encoded in the traversable DAG-JSON and DAG-CBOR formats. This is something that has been [requested before][ipfs/go-ipfs/issues/7552]. +In addition, both UX and DX are significantly improved, since every block can +now be inspected in a regular web browser via `?format=json`. This can remove the +need for additional tools, decoders, and/or libraries. + ### Compatibility This IPIP adds new response types and does not modify existing ones, From 556e500c22d077c49d6a31a17455f96055e123e1 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 28 Nov 2022 15:34:06 +0100 Subject: [PATCH 40/46] add some unixfs traversing info --- http-gateways/PATH_GATEWAY.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 6f95059e..670e22d1 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -79,6 +79,7 @@ where client prefers to perform all validation locally. - [Content resolution](#content-resolution) - [Finding the content root](#finding-the-content-root) - [Traversing remaining path](#traversing-remaining-path) + - [Traversing through UnixFS](#traversing-through-unixfs) - [Traversing through DAG-JSON and DAG-CBOR](#traversing-through-dag-json-and-dag-cbor) - [Handling traversal errors](#handling-traversal-errors) - [Best practices for HTTP caching](#best-practices-for-http-caching) @@ -642,13 +643,20 @@ and [DNSLINK_GATEWAY.md](./DNSLINK_GATEWAY.md)). ### Traversing remaining path -UnixFS pathing over files and directories is the implicit default used for -resolving content paths that start with `/ipfs/` and `/ipns/`. It is an abstraction -over the low level [logical pathing][dag-pb-format] from IPLD, providing a -better user experience: +After the content route is found, the remaining of the path should be traversed +and resolved. Depending on the data type, that may occur through UnixFS pathing, +or DAG-JSON, and DAG-CBOR pathing. + +### Traversing through UnixFS + +UnixFS is an abstraction over the low level [logical DAG-PB pathing][dag-pb-format] +from IPLD, providing a better user experience: - Example of UnixFS pathing: `/ipfs/cid/dir-name/file-name.txt` +For more details regarding DAG-PB pathing, please read the "Path Resolution" section +of [this document](https://ipld.io/design/tricky-choices/dag-pb-forms-impl-and-use/#path-resolution). + ### Traversing through DAG-JSON and DAG-CBOR Traversing through [DAG-JSON][dag-json] and [DAG-CBOR][dag-cbor] is possible From b2ac007e5e702740fa849cc2164b20ff2bf92492 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 28 Nov 2022 15:42:39 +0100 Subject: [PATCH 41/46] ipip-328: fix title --- IPIP/0328-gateway-json-cbor-response-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/IPIP/0328-gateway-json-cbor-response-format.md b/IPIP/0328-gateway-json-cbor-response-format.md index acd9b462..194e4339 100644 --- a/IPIP/0328-gateway-json-cbor-response-format.md +++ b/IPIP/0328-gateway-json-cbor-response-format.md @@ -1,4 +1,4 @@ -# IPIP 0328: Gateway JSON and TAR Response Formats +# IPIP-328: JSON and CBOR Response Formats on HTTP Gateways - Start Date: 2022-10-07 - Related Issues: From cfec9bd7bcf0c423023f030b928b55ccd7c14f7f Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 2 Dec 2022 14:40:51 +0100 Subject: [PATCH 42/46] fix: tar format --- http-gateways/PATH_GATEWAY.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/http-gateways/PATH_GATEWAY.md b/http-gateways/PATH_GATEWAY.md index 670e22d1..c728118b 100644 --- a/http-gateways/PATH_GATEWAY.md +++ b/http-gateways/PATH_GATEWAY.md @@ -251,13 +251,12 @@ Optional, `format=` can be used to request specific response format. This is a URL-friendly alternative to sending an [`Accept`](#accept-request-header) header. These are the equivalents: - `format=raw` → `Accept: application/vnd.ipld.raw` - `format=car` → `Accept: application/vnd.ipld.car` +- `format=tar` → `Accept: application/x-tar` - `format=dag-json` → `Accept: application/vnd.ipld.dag-json` - `format=dag-cbor` → `Accept: application/vnd.ipld.dag-cbor` - `format=json` → `Accept: application/json` - `format=cbor` → `Accept: application/cbor` -In case of `Accept: application/x-tar`, the `?format=` equivalent is `tar`. -