From 09d808b4eeac34af5c2a378f4f0524367c706083 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 17 Apr 2024 10:10:41 +0200 Subject: [PATCH 1/5] http gateway: document Content-Location --- src/http-gateways/path-gateway.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 5088728c..9c3296da 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -499,6 +499,17 @@ Returned only when request was a [`Range`](#range-request-header) request. See Section 14.4 of :cite[rfc9110]. +### `Content-Location` (response header) + +Returned when a non-default content format has been negotiated with the +[`Accept` header](#accept-request-header). The value of this field SHOULD include +the URL of the resource with the `format` query parameter included, so that +generic HTTP caches can store deserialized, CAR, and block responses separately. + +For example, a request to `/ipfs/{cid}` with `Accept: application/vnd.ipld.raw` +SHOULD return a `Content-Location: /ipfs/{cid}?format=raw` header in order for +this request to be able to be successfully cached. + ### `Accept-Ranges` (response header) Optional, returned to explicitly indicate if gateway supports partial HTTP From 209783a492732e9d5565971b317e6e91bf21cec0 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 17 Apr 2024 21:20:48 +0200 Subject: [PATCH 2/5] gateway: clarify SHOULDs for content-location and some cleanup --- src/http-gateways/path-gateway.md | 63 ++++++++++++++++---------- src/http-gateways/trustless-gateway.md | 21 +++++++-- 2 files changed, 56 insertions(+), 28 deletions(-) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index 9c3296da..f024fcab 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -4,7 +4,7 @@ description: > The comprehensive low-level HTTP Gateway enables the integration of IPFS resources into the HTTP stack through /ipfs and /ipns namespaces, supporting both deserialized and verifiable response types. -date: 2023-03-30 +date: 2024-04-17 maturity: reliable editors: - name: Marcin Rataj @@ -242,6 +242,16 @@ These are the equivalents: When both `Accept` HTTP header and `format` query parameter are present, `Accept` SHOULD take precedence. +A Client SHOULD include the `format` query parameter in the request URL, in +addition to the `Accept` header. This provides the best interoperability and +ensures consistent HTTP cache behavior across various gateway implementations. + +A Gateway SHOULD include the +[`Content-Location`](#content-location-response-header) header in the response when: +- the request contains an `Accept` header specifying a well-known response + format, but the URL does not include the `format` query parameter +- the `format` parameter is present, but does not match the format from `Accept` + ### `dag-scope` (request query parameter) Only used on CAR requests, same as :ref[dag-scope] from :cite[trustless-gateway]. @@ -486,12 +496,35 @@ To illustrate, `?filename=testтест.pdf` should produce: not attempt to render raw bytes. CID and `.bin` file extension should be used if a custom `filename` was not provided with the request. +### `Content-Location` (response header) + +Returned when a non-default content format has been negotiated with the +[`Accept` header](#accept-request-header) but `format` was missing from the URL. + +The value of this field SHOULD include +the URL of the resource with the `format` query parameter included, so that +generic HTTP caches can store deserialized, CAR, and block responses separately. + +:::note + +For example, a request to `/ipfs/{cid}` with `Accept: application/vnd.ipld.raw` +SHOULD return a `Content-Location: /ipfs/{cid}?format=raw` header in order for +block response to be cached separately from deserialized one. + +::: + ### `Content-Length` (response header) Represents the length of returned HTTP payload. +:::warning + + + NOTE: the value may differ from the real size of requested data if compression or chunked `Transfer-Encoding` are used. - +See [ipfs/specs#461](https://github.com/ipfs/specs/issues/461). + +::: ### `Content-Range` (response header) @@ -499,17 +532,6 @@ Returned only when request was a [`Range`](#range-request-header) request. See Section 14.4 of :cite[rfc9110]. -### `Content-Location` (response header) - -Returned when a non-default content format has been negotiated with the -[`Accept` header](#accept-request-header). The value of this field SHOULD include -the URL of the resource with the `format` query parameter included, so that -generic HTTP caches can store deserialized, CAR, and block responses separately. - -For example, a request to `/ipfs/{cid}` with `Accept: application/vnd.ipld.raw` -SHOULD return a `Content-Location: /ipfs/{cid}?format=raw` header in order for -this request to be able to be successfully cached. - ### `Accept-Ranges` (response header) Optional, returned to explicitly indicate if gateway supports partial HTTP @@ -524,8 +546,6 @@ deterministic. Returned only when response status code is [`301` Moved Permanently](#301-moved-permanently). The value informs the HTTP client about new URL for requested resource. -This header is more widely used in [SUBDOMAIN_GATEWAY.md](./SUBDOMAIN_GATEWAY.md#location-response-header). - #### Use in directory URL normalization Gateway MUST return a redirect when a valid UnixFS directory was requested @@ -541,6 +561,10 @@ It also ensures the same behavior on path gateways (`https://example.com/ipfs/ci and origin-isolated HTTP contexts `https://cid.ipfs.dweb.link` or non-HTTP URLs like `ipfs://cid`, where empty path component is implicit `/`. +#### Use in interop with Subdomain Gateway + +See [`Location` section](https://specs.ipfs.tech/http-gateways/subdomain-gateway/#location-response-header) of :cite[subdomain-gateway]. + ### `X-Ipfs-Path` (response header) Used for HTTP caching and indicating the IPFS address of the data. @@ -578,15 +602,6 @@ NOTE: while the first CID will change every time any article is changed, the last root (responsible for specific article or a subdirectory) may not change at all, allowing for smarter caching beyond what standard Etag offers. - - ### `X-Content-Type-Options` (response header) Optional, present in certain response types: diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index 71ae1f30..cf2f76b7 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -4,7 +4,7 @@ description: > The minimal subset of HTTP Gateway response types facilitates data retrieval via CID and ensures integrity verification, all while eliminating the need to trust the gateway itself. -date: 2023-06-20 +date: 2024-04-17 maturity: reliable editors: - name: Marcin Rataj @@ -90,7 +90,14 @@ mode (no deserialized responses) and `Accept` header is missing. ## Request Query Parameters -### :dfn[dag-scope] (request query parameter) +### :dfn[`format`] (request query parameter) + +Same as [`format`](https://specs.ipfs.tech/http-gateways/path-gateway/#format-request-query-parameter) in :cite[path-gateway], but with limited number of supported response types: +- `format=raw` → `application/vnd.ipld.raw` +- `format=car` → `application/vnd.ipld.car` +- `format=ipns-record` → `application/vnd.ipfs.ipns-record` + +### :dfn[`dag-scope`] (request query parameter) Optional, `dag-scope=(block|entity|all)` with default value `all`, only available for CAR requests. @@ -111,7 +118,7 @@ path segments. When present, returned `Etag` must include unique prefix based on the passed scope type. -### :dfn[entity-bytes] (request query parameter) +### :dfn[`entity-bytes`] (request query parameter) The optional `entity-bytes=from:to` parameter is available only for CAR requests. @@ -203,6 +210,12 @@ If a CAR stream was requested: MUST be returned and set to `attachment` to ensure requested bytes are not rendered by a web browser. +### `Content-Location` (response header) + +Same as in :cite[path-gateway], SHOULD be returned when Trustless Gateway +supports more than a single response format and the `format` query parameter is +missing or does not match well-known format from `Accept` header. + # Block Responses (application/vnd.ipld.raw) An opaque bytes matching the requested block CID @@ -217,7 +230,7 @@ A CAR stream for the requested content type (with optional `order` and `dups` params), path and optional `dag-scope` and `entity-bytes` URL parameters. -## CAR version +## CAR version (content type parameter) Value returned in [`CarV1Header.version`](https://ipld.io/specs/transport/car/carv1/#header) From 11100a79a45fee7eed39eb025e576572253f47b9 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 17 Apr 2024 21:24:48 +0200 Subject: [PATCH 3/5] gateway: explicit should for trustless clients This is duplicated content, but is harmless, and having it here will save everyone a lot of headache debugging weird HTTP caching setups. --- src/http-gateways/trustless-gateway.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index cf2f76b7..eaa59f5c 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -97,6 +97,10 @@ Same as [`format`](https://specs.ipfs.tech/http-gateways/path-gateway/#format-re - `format=car` → `application/vnd.ipld.car` - `format=ipns-record` → `application/vnd.ipfs.ipns-record` +A Client SHOULD include the `format` query parameter in the request URL, in +addition to the `Accept` header. This provides the best interoperability and +ensures consistent HTTP cache behavior across various gateway implementations. + ### :dfn[`dag-scope`] (request query parameter) Optional, `dag-scope=(block|entity|all)` with default value `all`, only available for CAR requests. From b94f597f3f88c86f41733d8151cad478c413bf4e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 17 Apr 2024 10:37:14 +0200 Subject: [PATCH 4/5] gateway: car-* request query parameters --- src/http-gateways/path-gateway.md | 12 ++++++++++++ src/http-gateways/trustless-gateway.md | 15 +++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index f024fcab..a73e16e7 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -260,6 +260,18 @@ Only used on CAR requests, same as :ref[dag-scope] from :cite[trustless-gateway] Only used on CAR requests, same as :ref[entity-bytes] from :cite[trustless-gateway]. +### `car-version` (request query parameter) + +Only used on CAR requests, same as :ref[car-version] from :cite[trustless-gateway]. + +### `car-order` (request query parameter) + +Only used on CAR requests, same as :ref[car-order] from :cite[trustless-gateway]. + +### `car-dups` (request query parameter) + +Only used on CAR requests, same as :ref[car-dups] from :cite[trustless-gateway]. + # HTTP Response ## Response Status Codes diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index eaa59f5c..51b238f4 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -194,6 +194,21 @@ returned: returned to the client, the HTTP status code has already been sent to the client. +### :dfn[`car-version`] (request query parameter) + +Only used on CAR requests, it is the same as [CAR version](#car-version) content +type parameter. In case both are present, the value in the HTTP Header has priority. + +### :dfn[`car-order`] (request query parameter) + +Only used on CAR requests, it is the same as [CAR `order`](#car-order-content-type-parameter) +content type parameter. In case both are present, the value in the HTTP Header has priority. + +### :dfn[`car-dups`] (request query parameter) + +Only used on CAR requests, it is the same as [CAR `dups`](#car-dups-content-type-parameter) +content type parameter. In case both are present, the value in the HTTP Header has priority. + # HTTP Response Below MUST be implemented **in addition** to "HTTP Response" of :cite[path-gateway]. From 1defdd3f1df8512a96f5a2e84f9083584417c2bc Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 17 Apr 2024 22:01:42 +0200 Subject: [PATCH 5/5] gateway: car params and Content-Location --- src/http-gateways/path-gateway.md | 10 +++++----- src/http-gateways/trustless-gateway.md | 23 ++++++++++++++++------- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index a73e16e7..ca89bb61 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -254,23 +254,23 @@ A Gateway SHOULD include the ### `dag-scope` (request query parameter) -Only used on CAR requests, same as :ref[dag-scope] from :cite[trustless-gateway]. +Optional, can be used to limit the scope of verifiable DAG requests such as CAR, same as :ref[dag-scope] from :cite[trustless-gateway]. ### `entity-bytes` (request query parameter) -Only used on CAR requests, same as :ref[entity-bytes] from :cite[trustless-gateway]. +Optional, can be used to limit the scope of verifiable DAG requests such as CAR, same as :ref[entity-bytes] from :cite[trustless-gateway]. ### `car-version` (request query parameter) -Only used on CAR requests, same as :ref[car-version] from :cite[trustless-gateway]. +Optional, specific to CAR requests, same as :ref[car-version] from :cite[trustless-gateway]. ### `car-order` (request query parameter) -Only used on CAR requests, same as :ref[car-order] from :cite[trustless-gateway]. +Optional, specific to CAR requests, same as :ref[car-order] from :cite[trustless-gateway]. ### `car-dups` (request query parameter) -Only used on CAR requests, same as :ref[car-dups] from :cite[trustless-gateway]. +Optional, specific to CAR requests, same as :ref[car-dups] from :cite[trustless-gateway]. # HTTP Response diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index 51b238f4..4e728d5d 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -196,18 +196,27 @@ returned: ### :dfn[`car-version`] (request query parameter) -Only used on CAR requests, it is the same as [CAR version](#car-version) content -type parameter. In case both are present, the value in the HTTP Header has priority. +Optional, only used on CAR requests. + +Serves same purpose as [CAR `version` content type parameter](#car-version-content-type-parameter). + +In case both are present in the request, the value from the [`Accept`](#accept-request-header) HTTP Header has priority and a matching [`Content-Location`](#content-location-response-header) SHOULD be returned with the response. ### :dfn[`car-order`] (request query parameter) -Only used on CAR requests, it is the same as [CAR `order`](#car-order-content-type-parameter) -content type parameter. In case both are present, the value in the HTTP Header has priority. +Optional, only used on CAR requests. + +Serves same purpose as [CAR `order` content type parameter](#car-order-content-type-parameter). + +In case both are present in the request, the value from the [`Accept`](#accept-request-header) HTTP Header has priority and a matching [`Content-Location`](#content-location-response-header) SHOULD be returned with the response. ### :dfn[`car-dups`] (request query parameter) -Only used on CAR requests, it is the same as [CAR `dups`](#car-dups-content-type-parameter) -content type parameter. In case both are present, the value in the HTTP Header has priority. +Optional, only used on CAR requests. + +Serves same purpose as [CAR `dups` content type parameter](#car-dups-content-type-parameter). + +In case both are present in the request, the value from the [`Accept`](#accept-request-header) HTTP Header has priority and a matching [`Content-Location`](#content-location-response-header) SHOULD be returned with the response. # HTTP Response @@ -249,7 +258,7 @@ A CAR stream for the requested content type (with optional `order` and `dups` params), path and optional `dag-scope` and `entity-bytes` URL parameters. -## CAR version (content type parameter) +## CAR `version` (content type parameter) Value returned in [`CarV1Header.version`](https://ipld.io/specs/transport/car/carv1/#header)