From 01a3f0954716a7e99979c9136735c89d372c00c9 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 12 May 2023 13:58:21 +0200 Subject: [PATCH] ipip: Streaming Routing V1 HTTP API --- src/ipips/ipip-0410.md | 83 ++++++++++++++++++++++++++++++++++ src/routing/http-routing-v1.md | 21 ++++++++- 2 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 src/ipips/ipip-0410.md diff --git a/src/ipips/ipip-0410.md b/src/ipips/ipip-0410.md new file mode 100644 index 00000000..d8d716c3 --- /dev/null +++ b/src/ipips/ipip-0410.md @@ -0,0 +1,83 @@ +--- +title: "IPIP-0410: Streaming Routing V1 HTTP API" +date: 2023-05-12 +ipip: proposal +editors: + - name: Henrique Dias + github: hacdias + url: https://hacdias.com/ +relatedIssues: + - https://github.com/ipfs/specs/issues/344 + - https://github.com/ipfs/boxo/pull/18 + - https://github.com/ipfs/kubo/pull/9868 + - https://github.com/ipfs/kubo/pull/9874 +order: 410 +tags: ['ipips'] +--- + +Introduce backwards-compatible streaming support to the Routing V1 HTTP API. +For this, we use the `Accept` HTTP header (:cite[rfc9110]) for content type negotiation, as well +as the Newline Delimited JSON ([NDJSON]) format. + +## Motivation + +The main motivation for this change is to allow servers to respond faster to the +client with provider records, as soon as they are available. In the current state, +the client requests a list of providers for a CID from the server. Then, the client +has to wait for the server to collect their final list of providers. After that, +the server can respond with the full list of providers. + +With streaming support, the server is able to respond with provider records as soon +as they are available. This reduces latency and allows for faster content discovery. + +In addition, streaming responses may produce an unlimited amount of results, which +is not the case for non-streamed responses. + +## Detailed Design + +In summary, streaming is supported by using the `Accept` HTTP header, which is used +for content type negotiation as described in :cite[rfc9110]. The client sends an +`Accept` HTTP header starting with `application/x-ndjson`, which is the content +type for [NDJSON]. The following happens: + +- The client adds the `Accept` HTTP header in the request starting with `application/x-ndjson`. +- The server checks the `Accept` HTTP header from the request and, if it contains +`application/x-ndjson`, they reply with NDJSON. If they don't support NDJSON, they +can reply with JSON. +- The server response MUST contain a `Content-Type` HTTP header indicating the +response type, which may be either `application/json` for non-streaming responses, +and `application/x-ndjson` for streamed responses. + +For more details regarding the design, check :cite[http-routing-v1]. + +## Design Rationale + +This feature is designed such that it does not break compatibility with existing +clients and servers. The `Accept` HTTP header is OPTIONAL. By default, the server +MUST respond with `application/json` unless the client explicitly asked for +`application/x-ndjson`. If the server does not support NDJSON, it is allowed +to still respond with non-streamed JSON. + +### User Benefit + +Users (clients) will benefit from this change as the servers will now be able +to respond more promptly to provider record requests. Instead of waiting for the whole +list to be constructed, servers can now return each provider record one by one, +in a streaming fashion. + +### Compatibility + +The introduced changes are backwards-compatible. The introduced header is completely +optional, and a server that does not support streaming is able to respond with a non-streaming +response to the client. Equally, non-streaming responses are the default. Therefore, a +client that does not support streaming will not receive a streamed response. + +### Security + +Security considerations are equivalent as the ones in :cite[ipip-0337]. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + +[NDJSON]: http://ndjson.org/ diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 27170a62..da64f20c 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -11,6 +11,9 @@ editors: github: guseggert - name: Masih H. Derkani github: masih + - name: Henrique Dias + url: https://hacdias.com/ + github: hacdias xref: - ipns-record order: 0 @@ -139,7 +142,23 @@ This API does not support pagination, but optional pagination can be added in a ## Streaming -This API does not currently support streaming, however it can be added in the future through a backwards-compatible update by using a content type other than `application/json`. +JSON-based endpoints support streaming requests made by sending an `Accept` HTTP Header containing +`application/x-ndjson`. The response will be formatted as [Newline Delimited JSON (ndjson)](https://github.com/ndjson/ndjson-spec), +with one *read provider record* per line: + + +```json +{"Protocol": "", "Schema": "", ...} +{"Protocol": "", "Schema": "", ...} +{"Protocol": "", "Schema": "", ...} +... +``` + +Streaming is backwards-compatible with clients that do not support streaming. Please note the following: + +- Requests without an `Accept` header MUST default to regular, non-streaming, responses. +- The server MAY respond with non-streaming response even if the client requests streaming. +- The server MUST NOT respond with streaming response if the client did not request so. ## Error Codes