From 633668989f17e2b613beeacbce3bbdd129ddac17 Mon Sep 17 00:00:00 2001 From: "Masih H. Derkani" Date: Mon, 13 Feb 2023 11:36:08 +0000 Subject: [PATCH 01/16] Reintroduce provide records via `PUT` to HTTP delegated routing Previous work reduced the scope of IPIP-337 to read operations only. The changes here re-introduce the write operations originally written by @guseggert back into IPIP-337. The specification documents the ability to provide Bitswap records over `PUT` requests with advisory TTL. An implementation of the specification is already present in go-libipfs. See: - https://github.com/ipfs/specs/pull/370 --- routing/DELEGATED_CONTENT_ROUTING_HTTP.md | 106 +++++++++++++++++++++- 1 file changed, 103 insertions(+), 3 deletions(-) diff --git a/routing/DELEGATED_CONTENT_ROUTING_HTTP.md b/routing/DELEGATED_CONTENT_ROUTING_HTTP.md index 1f441230c..761c7a4b2 100644 --- a/routing/DELEGATED_CONTENT_ROUTING_HTTP.md +++ b/routing/DELEGATED_CONTENT_ROUTING_HTTP.md @@ -66,13 +66,13 @@ Specifications for some transfer protocols are provided in the "Transfer Protoco ### `GET /routing/v1/providers/{CID}` -#### Response codes +#### `GET` Response codes - `200` (OK): the response body contains 0 or more records - `404` (Not Found): must be returned if no matching records are found - `422` (Unprocessable Entity): request does not conform to schema or semantic constraints -#### Response Body +#### `GET` Response Body ```json { @@ -90,6 +90,48 @@ Response limit: 100 providers Each object in the `Providers` list is a *read provider record*. +### `PUT /routing/v1/providers` + +#### `PUT` Response codes + +- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) +- `400` (Bad Request): the server deems the request to be invalid and cannot process it +- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints +- `501` (Not Implemented): the server does not support providing records + +#### `PUT` Request Body + +```json +{ + "Providers": [ + { + "Protocol": "", + "Schema": "bitswap", + ... + } + ] +} +``` + +Each object in the `Providers` list is a *write provider record*. + +#### `PUT` Response Body + + ```json + { + "ProvideResults": [ + { ... } + ] + } + ``` + +- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object (called "Write Provider Records Response" in the Known Transfer Protocols section) + - This may contain output information such as TTLs, errors, etc. + - It is undefined whether the server will allow partial results +- The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request +- Default limit of 100 keys per request +- Implements pagination according to the Pagination section + ## Pagination This API does not support pagination, but optional pagination can be added in a backwards-compatible spec update. @@ -118,7 +160,7 @@ limits, allowing every site to query the API for results: ```plaintext Access-Control-Allow-Origin: * -Access-Control-Allow-Methods: GET, OPTIONS +Access-Control-Allow-Methods: GET, PUT, OPTIONS ``` ## Known Transfer Protocols @@ -148,6 +190,60 @@ Specification: [ipfs/specs/BITSWAP.md](https://github.com/ipfs/specs/blob/main/B The server should respect a passed `transport` query parameter by filtering against the `Addrs` list. +#### Bitswap Write Provider Records + +```json +{ + "Protocol": "transport-bitswap", + "Schema": "bitswap", + "Signature": "", + "Payload": "" +} +``` + +- `Signature`: a multibase-encoded signature of the sha256 hash of the `Payload` field, signed using the private key of the Peer ID specified in the `Payload` JSON. Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise. + + - Servers may ignore this field if they do not require signature verification. +- `Payload`: a string containing a serialized JSON object which conforms with the following schema: + + ```json + { + "Keys": ["cid1", "cid2"], + "Timestamp": 0, + "AdvisoryTTL": 0, + "ID": "12D3K...", + "Addrs": ["/ip4/..."] + } + ``` + +- `Keys` is a list of the CIDs being provided +- `Timestamp` is the current time +- `AdvisoryTTL` is the time by which the caller expects the server to keep the record available + - If this value is unknown, the caller may use a value of 0 +- `ID` is the peer ID that was used to sign the record +- `Addrs` is a list of string-encoded multiaddrs + +A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code should be returned if the signature check fails. + +Note that this only supports Peer IDs expressed as identity multihashes. Peer IDs with older key types that exceed 42 bytes are not verifiable since they only contain a hash of the key, not the key itself. +Normally, if the Peer ID contains only a hash of the key, then the key is obtained out-of-band (e.g. by fetching the block via IPFS). +If support for these Peer IDs is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` and `PublicKeyType` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. + +The `Payload` field is a string, not a proper JSON object, to prevent its contents from being accidentally parsed and re-encoded by intermediaries, which may change the order of JSON fields and thus cause the record to fail validation. + +#### Write Provider Records Response + +```json +{ + "AdvisoryTTL": 0 +} +``` + +- `AdvisoryTTL` is the time at which the server expects itself to drop the record + - If less than the `AdvisoryTTL` in the request, then the client should re-issue the request by that point + - If greater than the `AdvisoryTTL` in the request, then the server expects the client to be responsible for the content for up to that amount of time (TODO: this is ambiguous) + - If 0, the server makes no claims about the lifetime of the record + ### Filecoin Graphsync Multicodec name: `transport-graphsync-filecoinv1` @@ -173,3 +269,7 @@ Specification: [ipfs/go-graphsync/blob/main/docs/architecture.md](https://github - `PieceCID`: the CID of the [piece](https://spec.filecoin.io/systems/filecoin_files/piece/#section-systems.filecoin_files.piece) within which the data is stored - `VerifiedDeal`: whether the deal corresponding to the data is verified - `FastRetrieval`: whether the provider claims there is an unsealed copy of the data available for fast retrieval + +#### Filecoin Graphsync Write Provider Records + +There is currently no specified schema. From 68c0dff937c68065c740e389400f4f139ec1ddac Mon Sep 17 00:00:00 2001 From: "Masih H. Derkani" Date: Mon, 13 Feb 2023 19:03:49 +0000 Subject: [PATCH 02/16] Remove pagination line --- routing/DELEGATED_CONTENT_ROUTING_HTTP.md | 1 - 1 file changed, 1 deletion(-) diff --git a/routing/DELEGATED_CONTENT_ROUTING_HTTP.md b/routing/DELEGATED_CONTENT_ROUTING_HTTP.md index 761c7a4b2..efca882e6 100644 --- a/routing/DELEGATED_CONTENT_ROUTING_HTTP.md +++ b/routing/DELEGATED_CONTENT_ROUTING_HTTP.md @@ -130,7 +130,6 @@ Each object in the `Providers` list is a *write provider record*. - It is undefined whether the server will allow partial results - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request - Default limit of 100 keys per request -- Implements pagination according to the Pagination section ## Pagination From 5bc8b14540cb2cc1b6446eb1f3b5b873aec16ce1 Mon Sep 17 00:00:00 2001 From: "Masih H. Derkani" Date: Tue, 14 Feb 2023 11:15:21 +0000 Subject: [PATCH 03/16] Add memo to /IPIP This is to facilitate a place for discussing compatibility with the existing nodes. --- IPIP/0337-delegated-routing-http-api.md | 2 +- ...egated-routing-http-api-provide-records.md | 64 +++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 IPIP/0338-delegated-routing-http-api-provide-records.md diff --git a/IPIP/0337-delegated-routing-http-api.md b/IPIP/0337-delegated-routing-http-api.md index ab9df0673..4945d7ee3 100644 --- a/IPIP/0337-delegated-routing-http-api.md +++ b/IPIP/0337-delegated-routing-http-api.md @@ -86,7 +86,7 @@ and increasing data availability. IPFS Stewards will implement this API in [go-delegated-routing](https://github.com/ipfs/go-delegated-routing), using breaking changes in a new minor version. Because the existing Reframe spec can't be safely used in JavaScript and we won't be investing time and resources into changing the wire format implemented in edelweiss to fix it, the experimental support for Reframe in Kubo will be deprecated in the next release and delegated content routing will subsequently use this HTTP API. -We may decide to re-add Reframe support in the future once these issues have been resolved.- +We may decide to re-add Reframe support in the future once these issues have been resolved. #### Forwards Compatibility diff --git a/IPIP/0338-delegated-routing-http-api-provide-records.md b/IPIP/0338-delegated-routing-http-api-provide-records.md new file mode 100644 index 000000000..09b9dd221 --- /dev/null +++ b/IPIP/0338-delegated-routing-http-api-provide-records.md @@ -0,0 +1,64 @@ +# IPIP-338: Delegated Content Routing HTTP Provide Records API + +- Start Date: 2023-02-14 +- Related Issues: + - https://github.com/ipfs/specs/pull/378 + +## Summary + +This IPIP extends the [IPIP-337 HTTP Delegated Routing API](0337-delegated-routing-http-api.md) to provide records over `PUT` requests. + +The work here was originally proposed as part of IPIP-337, and eventually was separated into its own IPIP in order to reduce the scope of original work, while enabling iterative release of the HTTP delegated routing APIs. + +## Motivation + +The IPFS interaction with DHT includes both read and write operations. +A user can provide records, advertising the presence of content, as well as looking up providers for a given CID. +The specification proposed by [IPIP-337](0337-delegated-routing-http-api.md) offers an idiomatic first-class support for offloading the lookup portion of this interaction onto other processes and/or servers. +Following the same motivations that inspired [IPIP-337](0337-delegated-routing-http-api.md), this document expands the HTTP APIs to also +offload the ability to provide records noto a third-party system. + +## Detailed design + +The API extensions are added to the [Delegated Content Routing HTTP API spec/`PUT`](../routing/DELEGATED_CONTENT_ROUTING_HTTP.md#put-routingv1providers) section, along with complimentary sections that outline known formats followed by example payload. + +## Design rationale + +The rationale for the design of `PUT` operations closely follows the reasoning listed in [IPIP-337](0337-delegated-routing-http-api.md#design-rationale). +The design uses a human-readable request/response structure with extensibility in mind. +The specification imposes no restrictions on the schema nor the protocol advertised in provider records. +The hope is that such extensibility will encourage and inspire innovation for better transfer protocols. + +### User benefit + +Expanding the user benefits listed as part of [IPIP-337](0337-delegated-routing-http-api.md#user-benefit), in the context of content routing write operations are typically more expensive than read operations. They involve book keeping such as TTL, gossip propagation, etc. +Therefore, it is highly desirable to reduce the burden of advertising provider records onto the network by means of delegation through simple to use HTTP APIs. + +### Compatibility + +#### Backwards Compatibility + +##### DHT + +The `PUT` APIs proposed here require a new data format for specifying provider records. +Since the records must include a valid signature, records published through HTTP delegated routing must be resigned. + +##### Reframe + +See [IPIP-337/Backwards Compatibility](0337-delegated-routing-http-api.md#backwards-compatibility). + +#### Forwards Compatibility + +See [IPIP-337/Forwads Compatibility](0337-delegated-routing-http-api.md#forwards-compatibility). + +### Security + +See [IPIP-337/Security](0337-delegated-routing-http-api.md#security). + +### Alternatives + +- Reframe (general-purpose RPC) was evaluated, see "Design rationale" section for rationale why it was not selected. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 4db804a4f311da8fac3e2c4e4b6d7c3e05377aa3 Mon Sep 17 00:00:00 2001 From: "Masih H. Derkani" Date: Wed, 15 Feb 2023 22:56:05 +0000 Subject: [PATCH 04/16] Rename file to match IPIP PR number --- ...s.md => 0378-delegated-routing-http-api-provide-records.md} | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) rename IPIP/{0338-delegated-routing-http-api-provide-records.md => 0378-delegated-routing-http-api-provide-records.md} (96%) diff --git a/IPIP/0338-delegated-routing-http-api-provide-records.md b/IPIP/0378-delegated-routing-http-api-provide-records.md similarity index 96% rename from IPIP/0338-delegated-routing-http-api-provide-records.md rename to IPIP/0378-delegated-routing-http-api-provide-records.md index 09b9dd221..65504e3b2 100644 --- a/IPIP/0338-delegated-routing-http-api-provide-records.md +++ b/IPIP/0378-delegated-routing-http-api-provide-records.md @@ -1,4 +1,4 @@ -# IPIP-338: Delegated Content Routing HTTP Provide Records API +# IPIP-378: Delegated Content Routing HTTP Provide Records API - Start Date: 2023-02-14 - Related Issues: @@ -28,6 +28,7 @@ The rationale for the design of `PUT` operations closely follows the reasoning l The design uses a human-readable request/response structure with extensibility in mind. The specification imposes no restrictions on the schema nor the protocol advertised in provider records. The hope is that such extensibility will encourage and inspire innovation for better transfer protocols. +In order to reduce barrier for adoption, the existing ### User benefit From 0f74972c4ef3e643c5b8d0343cb2773a40e5ea44 Mon Sep 17 00:00:00 2001 From: Will Scott Date: Sat, 11 Nov 2023 14:32:33 +0300 Subject: [PATCH 05/16] lint --- src/routing/http-routing-v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 27c3dbb73..08fe6c3f3 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -371,7 +371,7 @@ the `/ipfs/bitswap[/*]` libp2p protocol. - `AdvisoryTTL` is the time by which the caller expects the server to keep the record available, specified in milliseconds. - If this value is unknown, the caller may use a value of 0 - `ID` is the peer ID that was used to sign the record -- `Addrs` is a list of string-encoded multiaddrs +- `Addrs` is a list of string-encoded multiaddrs A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code should be returned if the signature check fails. From 921b0b6ab4361b3a6b7185a6371adb2bb84f4131 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 4 Dec 2023 02:45:14 +0100 Subject: [PATCH 06/16] ipip378: define announcement schema --- src/routing/http-routing-v1.md | 189 +++++++++++++++++++++------------ 1 file changed, 124 insertions(+), 65 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 08fe6c3f3..ae6207516 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -106,17 +106,27 @@ Each object in the `Providers` list is a record conforming to a schema, usually ```json { - "Providers": [ - { - "Protocol": "", - "Schema": "bitswap", - ... - } - ] + "Providers": [ + { + "Schema": "announcement", + ... + } + ] } ``` -Each object in the `Providers` list is a *write provider record*. + +Each object in the `Providers` list is a *write provider record* entry. + +Server SHOULD accept representing writes is [Announcement Schema](#announcement-schema). + +:::warn + +TODO: is below a sensible limit? + +There SHOULD be no more than 100 `Providers` per request. + +::: #### `PUT` Response Body @@ -130,9 +140,8 @@ Each object in the `Providers` list is a *write provider record*. - `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object - This may contain output information such as TTLs, errors, etc. - - It is undefined whether the server will allow partial results + - It is undefined whether the server will allow partial results - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request -- Default limit of 100 keys per request ## Peer Routing API @@ -155,10 +164,10 @@ represented as a CIDv1 encoded with `libp2p-key` codec. { "Peers": [ { - "Schema": "", - "Protocols": ["", "", ...], + "Schema": "peer", "ID": "bafz...", "Addrs": ["/ip4/..."], + "Protocols": ["", "", ...], ... }, ... @@ -172,6 +181,34 @@ The client SHOULD be able to make a request with `Accept: application/x-ndjson` Each object in the `Peers` list is a record conforming to the [Peer Schema](#peer-schema). +### `PUT /routing/v1/peers` + +#### `PUT` Response codes + +- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) +- `400` (Bad Request): the server deems the request to be invalid and cannot process it +- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints +- `501` (Not Implemented): the server does not support providing records + +#### `PUT` Request Body + +```json +{ + "Providers": [ + { + "Schema": "announcement", + ... + } + ] +} +``` + + +Each object in the `Providers` list is a *write provider record* entry. + +Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema) +objects with `CID` list. + ## IPNS API ### `GET /routing/v1/ipns/{name}` @@ -318,82 +355,104 @@ the case, the field MUST be ignored. ::: -### Legacy Schemas +### Announcement Schema -Legacy schemas include `ID` and optional `Addrs` list just like -the [`peer` schema](#peer-schema) does. +The `announcement` schema can be used in `PUT` operations to announce content providers or peer routing information. -These schemas are deprecated and SHOULD be replaced with `peer` over time, but -MAY be returned by some legacy endpoints. In such case, a client MAY parse -them the same way as the `peer` schema. - -#### Bitswap Schema - -A legacy schema used by some routers to indicate a peer supports retrieval over -the `/ipfs/bitswap[/*]` libp2p protocol. ```json -{ - "Protocol": "transport-bitswap", - "Schema": "bitswap", - "ID": "bafz...", - "Addrs": ["/ip4/..."] -} + { + "Schema": "announcement", + "Payload": { + "CID": ["cid1", "cid2", ...], + "Scope": "block", + "Timestamp": "YYYY-MM-DDT23:59:59Z", + "TTL": 0, + "ID": "12D3K...", + "Addrs": ["/ip4/...", ...], + "Protocols": ["foo", ...], + "Metadata": "mbase64-blob", + }, + "Signature": "mbase64-signature" + } ``` -#### Bitswap Write Provider Records +- `Schema`: tells the server to interpret the JSON object as announce provider +- `Payload`: is a DAG-JSON-compatible object with a subset of the below fields + - `CID` is a list of the CIDs being provided. + - Skipped when used for `PUT /routing/v1/peers` + - `Scope` is an optional hint that provides semantic meaning about announced identifies: + - `block` announces only the individual blocks (implicit default if `Scope` is missing). + - `entity` announces enumerable entities behind the CIDs (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate HAMT-sharded UnixFS directory). + - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). + - `Timetamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). + - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified in milliseconds. + - If this value is unknown, the caller may skip this field, or use a value of 0. The server's default will be used. + - `ID` is Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field. + - `Addrs` is an a list of string-encoded multiaddrs without `/p2p/peerID` suffix. + - `Protocols` is a list of protocols supported by `ID` and/or `Addrs`, if known upfront. + - `Metadata` is a string with multibase-encoded binary metadata that should be passed as-is +- `Signature` is a string with multibase-encoded binary signature that provides integrity and authenticity of the `Payload` field. + - Signature is created by following below steps: + 1. Convert `Payload` to deterministic, ordered [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/) map notation + 2. Prefix the DAG-JSON bytes with ASCII string `PUT /routing/v1 announcement:` + 3. Sign the bytes with the private key of the Peer ID specified in the `Payload.ID`. + - Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise. + - Client SHOULD sign every announcement. + - Servers SHOULD verify signature before accepting a record, unless running in a trusted environment. + - ED25519 and other small public keys MUST be inlined inside of the `ID` field with the identity multihash type. + - Key types that exceed 42 bytes (e.g. RSA) SHOULD NOT be inlined, the `ID` field should only include the multihash of the key. The key itself SHOULD be obtained out-of-band (e.g. by fetching the block via IPFS) and cached. + If support for big keys is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. + - A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if the `Signature` check fails. + +:::warn + +TODO: what should be the limits? Max number of CIDs per `announcement` ? + +::: + +#### Use in PUT responses + +Server MAY return additional TTL information if the TTL is not provided in the request, +or if server policy is to provide TTL different than the requested one. ```json { - "Protocol": "transport-bitswap", - "Schema": "bitswap", - "Signature": "", - "Payload": "" + "Schema": "announcement", + "Payload": { + "TTL": 17280000 + } } ``` -- `Signature`: a multibase-encoded signature of the sha256 multihash of the `Payload` field, signed using the private key of the Peer ID specified in the `Payload` DAG-JSON. Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise. - - Servers may ignore this field if they do not require signature verification. -- `Payload`: a string containing a serialized JSON object which conforms with the following schema: - - ```json - { - "Keys": ["cid1", "cid2"], - "Timestamp": 0, - "AdvisoryTTL": 0, - "ID": "12D3K...", - "Addrs": ["/ip4/..."] - } - ``` +- `TTL` in response is the time at which the server expects itself to drop the record + - If less than the `TTL` in the request, then the client SHOULD repeat announcement earlier, before the announcement TTL expires and is forgotten by the routing system + - If greater than the `TTL` in the request, then the server client SHOULD save resources and not repeat announcement until the announcement TTL expires and is forgotten by the routing system + - If `0`, the server makes no claims about the lifetime of the record -- `Keys` is a list of the CIDs being provided -- `Timestamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339) -- `AdvisoryTTL` is the time by which the caller expects the server to keep the record available, specified in milliseconds. - - If this value is unknown, the caller may use a value of 0 -- `ID` is the peer ID that was used to sign the record -- `Addrs` is a list of string-encoded multiaddrs +### Legacy Schemas -A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code should be returned if the signature check fails. +Legacy schemas include `ID` and optional `Addrs` list just like +the [`peer` schema](#peer-schema) does. -Note that this only supports Peer IDs expressed as identity multihashes. Peer IDs with older key types that exceed 42 bytes are not verifiable since they only contain a hash of the key, not the key itself. -Normally, if the Peer ID contains only a hash of the key, then the key is obtained out-of-band (e.g. by fetching the block via IPFS). -If support for these Peer IDs is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` and `PublicKeyType` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. +These schemas are deprecated and SHOULD be replaced with `peer` over time, but +MAY be returned by some legacy endpoints. In such case, a client MAY parse +them the same way as the `peer` schema. -The `Payload` field is a string, not a proper JSON object, to prevent its contents from being accidentally parsed and re-encoded by intermediaries, which may change the order of JSON fields and thus cause the record to fail validation. +#### Bitswap Schema -#### Write Provider Records Response +A legacy schema used by some routers to indicate a peer supports retrieval over +the `/ipfs/bitswap[/*]` libp2p protocol. ```json { - "AdvisoryTTL": 0 + "Protocol": "transport-bitswap", + "Schema": "bitswap", + "ID": "bafz...", + "Addrs": ["/ip4/..."] } ``` -- `AdvisoryTTL` is the time at which the server expects itself to drop the record - - If less than the `AdvisoryTTL` in the request, then the client should re-issue the request by that point - - If greater than the `AdvisoryTTL` in the request, then the server expects the client to be responsible for the content for up to that amount of time (TODO: this is ambiguous) - - If 0, the server makes no claims about the lifetime of the record - #### Graphsync Schema A legacy schema used by some routers to indicate a peer supports retrieval over From 94429a5da23fd5110dcd24586a13be11d6302737 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 4 Dec 2023 03:07:05 +0100 Subject: [PATCH 07/16] chore: cleanup --- src/routing/http-routing-v1.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index ae6207516..158347e89 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -471,10 +471,6 @@ libp2p protocol. } ``` -#### Filecoin Graphsync Write Provider Records - -There is currently no specified schema. - [multibase]: https://github.com/multiformats/multibase [CIDv1]: https://github.com/multiformats/cid#cidv1 [multiaddr]: https://github.com/multiformats/multiaddr#specification From 05b000ed00b228c17c02dc8abceb84c53814cb86 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 4 Dec 2023 03:37:49 +0100 Subject: [PATCH 08/16] fix: lint --- src/routing/http-routing-v1.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 158347e89..f7e49b049 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -115,7 +115,6 @@ Each object in the `Providers` list is a record conforming to a schema, usually } ``` - Each object in the `Providers` list is a *write provider record* entry. Server SHOULD accept representing writes is [Announcement Schema](#announcement-schema). @@ -203,7 +202,6 @@ Each object in the `Peers` list is a record conforming to the [Peer Schema](#pee } ``` - Each object in the `Providers` list is a *write provider record* entry. Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema) @@ -359,7 +357,6 @@ the case, the field MUST be ignored. The `announcement` schema can be used in `PUT` operations to announce content providers or peer routing information. - ```json { "Schema": "announcement", From 30a5e3bf3603f9c3207668254a672704d1c3c2e0 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 7 Jan 2024 23:45:53 +0100 Subject: [PATCH 09/16] ipip-378: simplify put schema make entry limited to a single CID https://github.com/ipfs/specs/pull/378#discussion_r1413291960 --- src/routing/http-routing-v1.md | 67 ++++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 23 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index f7e49b049..69c532aef 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -95,13 +95,6 @@ Each object in the `Providers` list is a record conforming to a schema, usually ### `PUT /routing/v1/providers` -#### `PUT` Response codes - -- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) -- `400` (Bad Request): the server deems the request to be invalid and cannot process it -- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints -- `501` (Not Implemented): the server does not support providing records - #### `PUT` Request Body ```json @@ -127,6 +120,14 @@ There SHOULD be no more than 100 `Providers` per request. ::: +#### `PUT` Response codes + +- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) +- `400` (Bad Request): the server deems the request to be invalid and cannot process it +- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints +- `501` (Not Implemented): the server does not support providing records + + #### `PUT` Response Body ```json @@ -138,8 +139,8 @@ There SHOULD be no more than 100 `Providers` per request. ``` - `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object - - This may contain output information such as TTLs, errors, etc. - - It is undefined whether the server will allow partial results + - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. + - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why PUT failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request ## Peer Routing API @@ -182,13 +183,6 @@ Each object in the `Peers` list is a record conforming to the [Peer Schema](#pee ### `PUT /routing/v1/peers` -#### `PUT` Response codes - -- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) -- `400` (Bad Request): the server deems the request to be invalid and cannot process it -- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints -- `501` (Not Implemented): the server does not support providing records - #### `PUT` Request Body ```json @@ -207,6 +201,33 @@ Each object in the `Providers` list is a *write provider record* entry. Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema) objects with `CID` list. +#### `PUT` Response codes + +- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) +- `400` (Bad Request): the server deems the request to be invalid and cannot process it +- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints +- `501` (Not Implemented): the server does not support providing records + +#### `PUT` Response Body + + ```json + { + "ProvideResults": [ + { ... } + ] + } + ``` + +- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object + - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. + - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why PUT failed. +- The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request + +#### `PUT` Response Status Codes + +- `200` (OK): processed - inspect response to see if there are any `Error` results. +- `400` (Bad Request): unable to process PUT request, make sure JSON schema and values are correct. + ## IPNS API ### `GET /routing/v1/ipns/{name}` @@ -361,7 +382,7 @@ The `announcement` schema can be used in `PUT` operations to announce content pr { "Schema": "announcement", "Payload": { - "CID": ["cid1", "cid2", ...], + "CID": "bafy..cid", "Scope": "block", "Timestamp": "YYYY-MM-DDT23:59:59Z", "TTL": 0, @@ -376,12 +397,12 @@ The `announcement` schema can be used in `PUT` operations to announce content pr - `Schema`: tells the server to interpret the JSON object as announce provider - `Payload`: is a DAG-JSON-compatible object with a subset of the below fields - - `CID` is a list of the CIDs being provided. - - Skipped when used for `PUT /routing/v1/peers` - - `Scope` is an optional hint that provides semantic meaning about announced identifies: - - `block` announces only the individual blocks (implicit default if `Scope` is missing). - - `entity` announces enumerable entities behind the CIDs (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate HAMT-sharded UnixFS directory). - - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). + - `CID` is the CID being provided. + - This field is not presend when used for `PUT /routing/v1/peers` + - `Scope` is an optional hint that provides semantic meaning about announced identifies: + - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). + - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). + - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). - `Timetamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified in milliseconds. - If this value is unknown, the caller may skip this field, or use a value of 0. The server's default will be used. From 7b32f538d661ac829ba7477b44f90c5e17a4e9c5 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 5 Feb 2024 23:19:42 +0100 Subject: [PATCH 10/16] =?UTF-8?q?ipip-378:=20PUT=20=E2=86=92=20POST?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Following suggestion from https://github.com/ipfs/specs/pull/378#discussion_r1476257372 Fixed outdated links and statements. --- src/ipips/ipip-0378.md | 70 ++++++++++++++++++++++++++-------- src/routing/http-routing-v1.md | 37 +++++++++--------- 2 files changed, 72 insertions(+), 35 deletions(-) diff --git a/src/ipips/ipip-0378.md b/src/ipips/ipip-0378.md index 39ff0bb65..c5b587d0c 100644 --- a/src/ipips/ipip-0378.md +++ b/src/ipips/ipip-0378.md @@ -1,7 +1,7 @@ --- -title: "IPIP-0378: Delegated Content Routing HTTP Provide Records API" -date: 2023-02-14 -ipip: ratified +title: "IPIP-0378: Delegated Routing HTTP POST API" +date: 2024-02-05 +ipip: draft editors: - name: Masih H. Derkani github: masih @@ -16,7 +16,7 @@ tags: ['ipips'] ## Summary -This IPIP extends the [IPIP-337 HTTP Delegated Routing API](0337-delegated-routing-http-api.md) to provide records over `PUT` requests. +This IPIP extends the [IPIP-337 HTTP Delegated Routing API](https://specs.ipfs.tech/ipips/ipip-0337/) to support announcement of content and peer provider records over `POST` requests. The work here was originally proposed as part of IPIP-337, and eventually was separated into its own IPIP in order to reduce the scope of original work, while enabling iterative release of the HTTP delegated routing APIs. @@ -24,26 +24,45 @@ The work here was originally proposed as part of IPIP-337, and eventually was se The IPFS interaction with DHT includes both read and write operations. A user can provide records, advertising the presence of content, as well as looking up providers for a given CID. -The specification proposed by [IPIP-337](0337-delegated-routing-http-api.md) offers an idiomatic first-class support for offloading the lookup portion of this interaction onto other processes and/or servers. -Following the same motivations that inspired [IPIP-337](0337-delegated-routing-http-api.md), this document expands the HTTP APIs to also + +The specification proposed by +[IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337) offers an idiomatic +first-class support for offloading the lookup portion of this interaction onto +other processes and/or servers. + +Following the same motivations that inspired IPIP-337, this document expands the HTTP APIs to also offload the ability to provide records noto a third-party system. ## Detailed design -The API extensions are added to the [Delegated Content Routing HTTP API spec/`PUT`](../routing/DELEGATED_CONTENT_ROUTING_HTTP.md#put-routingv1providers) section, along with complimentary sections that outline known formats followed by example payload. +HTTP POST support is added +to the [Delegated Content Routing HTTP API](https://specs.ipfs.tech/routing/http-routing-v1/) +at `/routing/v1/providers` and `/routing/v1/peers`, along with complimentary +sections that outline known formats followed by example payload. ## Design rationale -The rationale for the design of `PUT` operations closely follows the reasoning listed in [IPIP-337](0337-delegated-routing-http-api.md#design-rationale). +The rationale for the design of `POST` operations closely follows the reasoning +listed in [IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337/). The design uses a human-readable request/response structure with extensibility in mind. The specification imposes no restrictions on the schema nor the protocol advertised in provider records. The hope is that such extensibility will encourage and inspire innovation for better transfer protocols. -In order to reduce barrier for adoption, the existing + +In order to reduce barrier for adoption, standard HTTP POST semantics are used, +along with transport-agnostic signature of DAG-CBOR payload representation +similar to the data field in IPNS. This ensures this IPIP does not introduce +any new signature dependency because IPNS records are already supported on +`/routing/v1` since [IPIP-379](https://specs.ipfs.tech/ipips/ipip-0379/). ### User benefit -Expanding the user benefits listed as part of [IPIP-337](0337-delegated-routing-http-api.md#user-benefit), in the context of content routing write operations are typically more expensive than read operations. They involve book keeping such as TTL, gossip propagation, etc. -Therefore, it is highly desirable to reduce the burden of advertising provider records onto the network by means of delegation through simple to use HTTP APIs. +Expanding the user benefits listed as part of +[IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337/#user-benefit), in the +context of content routing write operations are typically more expensive than +read operations. They involve book keeping such as TTL, gossip propagation, +etc. Therefore, it is highly desirable to reduce the burden of advertising +provider records onto the network by means of delegation through simple to use +HTTP APIs. ### Compatibility @@ -51,24 +70,43 @@ Therefore, it is highly desirable to reduce the burden of advertising provider r ##### DHT -The `PUT` APIs proposed here require a new data format for specifying provider records. -Since the records must include a valid signature, records published through HTTP delegated routing must be resigned. + +Records published through HTTP delegated routing will not benefit from implicit +PeerID identity and libp2p handshake, and must be explicitly signed. + +At the time of writing of this IPIP the Amino DHT does not have a concept of +explicit signature. That is why the `POST` APIs proposed here introduce a new +DAG-CBOR data format for specifying provider records, and mechanism for signing +them. + ##### Reframe -See [IPIP-337/Backwards Compatibility](0337-delegated-routing-http-api.md#backwards-compatibility). +This API was superseded by `/routing/v1`. See [IPIP-337/Backwards Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#backwards-compatibility). #### Forwards Compatibility -See [IPIP-337/Forwads Compatibility](0337-delegated-routing-http-api.md#forwards-compatibility). +[IPIP-337/Forwads Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#forwards-compatibility) still applies. + +The signature format introduced in this IPIP is not tied to HTTP semantics, and +similar to IPNS records, could be propagated by various means, including adding +support for delegated announcement to Amino DHT, where client sends +announcement to HTTP API, and then API backend takes care of sending it +to 20 peers. + ### Security -See [IPIP-337/Security](0337-delegated-routing-http-api.md#security). +This IPIP reuses semantics of IPNS Record signatures but with a different prefix, to avoid signature reuse attacks. + +ID, creation date and TTL facilitate signed mechanism for expiration. + +[IPIP-337/Security](https://specs.ipfs.tech/ipips/ipip-0337/#security) still applies. ### Alternatives - Reframe (general-purpose RPC) was evaluated, see "Design rationale" section for rationale why it was not selected. +- HTTP `PUT` was evaluated, see [rationale](https://github.com/ipfs/specs/pull/378#discussion_r1476257372) why `POST` was used instead. ### Copyright diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 69c532aef..cdfdc5edc 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -4,7 +4,7 @@ description: > Delegated routing is a mechanism for IPFS implementations to use for offloading content routing, peer routing and naming to another process/server. This specification describes an HTTP API for delegated routing of content, peers, and IPNS. -date: 2023-08-31 +date: 2024-02-05 maturity: reliable editors: - name: Gus Eggert @@ -93,9 +93,9 @@ The client SHOULD be able to make a request with `Accept: application/x-ndjson` Each object in the `Providers` list is a record conforming to a schema, usually the [Peer Schema](#peer-schema). -### `PUT /routing/v1/providers` +### `POST /routing/v1/providers` -#### `PUT` Request Body +#### `POST` Request Body ```json { @@ -120,15 +120,14 @@ There SHOULD be no more than 100 `Providers` per request. ::: -#### `PUT` Response codes +#### `POST` Response codes - `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) - `400` (Bad Request): the server deems the request to be invalid and cannot process it - `422` (Unprocessable Entity): request does not conform to schema or semantic constraints - `501` (Not Implemented): the server does not support providing records - -#### `PUT` Response Body +#### `POST` Response Body ```json { @@ -140,7 +139,7 @@ There SHOULD be no more than 100 `Providers` per request. - `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. - - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why PUT failed. + - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request ## Peer Routing API @@ -181,9 +180,9 @@ The client SHOULD be able to make a request with `Accept: application/x-ndjson` Each object in the `Peers` list is a record conforming to the [Peer Schema](#peer-schema). -### `PUT /routing/v1/peers` +### `POST /routing/v1/peers` -#### `PUT` Request Body +#### `POST` Request Body ```json { @@ -201,14 +200,14 @@ Each object in the `Providers` list is a *write provider record* entry. Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema) objects with `CID` list. -#### `PUT` Response codes +#### `POST` Response codes - `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) - `400` (Bad Request): the server deems the request to be invalid and cannot process it - `422` (Unprocessable Entity): request does not conform to schema or semantic constraints - `501` (Not Implemented): the server does not support providing records -#### `PUT` Response Body +#### `POST` Response Body ```json { @@ -220,13 +219,13 @@ objects with `CID` list. - `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. - - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why PUT failed. + - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request -#### `PUT` Response Status Codes +#### `POST` Response Status Codes - `200` (OK): processed - inspect response to see if there are any `Error` results. -- `400` (Bad Request): unable to process PUT request, make sure JSON schema and values are correct. +- `400` (Bad Request): unable to process POST request, make sure JSON schema and values are correct. ## IPNS API @@ -322,7 +321,7 @@ limits, allowing every site to query the API for results: ```plaintext Access-Control-Allow-Origin: * -Access-Control-Allow-Methods: GET, PUT, OPTIONS +Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS ``` ## Known Schemas @@ -376,7 +375,7 @@ the case, the field MUST be ignored. ### Announcement Schema -The `announcement` schema can be used in `PUT` operations to announce content providers or peer routing information. +The `announcement` schema can be used in `POST` operations to announce content providers or peer routing information. ```json { @@ -398,7 +397,7 @@ The `announcement` schema can be used in `PUT` operations to announce content pr - `Schema`: tells the server to interpret the JSON object as announce provider - `Payload`: is a DAG-JSON-compatible object with a subset of the below fields - `CID` is the CID being provided. - - This field is not presend when used for `PUT /routing/v1/peers` + - This field is not presend when used for `POST /routing/v1/peers` - `Scope` is an optional hint that provides semantic meaning about announced identifies: - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). @@ -413,7 +412,7 @@ The `announcement` schema can be used in `PUT` operations to announce content pr - `Signature` is a string with multibase-encoded binary signature that provides integrity and authenticity of the `Payload` field. - Signature is created by following below steps: 1. Convert `Payload` to deterministic, ordered [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/) map notation - 2. Prefix the DAG-JSON bytes with ASCII string `PUT /routing/v1 announcement:` + 2. Prefix the DAG-JSON bytes with ASCII string `POST /routing/v1 announcement:` 3. Sign the bytes with the private key of the Peer ID specified in the `Payload.ID`. - Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise. - Client SHOULD sign every announcement. @@ -429,7 +428,7 @@ TODO: what should be the limits? Max number of CIDs per `announcement` ? ::: -#### Use in PUT responses +#### Use in POST responses Server MAY return additional TTL information if the TTL is not provided in the request, or if server policy is to provide TTL different than the requested one. From fbefb1b598075e3c5621cb89e3a691a5e48c60f6 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 5 Feb 2024 23:49:46 +0100 Subject: [PATCH 11/16] chore: apply suggestions from code review Co-authored-by: Henrique Dias --- src/routing/http-routing-v1.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index cdfdc5edc..ece61570e 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -186,7 +186,7 @@ Each object in the `Peers` list is a record conforming to the [Peer Schema](#pee ```json { - "Providers": [ + "Peers": [ { "Schema": "announcement", ... @@ -402,7 +402,7 @@ The `announcement` schema can be used in `POST` operations to announce content p - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). - - `Timetamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). + - `Timestamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified in milliseconds. - If this value is unknown, the caller may skip this field, or use a value of 0. The server's default will be used. - `ID` is Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field. From f6f0d9b0d5fa086121fc5d6872d46316e1debb75 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 6 Feb 2024 00:01:48 +0100 Subject: [PATCH 12/16] ipip-378: editorial and 2MiB limit https://github.com/ipfs/specs/pull/378/files#r1420434379 --- src/ipips/ipip-0378.md | 6 ++---- src/routing/http-routing-v1.md | 20 +++++++------------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/src/ipips/ipip-0378.md b/src/ipips/ipip-0378.md index c5b587d0c..742750b4c 100644 --- a/src/ipips/ipip-0378.md +++ b/src/ipips/ipip-0378.md @@ -70,7 +70,6 @@ HTTP APIs. ##### DHT - Records published through HTTP delegated routing will not benefit from implicit PeerID identity and libp2p handshake, and must be explicitly signed. @@ -79,10 +78,10 @@ explicit signature. That is why the `POST` APIs proposed here introduce a new DAG-CBOR data format for specifying provider records, and mechanism for signing them. - ##### Reframe -This API was superseded by `/routing/v1`. See [IPIP-337/Backwards Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#backwards-compatibility). +The Reframe API was superseded by `/routing/v1`. +See [IPIP-337/Backwards Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#backwards-compatibility). #### Forwards Compatibility @@ -94,7 +93,6 @@ support for delegated announcement to Amino DHT, where client sends announcement to HTTP API, and then API backend takes care of sending it to 20 peers. - ### Security This IPIP reuses semantics of IPNS Record signatures but with a different prefix, to avoid signature reuse attacks. diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index ece61570e..bda97157f 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -114,9 +114,8 @@ Server SHOULD accept representing writes is [Announcement Schema](#announcement :::warn -TODO: is below a sensible limit? - -There SHOULD be no more than 100 `Providers` per request. +Since non-streaming results have to be buffered before sending, +server SHOULD be no more than 100 `Providers` per `application/json` response. ::: @@ -195,10 +194,9 @@ Each object in the `Peers` list is a record conforming to the [Peer Schema](#pee } ``` -Each object in the `Providers` list is a *write provider record* entry. +Each object in the `Peers` list is a *write peer record* entry. -Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema) -objects with `CID` list. +Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema). #### `POST` Response codes @@ -396,9 +394,9 @@ The `announcement` schema can be used in `POST` operations to announce content p - `Schema`: tells the server to interpret the JSON object as announce provider - `Payload`: is a DAG-JSON-compatible object with a subset of the below fields - - `CID` is the CID being provided. + - `CID` is the CID being provided (`/routing/v1/providers` only). - This field is not presend when used for `POST /routing/v1/peers` - - `Scope` is an optional hint that provides semantic meaning about announced identifies: + - `Scope` is an optional hint that provides semantic meaning about CID (`/routing/v1/providers` only): - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). @@ -422,11 +420,7 @@ The `announcement` schema can be used in `POST` operations to announce content p If support for big keys is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. - A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if the `Signature` check fails. -:::warn - -TODO: what should be the limits? Max number of CIDs per `announcement` ? - -::: +Server SHOULD return HTTP 400 Bad Request when announcement `Payload` serizalized to DAG-CBOR is bigger than 2MiB. #### Use in POST responses From da12e11f6525fcf6a312ebc5f02238f12ba9b6b1 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 6 Feb 2024 00:22:15 +0100 Subject: [PATCH 13/16] ipip-378: clarify string values https://github.com/ipfs/specs/pull/378#discussion_r1445901679 --- src/routing/http-routing-v1.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index bda97157f..d379310b3 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -119,7 +119,7 @@ server SHOULD be no more than 100 `Providers` per `application/json` response. ::: -#### `POST` Response codes +#### `POST` Response Status Codes - `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) - `400` (Bad Request): the server deems the request to be invalid and cannot process it @@ -198,7 +198,7 @@ Each object in the `Peers` list is a *write peer record* entry. Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema). -#### `POST` Response codes +#### `POST` Response Status Codes - `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records) - `400` (Bad Request): the server deems the request to be invalid and cannot process it @@ -393,20 +393,20 @@ The `announcement` schema can be used in `POST` operations to announce content p ``` - `Schema`: tells the server to interpret the JSON object as announce provider -- `Payload`: is a DAG-JSON-compatible object with a subset of the below fields - - `CID` is the CID being provided (`/routing/v1/providers` only). - - This field is not presend when used for `POST /routing/v1/peers` - - `Scope` is an optional hint that provides semantic meaning about CID (`/routing/v1/providers` only): +- `Payload`: is a map object with a subset of the below fields. + - `CID` is a string with multibase-encoded CID being provided (`/routing/v1/providers` only). + - This field is not present when used for `POST /routing/v1/peers` + - `Scope` (optional) is a string hint that provides semantic meaning about CID (`/routing/v1/providers` only): - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). - `Timestamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). - - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified in milliseconds. - - If this value is unknown, the caller may skip this field, or use a value of 0. The server's default will be used. - - `ID` is Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field. - - `Addrs` is an a list of string-encoded multiaddrs without `/p2p/peerID` suffix. - - `Protocols` is a list of protocols supported by `ID` and/or `Addrs`, if known upfront. - - `Metadata` is a string with multibase-encoded binary metadata that should be passed as-is + - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified as integer in milliseconds. + - If this value is unknown, the caller may skip this field or set it to 0. The server's default will be used. + - `ID` is a multibase-encoded Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field. + - `Addrs` (optional) is an a list of string-encoded multiaddrs without `/p2p/peerID` suffix. + - `Protocols` (optional) is a list of strings with protocols supported by `ID` and/or `Addrs`, if known upfront. + - `Metadata` (optional) is a string with multibase-encoded binary metadata that should be passed as-is - `Signature` is a string with multibase-encoded binary signature that provides integrity and authenticity of the `Payload` field. - Signature is created by following below steps: 1. Convert `Payload` to deterministic, ordered [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/) map notation @@ -418,9 +418,10 @@ The `announcement` schema can be used in `POST` operations to announce content p - ED25519 and other small public keys MUST be inlined inside of the `ID` field with the identity multihash type. - Key types that exceed 42 bytes (e.g. RSA) SHOULD NOT be inlined, the `ID` field should only include the multihash of the key. The key itself SHOULD be obtained out-of-band (e.g. by fetching the block via IPFS) and cached. If support for big keys is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. - - A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if the `Signature` check fails. -Server SHOULD return HTTP 400 Bad Request when announcement `Payload` serizalized to DAG-CBOR is bigger than 2MiB. +A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if either +- `Signature` is not valid +- `Payload` serialized to DAG-CBOR is bigger than 2MiB #### Use in POST responses From 05dceba203cbd7da79bea536ec3e6876a52b98b8 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 6 Feb 2024 01:01:08 +0100 Subject: [PATCH 14/16] ipip-378: switch signature to dag-cbor Rationale: https://github.com/ipfs/specs/pull/378#discussion_r1460390776 --- src/routing/http-routing-v1.md | 47 ++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index d379310b3..73f979dcb 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -136,7 +136,7 @@ server SHOULD be no more than 100 `Providers` per `application/json` response. } ``` -- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object +- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request @@ -209,13 +209,13 @@ Server SHOULD accept writes represented with [Announcement Schema](#announcement ```json { - "ProvideResults": [ + "PeersResults": [ { ... } ] } ``` -- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Protocol` of the corresponding write object +- `PeersResults` is a list of results in the same order as the `Peers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request @@ -392,7 +392,8 @@ The `announcement` schema can be used in `POST` operations to announce content p } ``` -- `Schema`: tells the server to interpret the JSON object as announce provider +#### Announcement Payload + - `Payload`: is a map object with a subset of the below fields. - `CID` is a string with multibase-encoded CID being provided (`/routing/v1/providers` only). - This field is not present when used for `POST /routing/v1/peers` @@ -400,28 +401,48 @@ The `announcement` schema can be used in `POST` operations to announce content p - `block` announces only the individual block (this is the implicit default if `Scope` field is not present). - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc). - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories). + - `Timestamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339). + - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified as integer in milliseconds. - If this value is unknown, the caller may skip this field or set it to 0. The server's default will be used. + - `ID` is a multibase-encoded Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field. + - ED25519 and other small public keys MUST be inlined inside of the `ID` field + with the identity multihash type. + - Key types that exceed 42 bytes (e.g. RSA) SHOULD NOT be inlined, the `ID` + field should only include the multihash of the key. The key itself SHOULD be + obtained out-of-band (e.g. by fetching the block via IPFS) and cached to + reduce the size of the signed `Payload`. + + If support for big keys is needed in + the future, this spec can be updated to allow the client to provide the key + and key type out-of-band by adding optional `PublicKey` fields, and if the + Peer ID is a CID, then the server can verify the public key's authenticity + against the CID, and then proceed with the rest of the verification scheme. + - `Addrs` (optional) is an a list of string-encoded multiaddrs without `/p2p/peerID` suffix. + - `Protocols` (optional) is a list of strings with protocols supported by `ID` and/or `Addrs`, if known upfront. + - `Metadata` (optional) is a string with multibase-encoded binary metadata that should be passed as-is + +#### Announcement Signature + - `Signature` is a string with multibase-encoded binary signature that provides integrity and authenticity of the `Payload` field. + - Signature is created by following below steps: - 1. Convert `Payload` to deterministic, ordered [DAG-JSON](https://ipld.io/specs/codecs/dag-json/spec/) map notation - 2. Prefix the DAG-JSON bytes with ASCII string `POST /routing/v1 announcement:` + 1. Convert `Payload` JSON to deterministic, ordered [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/) map notation + - Specification intention here is to use similar signature normalization as with DAG-CBOR `Data` field in IPNS Records, allowing for partial code and dependency reuse. + 2. Prefix the DAG-CBOR bytes with ASCII string `routing-record:` 3. Sign the bytes with the private key of the Peer ID specified in the `Payload.ID`. - Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise. + - Client SHOULD sign every announcement. - Servers SHOULD verify signature before accepting a record, unless running in a trusted environment. - - ED25519 and other small public keys MUST be inlined inside of the `ID` field with the identity multihash type. - - Key types that exceed 42 bytes (e.g. RSA) SHOULD NOT be inlined, the `ID` field should only include the multihash of the key. The key itself SHOULD be obtained out-of-band (e.g. by fetching the block via IPFS) and cached. - If support for big keys is needed in the future, this spec can be updated to allow the client to provide the key and key type out-of-band by adding optional `PublicKey` fields, and if the Peer ID is a CID, then the server can verify the public key's authenticity against the CID, and then proceed with the rest of the verification scheme. - -A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if either -- `Signature` is not valid -- `Payload` serialized to DAG-CBOR is bigger than 2MiB + - A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400) response code SHOULD be returned if (in order): + - `Payload` serialized to DAG-CBOR is bigger than 2MiB + - `Signature` is not valid #### Use in POST responses From ccbc0854a27a4bad9a42b06a1bd06eed1ce6995c Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 6 Feb 2024 01:10:25 +0100 Subject: [PATCH 15/16] ipip-378: add error schema https://github.com/ipfs/specs/pull/378#discussion_r1418773016 --- src/routing/http-routing-v1.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 73f979dcb..19f94db2f 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -463,6 +463,18 @@ or if server policy is to provide TTL different than the requested one. - If greater than the `TTL` in the request, then the server client SHOULD save resources and not repeat announcement until the announcement TTL expires and is forgotten by the routing system - If `0`, the server makes no claims about the lifetime of the record +### Error Schema + +The `error` schema SHOULD be used in POST and PUT responses to indicate errors related to specific announcement record. + +```json +{ + "Schema": "error", + "Error": "Invalid signature", + ... +} +``` + ### Legacy Schemas Legacy schemas include `ID` and optional `Addrs` list just like From a69d2b8fdfb58261c665e63408ca148f2dc0fd4c Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Tue, 20 Feb 2024 14:36:27 +0100 Subject: [PATCH 16/16] convert error schema into announcement response schema --- src/routing/http-routing-v1.md | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-) diff --git a/src/routing/http-routing-v1.md b/src/routing/http-routing-v1.md index 19f94db2f..465c64502 100644 --- a/src/routing/http-routing-v1.md +++ b/src/routing/http-routing-v1.md @@ -215,7 +215,7 @@ Server SHOULD accept writes represented with [Announcement Schema](#announcement } ``` -- `PeersResults` is a list of results in the same order as the `Peers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object +- `PeersResults` is a list of results in the same order as the `Peers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object: - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted. - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed. - The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request @@ -444,37 +444,25 @@ The `announcement` schema can be used in `POST` operations to announce content p - `Payload` serialized to DAG-CBOR is bigger than 2MiB - `Signature` is not valid -#### Use in POST responses +### Announcement Response Schema -Server MAY return additional TTL information if the TTL is not provided in the request, -or if server policy is to provide TTL different than the requested one. +The `announcement-response` schema can be used as `POST` responses when announcing content providers or peer routing information. This schema allows the server to return additional TTL information if the TTL is not provided in the request, or if the server policy is to provide TTL different than the requested one. ```json { - "Schema": "announcement", - "Payload": { - "TTL": 17280000 - } + "Schema": "announcement-response", + "Error": "error in case there was error", + "TTL": 17280000 } ``` +- `Error` is a string representing the error that might have happened when announcing. + - `TTL` in response is the time at which the server expects itself to drop the record - If less than the `TTL` in the request, then the client SHOULD repeat announcement earlier, before the announcement TTL expires and is forgotten by the routing system - If greater than the `TTL` in the request, then the server client SHOULD save resources and not repeat announcement until the announcement TTL expires and is forgotten by the routing system - If `0`, the server makes no claims about the lifetime of the record -### Error Schema - -The `error` schema SHOULD be used in POST and PUT responses to indicate errors related to specific announcement record. - -```json -{ - "Schema": "error", - "Error": "Invalid signature", - ... -} -``` - ### Legacy Schemas Legacy schemas include `ID` and optional `Addrs` list just like