From 319593e6b019ace45b52a698a6c7e9f6bf1d92f3 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Tue, 27 Apr 2021 16:45:40 +0300 Subject: [PATCH 01/12] Add Context and Alternative approaches --- .../adr-003-application-data-retrieval.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/lazy-adr/adr-003-application-data-retrieval.md diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md new file mode 100644 index 0000000000..2ae33f914b --- /dev/null +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -0,0 +1,105 @@ +# ADR 003: Retrieving Application messages + +## Changelog + +- 2021-04-25: initial draft + +## Context + +The academic Lazyledger [paper](https://arxiv.org/abs/1905.09274) describes the motivation and context for this API. +The main motivation can be quoted from section 3.3 of that paper: + +> **Application message retrieval partitioning.** Client nodes must be able to download all of the messages relevant to the applications they use [...], without needing to downloading any messages for other applications. + +> **Application message retrieval completeness.** When client nodes download messages relevant to the applications they use [...] +nodes, they must be able to verify that the messages they received are the complete set of messages relevant to their applications, for specific +blocks, and that there are no omitted messages. + + + +The main data structure that enables above properties is called a Namespaced Merkle Tree (NMT), an ordered binary Merkle tree where: +1. each node in the tree includes the range of namespaces of the messages in all descendants of each node +2. leaves in the tree are ordered by the namespace identifiers of the leaf messages + +A more formal description can be found the [specification](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#namespace-merkle-tree). +An implementation can be found in [this repository](https://github.com/lazyledger/nmt). + +This ADR basically describes version of the [`GetWithProof`](https://github.com/lazyledger/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L193-L196) of the NMT that leverages the fact that IPFS uses content addressing and that we have implemented an [IPLD plugin](https://github.com/lazyledger/lazyledger-core/tree/37502aac69d755c189df37642b87327772f4ac2a/p2p/ipld) for an NMT. + +**Note**: The APIs defined here will be particularly relevant for Optimistic Rollup (full) nodes that want to download their Rollup's data (see [lazyledger/optimint#48](https://github.com/lazyledger/optimint/issues/48)). + +## Alternative Approaches + +The approach described below will rely on IPFS' block exchange protocol (bitswap) and DHT; IPFS's implementation will be used as a black box to find peers that can serve the requested data. +This will likely be much slower than it potentially could be and for a first implementation we intentionally do not incorporate the optimizations that we could. + +We briefly mention potential optimizations for the future here: +- Use of [graphsync](https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/block-layer/graphsync/graphsync.md) instead of [bitswap](https://docs.ipfs.io/concepts/bitswap/) and use of [IPLD selectors](https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/design/history/exploration-reports/2018.10-selectors-design-goals.md) +- expose an API to be able to download application specific data by namespace (including proofs) with the minimal number of round-trips (e.g. finding nodes that expose an RPC endpoint like [`GetWithProof`](https://github.com/lazyledger/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L193-L196)) + +## Decision + +> This section records the decision that was made. +> It is best to record as much info as possible from the discussion that happened. This aids in not having to go back to the Pull Request to get the needed information. + +## Detailed Design + +> This section does not need to be filled in at the start of the ADR, but must be completed prior to the merging of the implementation. +> +> Here are some common questions that get answered as part of the detailed design: +> +> - What are the user requirements? +> +> - What systems will be affected? +> +> - What new data structures are needed, what data structures will be changed? +> +> - What new APIs will be needed, what APIs will be changed? +> +> - What are the efficiency considerations (time/space)? +> +> - What are the expected access patterns (load/throughput)? +> +> - Are there any logging, monitoring or observability needs? +> +> - Are there any security considerations? +> +> - Are there any privacy considerations? +> +> - How will the changes be tested? +> +> - If the change is large, how will the changes be broken up for ease of review? +> +> - Will these changes require a breaking (major) release? +> +> - Does this change require coordination with the LazyLedger fork of the SDK or lazyledger-app? + +## Status + +Proposed + +## Consequences + +> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. + +### Positive + +- easy to implement with the existing code (see [ADR 002](https://github.com/lazyledger/lazyledger-core/blob/47d6c965704e102ae877b2f4e10aeab782d9c648/docs/lazy-adr/adr-002-ipld-da-sampling.md#detailed-design)) +- resilient data retrieval via a p2p network +- dependence on a mature and well-tested code-base with a large and welcoming community +- + +### Negative + +- with IPFS, we inherit the fact that potentially a lot of round-trips are done until the data is fully downloaded +- in other words: this could end up way slower than potentially possible + +### Neutral + +- optimizations can happen incrementally once we have an initial working version + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +- IPLD selectors: https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/selectors/selectors.md From 9e6d782e4c046012674a5b188b9c672a5444ebf2 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Wed, 28 Apr 2021 01:10:28 +0300 Subject: [PATCH 02/12] Document decision too --- .../adr-003-application-data-retrieval.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 2ae33f914b..58a831752a 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -9,10 +9,9 @@ The academic Lazyledger [paper](https://arxiv.org/abs/1905.09274) describes the motivation and context for this API. The main motivation can be quoted from section 3.3 of that paper: -> **Application message retrieval partitioning.** Client nodes must be able to download all of the messages relevant to the applications they use [...], without needing to downloading any messages for other applications. +> (Property1) **Application message retrieval partitioning.** Client nodes must be able to download all of the messages relevant to the applications they use [...], without needing to downloading any messages for other applications. -> **Application message retrieval completeness.** When client nodes download messages relevant to the applications they use [...] -nodes, they must be able to verify that the messages they received are the complete set of messages relevant to their applications, for specific +> (Property2) **Application message retrieval completeness.** When client nodes download messages relevant to the applications they use [...], they must be able to verify that the messages they received are the complete set of messages relevant to their applications, for specific blocks, and that there are no omitted messages. @@ -39,11 +38,20 @@ We briefly mention potential optimizations for the future here: ## Decision -> This section records the decision that was made. -> It is best to record as much info as possible from the discussion that happened. This aids in not having to go back to the Pull Request to get the needed information. +Most discussions on this particular API happened either on calls or on other non-documented way. +We only describe the decision in this section. + +We decide to implement the simplest approach first. +We informally describe the protocol informally here first and explain why this fulfils Property1 and Property2 in the [Context](#context) section above. + +In the case that leaves with the requested namespace exist, this basically boils down to the following: traverse the tree starting from the root until finding first leaf (start) with the namespace in question, then directly request and download all leaves coming after the start until the namespace changes to a greater than the requested one again. +In the case that no leaves with the requested namespace exist in the tree, we traverse the tree to find the leaf in the position in the tree where the namespace would have been and download the neighbouring leaves. + +This is pretty much what the [`ProveNamespace`](https://github.com/lazyledger/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L132-L146) method does but using IPFS we can simply locate and then request the leaves, and the corresponding inner proof nodes will automatically be downloaded on the way, too. ## Detailed Design + > This section does not need to be filled in at the start of the ADR, but must be completed prior to the merging of the implementation. > > Here are some common questions that get answered as part of the detailed design: From ab1084a34b302ac0a79b1ad4ad9d1ff29d52a0d0 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Wed, 28 Apr 2021 13:04:54 +0300 Subject: [PATCH 03/12] Sketch API --- .../adr-003-application-data-retrieval.md | 92 ++++++++++++------- 1 file changed, 59 insertions(+), 33 deletions(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 58a831752a..ad268a8068 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -51,36 +51,60 @@ This is pretty much what the [`ProveNamespace`](https://github.com/lazyledger/nm ## Detailed Design - -> This section does not need to be filled in at the start of the ADR, but must be completed prior to the merging of the implementation. -> -> Here are some common questions that get answered as part of the detailed design: -> -> - What are the user requirements? -> -> - What systems will be affected? -> -> - What new data structures are needed, what data structures will be changed? -> -> - What new APIs will be needed, what APIs will be changed? -> -> - What are the efficiency considerations (time/space)? -> -> - What are the expected access patterns (load/throughput)? -> -> - Are there any logging, monitoring or observability needs? -> -> - Are there any security considerations? -> -> - Are there any privacy considerations? -> -> - How will the changes be tested? -> -> - If the change is large, how will the changes be broken up for ease of review? -> -> - Will these changes require a breaking (major) release? -> -> - Does this change require coordination with the LazyLedger fork of the SDK or lazyledger-app? +We define one function that returns all shares of a block belonging to a requested namespace and block (via the block's data availability header). + +```go +// RetrieveShares returns all raw data (raw shares) of the passed-in +// namespace ID and included in the block with the DataAvailabilityHeader dah. +func RetrieveShares( +ctx context.Context, +nID namespace.ID, // the Namespace ID we are requesting data for +dah *types.DataAvailabilityHeader, +api coreiface.CoreAPI, +) ([]byte, error) { + // 1. Find the row root(s) that contains the namespace ID nID + // 2. Traverse the corresponding tree(s) according to the + // above informally described algorithm and get the corresponding + // leaves (if any) + // 3. Return all (raw) shares corresponding to the nID +} + +``` + +Additionally, we define two functions that use the first one above to: +1. return all the parsed (non-padding) data with [reserved namespace IDs](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/consensus.md#reserved-namespace-ids): transactions, intermediate state roots, evidence +2. return all application specific blobs belonging to one namespace ID parsed as a slice of [Messages](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#message) + +The latter two methods might require moving or exporting a few currently unexported functions that (currently) live in [share_merging.go](https://github.com/lazyledger/lazyledger-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/share_merging.go#L57-L76) and could be implemented in a separate pull request. + +```go +// RetrieveStateRelevantMessages returns all state-relevant transactions +// (transactions, intermediate state roots, and evidence) included in a block +// with the DataAvailabilityHeader dah. +func RetrieveStateRelevantMessages( +ctx context.Context, +nID namespace.ID, +dah *types.DataAvailabilityHeader, +api coreiface.CoreAPI, +) (Txs, IntermediateStateRoots, Evidence, error) { + // like RetrieveShares but for all reserved namespaces + // additionally the shares are parsed (merged) into the + // corresponding types in the return arguments +} +``` + +```go +// RetrieveStateMessages returns all Messages of the passed-in +// namespace ID and included in the block with the DataAvailabilityHeader dah. +func RetrieveMessages( +ctx context.Context, +dah *types.DataAvailabilityHeader, +api coreiface.CoreAPI, +) (Messages, error) { + // like RetrieveShares but this additionally parsed the shares + // into the Messages type +} +``` ## Status @@ -88,7 +112,8 @@ Proposed ## Consequences -> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. +This API will most likely be used by Rollups too. +We should document it properly and move it together with relevant parts from ADR 002 into a separate go-package. ### Positive @@ -99,8 +124,8 @@ Proposed ### Negative -- with IPFS, we inherit the fact that potentially a lot of round-trips are done until the data is fully downloaded -- in other words: this could end up way slower than potentially possible +- with IPFS, we inherit the fact that potentially a lot of round-trips are done until the data is fully downloaded; in other words: this could end up way slower than potentially possible +- anyone interacting with that API needs to run an IPFS node ### Neutral @@ -110,4 +135,5 @@ Proposed > Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! +- [ADR 002](./adr-002-ipld-da-sampling.md) - IPLD selectors: https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/selectors/selectors.md From d2939b3be7c615d4a9682ec2016162ae7bf1fa03 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Wed, 28 Apr 2021 13:23:43 +0300 Subject: [PATCH 04/12] Cleanup and some clarifications --- .../adr-003-application-data-retrieval.md | 52 +++++++++---------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index ad268a8068..f3ee978827 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -52,28 +52,29 @@ This is pretty much what the [`ProveNamespace`](https://github.com/lazyledger/nm ## Detailed Design We define one function that returns all shares of a block belonging to a requested namespace and block (via the block's data availability header). +See [`ComputeShares`](https://github.com/lazyledger/lazyledger-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/block.go#L1371) for reference how encode the block data into namespace shares. ```go // RetrieveShares returns all raw data (raw shares) of the passed-in -// namespace ID and included in the block with the DataAvailabilityHeader dah. +// namespace ID nID and included in the block with the DataAvailabilityHeader dah. func RetrieveShares( -ctx context.Context, -nID namespace.ID, // the Namespace ID we are requesting data for -dah *types.DataAvailabilityHeader, -api coreiface.CoreAPI, + ctx context.Context, + nID namespace.ID, + dah *types.DataAvailabilityHeader, + api coreiface.CoreAPI, ) ([]byte, error) { - // 1. Find the row root(s) that contains the namespace ID nID - // 2. Traverse the corresponding tree(s) according to the - // above informally described algorithm and get the corresponding - // leaves (if any) - // 3. Return all (raw) shares corresponding to the nID + // 1. Find the row root(s) that contains the namespace ID nID + // 2. Traverse the corresponding tree(s) according to the + // above informally described algorithm and get the corresponding + // leaves (if any) + // 3. Return all (raw) shares corresponding to the nID } ``` Additionally, we define two functions that use the first one above to: -1. return all the parsed (non-padding) data with [reserved namespace IDs](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/consensus.md#reserved-namespace-ids): transactions, intermediate state roots, evidence -2. return all application specific blobs belonging to one namespace ID parsed as a slice of [Messages](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#message) +1. return all the parsed (non-padding) data with [reserved namespace IDs](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/consensus.md#reserved-namespace-ids): transactions, intermediate state roots, evidence. +2. return all application specific blobs (shares) belonging to one namespace ID parsed as a slice of Messages ([specification](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#message) and [code](https://github.com/lazyledger/lazyledger-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/block.go#L1336)). The latter two methods might require moving or exporting a few currently unexported functions that (currently) live in [share_merging.go](https://github.com/lazyledger/lazyledger-core/blob/1a08b430a8885654b6e020ac588b1080e999170c/types/share_merging.go#L57-L76) and could be implemented in a separate pull request. @@ -82,27 +83,27 @@ The latter two methods might require moving or exporting a few currently unexpor // (transactions, intermediate state roots, and evidence) included in a block // with the DataAvailabilityHeader dah. func RetrieveStateRelevantMessages( -ctx context.Context, -nID namespace.ID, -dah *types.DataAvailabilityHeader, -api coreiface.CoreAPI, + ctx context.Context, + nID namespace.ID, + dah *types.DataAvailabilityHeader, + api coreiface.CoreAPI, ) (Txs, IntermediateStateRoots, Evidence, error) { - // like RetrieveShares but for all reserved namespaces - // additionally the shares are parsed (merged) into the - // corresponding types in the return arguments + // like RetrieveShares but for all reserved namespaces + // additionally the shares are parsed (merged) into the + // corresponding types in the return arguments } ``` ```go -// RetrieveStateMessages returns all Messages of the passed-in +// RetrieveMessages returns all Messages of the passed-in // namespace ID and included in the block with the DataAvailabilityHeader dah. func RetrieveMessages( -ctx context.Context, -dah *types.DataAvailabilityHeader, -api coreiface.CoreAPI, + ctx context.Context, + dah *types.DataAvailabilityHeader, + api coreiface.CoreAPI, ) (Messages, error) { - // like RetrieveShares but this additionally parsed the shares - // into the Messages type + // like RetrieveShares but this additionally parsed the shares + // into the Messages type } ``` @@ -120,7 +121,6 @@ We should document it properly and move it together with relevant parts from ADR - easy to implement with the existing code (see [ADR 002](https://github.com/lazyledger/lazyledger-core/blob/47d6c965704e102ae877b2f4e10aeab782d9c648/docs/lazy-adr/adr-002-ipld-da-sampling.md#detailed-design)) - resilient data retrieval via a p2p network - dependence on a mature and well-tested code-base with a large and welcoming community -- ### Negative From d8b0b1efb40e263a9ea47536f6be0e9e5df08bf7 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Wed, 28 Apr 2021 15:13:47 +0300 Subject: [PATCH 05/12] more cleanup --- docs/lazy-adr/adr-003-application-data-retrieval.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index f3ee978827..9650e3c34d 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -133,7 +133,5 @@ We should document it properly and move it together with relevant parts from ADR ## References -> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! - - [ADR 002](./adr-002-ipld-da-sampling.md) - IPLD selectors: https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/selectors/selectors.md From 582dc9ff9fc0e2adad346cd5d1a6875c404efcd1 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Thu, 29 Apr 2021 13:15:50 +0300 Subject: [PATCH 06/12] minor clarifications --- docs/lazy-adr/adr-003-application-data-retrieval.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 9650e3c34d..4bd81fd27a 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -42,7 +42,7 @@ Most discussions on this particular API happened either on calls or on other non We only describe the decision in this section. We decide to implement the simplest approach first. -We informally describe the protocol informally here first and explain why this fulfils Property1 and Property2 in the [Context](#context) section above. +We first describe the protocol informally here and explain why this fulfils (Property1) and (Property2) in the [Context](#context) section above. In the case that leaves with the requested namespace exist, this basically boils down to the following: traverse the tree starting from the root until finding first leaf (start) with the namespace in question, then directly request and download all leaves coming after the start until the namespace changes to a greater than the requested one again. In the case that no leaves with the requested namespace exist in the tree, we traverse the tree to find the leaf in the position in the tree where the namespace would have been and download the neighbouring leaves. From 8cff58655954628565686a0586298495dccc7b9b Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 12:19:37 +0300 Subject: [PATCH 07/12] Update docs/lazy-adr/adr-003-application-data-retrieval.md Co-authored-by: Hlib Kanunnikov --- docs/lazy-adr/adr-003-application-data-retrieval.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 4bd81fd27a..b06bd61c03 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -18,7 +18,7 @@ blocks, and that there are no omitted messages. The main data structure that enables above properties is called a Namespaced Merkle Tree (NMT), an ordered binary Merkle tree where: 1. each node in the tree includes the range of namespaces of the messages in all descendants of each node -2. leaves in the tree are ordered by the namespace identifiers of the leaf messages +2. leaves in the tree are ordered by the namespace identifiers of the leaf messages A more formal description can be found the [specification](https://github.com/lazyledger/lazyledger-specs/blob/de5f4f74f56922e9fa735ef79d9e6e6492a2bad1/specs/data_structures.md#namespace-merkle-tree). An implementation can be found in [this repository](https://github.com/lazyledger/nmt). From 3879e4f0aab9801399a63eadabd36dd304bee0f2 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 12:45:11 +0300 Subject: [PATCH 08/12] Evidence->EvidenceData --- docs/lazy-adr/adr-003-application-data-retrieval.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index b06bd61c03..72ea9a62c3 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -87,7 +87,7 @@ func RetrieveStateRelevantMessages( nID namespace.ID, dah *types.DataAvailabilityHeader, api coreiface.CoreAPI, -) (Txs, IntermediateStateRoots, Evidence, error) { +) (Txs, IntermediateStateRoots, EvidenceData, error) { // like RetrieveShares but for all reserved namespaces // additionally the shares are parsed (merged) into the // corresponding types in the return arguments From 18f403a4feb2d8ae59ad5d227bafba07c216503b Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 12:45:23 +0300 Subject: [PATCH 09/12] return a slice of shares --- docs/lazy-adr/adr-003-application-data-retrieval.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 72ea9a62c3..813f9a37c7 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -62,7 +62,7 @@ func RetrieveShares( nID namespace.ID, dah *types.DataAvailabilityHeader, api coreiface.CoreAPI, -) ([]byte, error) { +) ([][]byte, error) { // 1. Find the row root(s) that contains the namespace ID nID // 2. Traverse the corresponding tree(s) according to the // above informally described algorithm and get the corresponding From cb6b4e10a8111d68ebfc52030c577bdc20752806 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 12:50:19 +0300 Subject: [PATCH 10/12] clarify relation to adr 002 and link to erasure coding in spec --- docs/lazy-adr/adr-003-application-data-retrieval.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 813f9a37c7..275b8ae2d2 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -6,6 +6,9 @@ ## Context +This ADR builds on top of [ADR 002](adr-002-ipld-da-sampling.md) and will use the implemented APIs described there. +The reader should familiarize themselves at least with the high-level concepts the as well as in the [specs](https://github.com/lazyledger/lazyledger-specs/blob/master/specs/data_structures.md#2d-reed-solomon-encoding-scheme). + The academic Lazyledger [paper](https://arxiv.org/abs/1905.09274) describes the motivation and context for this API. The main motivation can be quoted from section 3.3 of that paper: From 916b289ca35a85f4b752fba7ddf4f500e036f4ee Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 13:07:29 +0300 Subject: [PATCH 11/12] Add potential use-case: light-validator nodes --- docs/lazy-adr/adr-003-application-data-retrieval.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index 275b8ae2d2..b32017f6a9 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -29,6 +29,7 @@ An implementation can be found in [this repository](https://github.com/lazyledge This ADR basically describes version of the [`GetWithProof`](https://github.com/lazyledger/nmt/blob/ddcc72040149c115f83b2199eafabf3127ae12ac/nmt.go#L193-L196) of the NMT that leverages the fact that IPFS uses content addressing and that we have implemented an [IPLD plugin](https://github.com/lazyledger/lazyledger-core/tree/37502aac69d755c189df37642b87327772f4ac2a/p2p/ipld) for an NMT. **Note**: The APIs defined here will be particularly relevant for Optimistic Rollup (full) nodes that want to download their Rollup's data (see [lazyledger/optimint#48](https://github.com/lazyledger/optimint/issues/48)). +Another potential use-case of this API could be for so-called [light validator nodes](https://github.com/lazyledger/lazyledger-specs/blob/master/specs/node_types.md#node-type-definitions) that want to download and replay the state-relevant portion of the block data, i.e. transactions with [reserved namespace IDs](https://github.com/lazyledger/lazyledger-specs/blob/master/specs/consensus.md#reserved-namespace-ids). ## Alternative Approaches From 6e6d87b92cc185f796dc18b2c96d160d334c08c5 Mon Sep 17 00:00:00 2001 From: Ismail Khoffi Date: Sat, 1 May 2021 13:09:32 +0300 Subject: [PATCH 12/12] remove redundant refs --- docs/lazy-adr/adr-003-application-data-retrieval.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/lazy-adr/adr-003-application-data-retrieval.md b/docs/lazy-adr/adr-003-application-data-retrieval.md index b32017f6a9..abbe619088 100644 --- a/docs/lazy-adr/adr-003-application-data-retrieval.md +++ b/docs/lazy-adr/adr-003-application-data-retrieval.md @@ -137,5 +137,4 @@ We should document it properly and move it together with relevant parts from ADR ## References -- [ADR 002](./adr-002-ipld-da-sampling.md) -- IPLD selectors: https://github.com/ipld/specs/blob/5d3a3485c5fe2863d613cd9d6e18f96e5e568d16/selectors/selectors.md +We've linked to all references throughout the ADR.