From 4c2522709f4690755552f42156b17f4ee7dfadbe Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Mon, 4 Feb 2019 10:55:52 +0200 Subject: [PATCH 01/22] add syncing refactor syncing to new rfc as well as pruning and cut through --- RFC/src/Glossary.md | 61 +++++++---- RFC/src/RFC-0110_BaseNodes.md | 48 +-------- RFC/src/RFC-0130_Mining.md | 2 +- RFC/src/RFC-0140 Syncing and seeding.md | 131 ++++++++++++++++++++++++ 4 files changed, 175 insertions(+), 67 deletions(-) create mode 100644 RFC/src/RFC-0140 Syncing and seeding.md diff --git a/RFC/src/Glossary.md b/RFC/src/Glossary.md index a723fee344..16d1b31ef7 100644 --- a/RFC/src/Glossary.md +++ b/RFC/src/Glossary.md @@ -5,19 +5,19 @@ glossary to disambiguate ideas, and work towards a [ubiquitous language](https://blog.carbonfive.com/2016/10/04/ubiquitous-language-the-joy-of-naming/) for this project. ## Asset Issuer -[Asset Issuer]: #asset-issuer 'An entity that creates digital assets on the Tari DAN' +[Asset Issuer]: #asset-issuer "An entity that creates digital assets on the Tari DAN" An entity that creates digital assets on the Tari DAN. The Asset Issuer will specify the parameters of the contract template that defines the rules that govern the asset and the number and nature of its constituent tokens on issuance. The Asset Issuer will, generally, be the initial owner of the tokens. ## Bad Actor -[Bad Actor]: #bad-actor 'A participant that acts maliciously or negligently to the detriment of the network or another participant' +[Bad Actor]: #bad-actor "A participant that acts maliciously or negligently to the detriment of the network or another participant" A participant that acts maliciously or negligently to the detriment of the network or another participant. ## Base layer -[Base Layer]: #base-layer 'The Tari layer handling payments and secured by proof of work' +[Base Layer]: #base-layer "The Tari layer handling payments and secured by proof of work" The Tari Base layer is a merge-mined [blockchain] secured by proof-of-work. The base layer is primarily responsible for @@ -25,35 +25,42 @@ the emission of new Tari, for securing and managing [Tari coin] transfers. ## Base Node -[base node]: #base-node 'A full Tari node running on the base layer, validating and propagating Tari coin transactions and blocks' +[base node]: #base-node "A full Tari node running on the base layer, validating and propagating Tari coin transactions and blocks" A full Tari node running on the base layer. It's primary role is validating and propagating [Tari coin] transactions and blocks to the rest of the network. ## Block -[block]: #block 'A collection transactions and associated metadata recorded as a single entity in the Tari blockchain' +[block]: #block "A collection transactions and associated metadata recorded as a single entity in the Tari blockchain" A collection transactions and associated metadata recorded as a single entity in the Tari blockchain. The ordering of Tari transactions is set purely by the block height of the block they are recorded in. ## Block reward -[block reward]: #block-reward 'The amount of Tari created in every block' +[block reward]: #block-reward "The amount of Tari created in every block" The amount of Tari created by the coinbase transaction in every block. The block reward is set by the [emission schedule]. ## Blockchain -[blockchain]: #blockchain 'The linked sequence of Tari blocks on the Tari base layer' +[blockchain]: #blockchain "The linked sequence of Tari blocks on the Tari base layer" A sequence of tari [block]s. Each block contains a hash of the previous valid block. Thus the blocks form a chain with the property that changing anything in a block other than the head block requires rewriting the entire blockchain from that point on. +## Current head + +[currenthead]: #currenthead "The last valid block of the longest chain" + +The last [block] of the base layer that represents the latest valid block. This [block] must be from the longest proof of work chain to be the current head. + ## Checkpoint -[checkpoint]: #checkpoint 'A summary of the state of a Digital Asset that is recorded on the base layer' + +[checkpoint]: #checkpoint "A summary of the state of a Digital Asset that is recorded on the base layer" A hash of the state of a Digital Asset that is recorded on the base layer. @@ -69,17 +76,17 @@ The first transaction in every Tari block yields a [Block Reward] according to t awarded to the miner that performed the Proof of Work for the block. ## Committee -[committee]: #committee 'A group of validator nodes that are responsible for managing a specific Digital Asset' +[committee]: #committee "A group of validator nodes that are responsible for managing a specific Digital Asset" A group of [Validator Node]s that are responsible for managing the state of a specific [Digital Asset]. A committee is selected during asset issuance and can be updated at [Checkpoint]s. ## CommitteeSelectionStrategy -[CommitteeSelectionStrategy]: #committeeselectionstrategy 'A strategy for the DAN to select candidates for the committee from the available registered Validator Nodes' +[CommitteeSelectionStrategy]: #committeeselectionstrategy "A strategy for the DAN to select candidates for the committee from the available registered Validator Nodes" A strategy for the DAN to algorithmically select candidates for the committee from the available registered Validator Nodes. The VNs will need accept the nomination to become part of the committee. ## ConsensusStrategy -[ConsensusStrategy]: #consensusstrategy 'The approach that will be taken for a committee to reach consensus on instructions' +[ConsensusStrategy]: #consensusstrategy "The approach that will be taken for a committee to reach consensus on instructions" The approach that will be taken for a committee to reach consensus on the validity of instructions that are performed on a given Digital Asset. @@ -102,13 +109,13 @@ created by [asset issuer]s on the Tari 2nd layer. For example, a promoter might ## Digital Asset Network -[Digital Asset Network]: #digital-asset-network 'The Tari second layer. All digital asset interactions are managed here.' +[Digital Asset Network]: #digital-asset-network "The Tari second layer. All digital asset interactions are managed here." The Tari second layer. All digital asset interactions are managed on the Tari Digital Assets Network (DAN). These interactions (defined in [instruction]s) are processed and validated by [Validator Node]s. ## DigitalAssetTemplate -[DigitalAssetTemplate]: #digitalassettemplate 'A set of non-turing complete contract types supported by the DAN' +[DigitalAssetTemplate]: #digitalassettemplate "A set of non-turing complete contract types supported by the DAN" A DigitalAssetTemplate is one of a set of contract types supported by the DAN. These contracts are non-turing complete and consist of rigid rule-sets with parameters that can be set by Asset Issuers. @@ -123,7 +130,7 @@ asset. Depending on the DA created, tokens can represent tickets, in-game items, ## Instructions -[instruction]: #instructions 'Second-layer network commands for managing digital asset state' +[instruction]: #instructions "Second-layer network commands for managing digital asset state" Instructions are the [digital asset network] equivalent of [transaction]s. Instructions are issued by asset issuers and client applications and are relayed by the DAN to the [validator node]s that are managing the associated @@ -145,7 +152,7 @@ longest proof-of-work chain. Miners usually draw from the mempool to build up tr ## Mimblewimble -[mimblewimble]: #mimblewimble 'a privacy-centric cryptocurrency protocol' +[mimblewimble]: #mimblewimble "a privacy-centric cryptocurrency protocol" Mimblewimble is a privacy-centric cryptocurrency protocol. It was [dropped](https://download.wpsoftware.net/bitcoin/wizardry/mimblewimble.txt) in the Bitcoin Developers chatroom by an @@ -168,14 +175,14 @@ A mathematical demonstration that a value inside a [commitment] (i.e. it is hidd [Mimblewimble], range proofs are used to prove that outputs are positive values. ## RegistrationCollateral -[RegistrationCollateral]: #registrationcollateral 'An amount of tari coin that is locked up on the base layer when a [Validator Node] is registered' +[RegistrationCollateral]: #registrationcollateral "An amount of tari coin that is locked up on the base layer when a [Validator Node] is registered" An amount of tari coin that is locked up on the base layer when a [Validator Node] is registered. In order to make Sybil attacks expensive and to provide an authorative base layer registry of [validator node]s they will need to lock up a amount of [Tari Coin] on the [Base Layer] using a registration transaction to begin acting as a VN on the DAN. ## RegistrationTerm -[RegistrationTerm]: #registrationterm 'The minimum amount of time that a VN registration lasts' +[RegistrationTerm]: #registrationterm "The minimum amount of time that a VN registration lasts" The minimum amount of time that a VN registration lasts, the RegistrationCollateral can only be released after this minimum period has elapsed. @@ -195,7 +202,7 @@ The current synchronisation state of a [Base Node]. This can either be transactions. ## Transaction -[transaction]: #transaction 'Base layer tari coin transfers.' +[transaction]: #transaction "Base layer tari coin transfers." Transactions are activities recorded on the Tari [blockchain] running on the [base layer]. Transactions always involve a transfer of [Tari coin]s. @@ -210,13 +217,13 @@ Transactions or blocks are `unvalidated` when first received by a [Base Node]. A `Validated` blocks are added to the [blockchain] and propagated to peers. ## Tari Coin -[tari coin]: #tari-coin 'The base layer token' +[tari coin]: #tari-coin "The base layer token" The base layer token. Tari coins are released according to the [emission schedule] on the Tari [base layer] [blockchain] in [coinbase transaction]s. ## Trusted Node -[trusted node]: #trusted-node 'A permissioned Validator Node nominated by an Asset Issuer' +[trusted node]: #trusted-node "A permissioned Validator Node nominated by an Asset Issuer" A permissioned Validator Node nominated by an Asset Issuer that will form part of the committee for that Digital Asset. @@ -232,7 +239,7 @@ UTXO values are hidden by their [commitment]s. Only the owner of the UTXO and (p ## Validator Node -[validator node]: #validator-node 'A second-layer node that manages and validates digital asset state transitions' +[validator node]: #validator-node "A second-layer node that manages and validates digital asset state transitions" Validator nodes (VNs) make up the Tari second layer, or [Digital Asset Network]. VNs are responsible for creating and updating [digital asset]s living on the Tari network. @@ -242,6 +249,18 @@ updating [digital asset]s living on the Tari network. A Tari Wallet is responsible for managing key pairs, and for constructing and negotiating [transaction]s for transferring and receiving [tari coin]s on the [Base Layer]. +## Pruning horizon + +[pruninghorizon]: #pruninghorizon "Block height at which pruning must commence" + +This is a local setting for each node to help reduce required syncing. This is the age in block height after which a block may be pruned. + +## Archive node + +[archivenode]: #archivenode "a full history node" + +This is a full history [base node]. It will keep a complete history of every transaction ever received and it will not implement pruning. + # Disclaimer This document is subject to the [disclaimer](../DISCLAIMER.md). \ No newline at end of file diff --git a/RFC/src/RFC-0110_BaseNodes.md b/RFC/src/RFC-0110_BaseNodes.md index 7deda5563f..d37dc118ce 100644 --- a/RFC/src/RFC-0110_BaseNodes.md +++ b/RFC/src/RFC-0110_BaseNodes.md @@ -164,53 +164,11 @@ In addition, when a block has been validated and added to the blockchain: ### Seeding nodes and Synchronising the chain -When base nodes start up, they need to synchronize the blockchain with their peers. - -Base Nodes that have just started up MUST perform the following in order to synchronize their blockchain state with the -network: - -1. The Base Node's [SynchronisationState] is set to `Synchronising`. -1. Load a bootstrap list of peers from a configuration file, or a cached list, if this is not the first time that the - node has started. -1. For each peer in this list: - 1. Establish a connection with the peer. - 1. Request a peer list from that peer. - 1. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. - -The Base Node will now be able to build a strategy for catching up to the network. The Base Node will implement its -[SynchronisationStrategy], which reduces load on any single peer and optimises bandwidth usage to synchronise the -blockchain as quickly as possible. - -In particular, Mimblewimble has some unique properties that could lead to very fast synchronisation strategies. For -example, because of cut-through and pruning, the entire blockchain state can be represented by the current [UTXO] set -and all the coinbase transaction inputs. - -The upshot of this is that a new node can be perfectly sure of the current blockchain state and not download any block -history at all. All that is required is downloading the block _header_ history and the current UTXO set. Then -verification is achieved by - -1. The UTXO set and knowledge of the emission rate are used to verify the coin supply. -1. The transaction kernel history (present in the block headers) and the UTXO range proofs are used to verify that every - UTXO is legitimate. -1. The proof of work can be verified from the block headers. Furthermore, if a commitment (e.g. a Merkle tree root) for - the UTXO set is stored in the block headers, it is straightforward to verify that the UTXO set corresponds to a block - in the chain. - -When Base Nodes receive blocks from peers while synchronizing, the usual -[block validation](#block-validation-and-propagation) process is followed. +Syncing is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) ### Pruning and cut-through -[Pruning and cut-through]: #Pruning-and-cut-through "Remove already spent outputs from the [utxo]" - -In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. - -Pruning is only for the benefit of the local base node as it reduces the local blockchain size. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. Pruned nodes will not be able to safely handle a sufficiently deep re-org. Only Archival nodes can roll back arbitrarily to handle large re-orgs, and so the network cannot run on pruned nodes alone. - -When running in pruning mode, [base node]s have the following responsibilities: - -1. MUST remove all spent outputs in it's current stored [UTXO](utxo) when a new block is received from another [base node]. - +pruning and cut-through is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) [tari coin]: Glossary.md#tari-coin @@ -227,4 +185,4 @@ When running in pruning mode, [base node]s have the following responsibilities: [SynchronisationStrategy]: Glossary.md#synchronisationstrategy [SynchronisationState]: Glossary.md#synchronisationstate [mining server]: Glossary.md#mining-server -[cut-through]: RFC-0110_BaseNodes.md#Pruning-and-cut-through +[cut-through]: RFC-0140_Syncing.md#Pruning-and-cut-through diff --git a/RFC/src/RFC-0130_Mining.md b/RFC/src/RFC-0130_Mining.md index ec21a3e639..7248b9a4d5 100644 --- a/RFC/src/RFC-0130_Mining.md +++ b/RFC/src/RFC-0130_Mining.md @@ -151,4 +151,4 @@ and Auxiliary blockchain is provided in the [Merged Mining TLU report] (https:// [block]: Glossary.md#block [mempool]: Glossary.md#mempool -[cut-through]: RFC-0110_BaseNodes.md#Pruning-and-cut-through +[cut-through]: RFC-0140_Syncing.md#Pruning-and-cut-through diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md new file mode 100644 index 0000000000..8805429f0a --- /dev/null +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -0,0 +1,131 @@ +# RFC-0140/Sync and Seeding + +## Syncing strategies and objectives + +![status: draft](theme/images/status-draft.svg) + +**Maintainer(s)**: [SW van heerden](https://github.com/SWvheerden) + +# License + +[ The 3-Clause BSD License](https://opensource.org/licenses/BSD-3-Clause). + +Copyright 2018 The Tari Development Community + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the +following conditions are met: + +1. Redistributions of this document must retain the above copyright notice, this list of conditions and the following + disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided with the distribution. +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, +INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +## Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", +"NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in +[BCP 14](https://tools.ietf.org/html/bcp14) (covering RFC2119 and RFC8174) when, and only when, they appear in all capitals, as +shown here. + +## Disclaimer + +The purpose of this document and its content is for information purposes only and may be subject to change or update +without notice. + +This document may include preliminary concepts that may or may not be in the process of being developed by the Tari +community. The release of this document is intended solely for review and discussion by the community regarding the +technological merits of the potential system outlined herein. + +## Goals + +This document describes the process of Syncing, seeding, pruning and cut-through. + +## Related RFCs + +* [RFC-0110: Base Nodes](RFC-0110_BaseNodes.md) + +## Descriptions + +### Syncing + +When a new node comes online, looses connection or encounters a fork that is longer than is longer than it can solve, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Sycning can be diveded into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [puning horizon](pruninghorizon) to current head, as well as every block header from genesis block. + +#### Complete Sync + +Complete sync is only available from archive nodes, as these will be the only nodes that will be able to supply the complet history required to sync every block with every transaction from genesis block up onto [current head](currenthead). + + + +#### Syncing process + +The syncing process is done in the following steps: + +1. Set its [SynchronisationState] to `Synchronising`. +2. Load a bootstrap list of peers from a configuration file, or a cached list. +3. For each peer in this list: + 1. Establish a connection with the peer. + 2. Request a peer list from that peer. + 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. +4. Choose the longest chain based on pow. +5. Download all headers from genesis block up onto [current head](currenthead), an validate the headers as you receive them. +6. Download all blocks from [puning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. +7. Validate blocks as if they where just mined and then received, in order of ascending age. + +After this process the node will be in sync and able to process blocks and transaction normally. + +#### Keeping in sync + +The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should enclude the current total pow, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will replay with a pong message also including the total pow, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest pow has the responsiblity to ask the peer for syncing information. + +If the genesis block hash's dont match the node is removed from its peer list as this node is running a different block chain, we can remove it. + +This will be hanled by the node asking for each block from the [current head](currenthead) going backword untill a known block is found. If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). + +#### Chain forks + +Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [utxo]. Only keepinng the current [utxo] will not be possible, because if you encounter a fork where there are two running version op the blockchain, you will not be able to swop without doing sync again. + +To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. + +### Pruning and cut-through + +[Pruning and cut-through]: #Pruning-and-cut-through "Remove already spent outputs from the [utxo]" + +In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. + +Pruning is only for the benefit of the local base node as it reduces the local blockchain size. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. Pruned nodes will not be able to safely handle a sufficiently deep re-org. Only Archival nodes can roll back arbitrarily to handle large re-orgs, and so the network cannot run on pruned nodes alone. + +When running in pruning mode, [base node]s have the following responsibilities: + +1. MUST remove all spent outputs in it's current stored [UTXO](utxo) when a new block is received from another [base node]. + + + +[archivenode]: Glossary.md#archivenode +[pruninghorizon]: Glossary.md#pruninghorizon +[tari coin]: Glossary.md#tari-coin +[blockchain]: Glossary.md#blockchain +[currenthead]: Glossary.md#currenthead +[block]: Glossary.md#block +[transaction]: Glossary.md#transaction +[base node]: Glossary.md#base-node +[utxo]: Glossary.md#unspent-transaction-outputs +[mimblewimble]: Glossary.md#mimblewimble +[mempool]: Glossary.md#mempool +[ValidationState]: Glossary.md#validationstate +[BroadcastStrategy]: Glossary.md#broadcaststrategy +[range proof]: Glossary.md#range-proof +[SynchronisationStrategy]: Glossary.md#synchronisationstrategy +[SynchronisationState]: Glossary.md#synchronisationstate +[mining server]: Glossary.md#mining-server +[cut-through]: RFC-0110_BaseNodes.md#Pruning-and-cut-through From f840eb582f3ee76aa15d5973a5503a60ca2d0263 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Mon, 4 Feb 2019 10:57:35 +0200 Subject: [PATCH 02/22] add mention of pruninghorizon in pruning --- RFC/src/RFC-0140 Syncing and seeding.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 8805429f0a..471d232f83 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -103,11 +103,11 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. -Pruning is only for the benefit of the local base node as it reduces the local blockchain size. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. Pruned nodes will not be able to safely handle a sufficiently deep re-org. Only Archival nodes can roll back arbitrarily to handle large re-orgs, and so the network cannot run on pruned nodes alone. +Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens afterthe [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. Pruned nodes will not be able to safely handle a sufficiently deep re-org. Only Archival nodes can roll back arbitrarily to handle large re-orgs, and so the network cannot run on pruned nodes alone. When running in pruning mode, [base node]s have the following responsibilities: -1. MUST remove all spent outputs in it's current stored [UTXO](utxo) when a new block is received from another [base node]. +1. MUST remove all spent outputs thats older than the [pruning horizon](pruninghorizon) in it's current stored [UTXO](utxo) when a new block is received from another [base node]. From e5f8201cf12a07df7d48878b48d14ae5ff440423 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Mon, 4 Feb 2019 11:27:01 +0200 Subject: [PATCH 03/22] typo --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 471d232f83..b49f860987 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -78,7 +78,7 @@ The syncing process is done in the following steps: 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. 4. Choose the longest chain based on pow. 5. Download all headers from genesis block up onto [current head](currenthead), an validate the headers as you receive them. -6. Download all blocks from [puning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. +6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. 7. Validate blocks as if they where just mined and then received, in order of ascending age. After this process the node will be in sync and able to process blocks and transaction normally. From f2e96afaa6861cc41d79834fd508f100a552409a Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Mon, 4 Feb 2019 14:05:32 +0200 Subject: [PATCH 04/22] add mention of archive nodes --- RFC/src/RFC-0110_BaseNodes.md | 7 +++++++ RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/RFC/src/RFC-0110_BaseNodes.md b/RFC/src/RFC-0110_BaseNodes.md index d37dc118ce..30e133ff1c 100644 --- a/RFC/src/RFC-0110_BaseNodes.md +++ b/RFC/src/RFC-0110_BaseNodes.md @@ -170,6 +170,13 @@ Syncing is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) pruning and cut-through is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) +### Archival nodes + +[Archival nodes](archivenode) are used to keep a complete history of the blockchain since genesis block, they do not employ pruning at all. These nodes will allow full syncing of the blockchain because normal nodes will not keep the full history to enable this. + + + +[archivenode]: Glossary.md#archivenode [tari coin]: Glossary.md#tari-coin [blockchain]: Glossary.md#blockchain diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index b49f860987..5bc3726020 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -103,7 +103,7 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. -Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens afterthe [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. Pruned nodes will not be able to safely handle a sufficiently deep re-org. Only Archival nodes can roll back arbitrarily to handle large re-orgs, and so the network cannot run on pruned nodes alone. +Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens afterthe [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. When running in pruning mode, [base node]s have the following responsibilities: From c3bcabfcfcadad9e2c2197b19b16b28d02f7e794 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 08:11:32 +0200 Subject: [PATCH 05/22] removed duplicate text --- RFC/src/RFC-0110_BaseNodes.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/RFC/src/RFC-0110_BaseNodes.md b/RFC/src/RFC-0110_BaseNodes.md index 30e133ff1c..5ca68ef645 100644 --- a/RFC/src/RFC-0110_BaseNodes.md +++ b/RFC/src/RFC-0110_BaseNodes.md @@ -162,13 +162,9 @@ In addition, when a block has been validated and added to the blockchain: * The mempool MUST also remove all transactions that are present in the newly validated block. * The UTXO set MUST be updated; removing all inputs in the block, and adding all the new outputs in it. -### Seeding nodes and Synchronising the chain +### Synchronising and pruning of the chain -Syncing is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) - -### Pruning and cut-through - -pruning and cut-through is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) +Syncing, pruning and cut-through is discussed in detail in [RFC-0140](RFC-0140_Syncing.md) ### Archival nodes From ac90f4e16c4f4077cff5eca69b12cbb2f1e08bf1 Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:21:52 +0200 Subject: [PATCH 06/22] Update RFC/src/Glossary.md Co-Authored-By: SWvheerden --- RFC/src/Glossary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RFC/src/Glossary.md b/RFC/src/Glossary.md index 16d1b31ef7..47ce4b81dd 100644 --- a/RFC/src/Glossary.md +++ b/RFC/src/Glossary.md @@ -253,7 +253,7 @@ A Tari Wallet is responsible for managing key pairs, and for constructing and ne [pruninghorizon]: #pruninghorizon "Block height at which pruning must commence" -This is a local setting for each node to help reduce required syncing. This is the age in block height after which a block may be pruned. +This is a local setting for each node to help reduce syncing time and bandwidth. This is the number of blocks from the chain tip beyond which a chain will be pruned. ## Archive node @@ -263,4 +263,4 @@ This is a full history [base node]. It will keep a complete history of every tra # Disclaimer -This document is subject to the [disclaimer](../DISCLAIMER.md). \ No newline at end of file +This document is subject to the [disclaimer](../DISCLAIMER.md). From 9d9b012bbc7dfb1931e1f761e672679f97af17b1 Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:22:11 +0200 Subject: [PATCH 07/22] Update RFC/src/RFC-0140 Syncing and seeding.md Co-Authored-By: SWvheerden --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 5bc3726020..af0d405e54 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -1,4 +1,4 @@ -# RFC-0140/Sync and Seeding +# RFC-0140/SyncAndSeeding ## Syncing strategies and objectives From c4be87f5a77dbb0e5fd2004c6623abc2bf50b6dc Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:22:53 +0200 Subject: [PATCH 08/22] Update RFC/src/RFC-0140 Syncing and seeding.md Co-Authored-By: SWvheerden --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index af0d405e54..e95afbb628 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -58,7 +58,7 @@ This document describes the process of Syncing, seeding, pruning and cut-through ### Syncing -When a new node comes online, looses connection or encounters a fork that is longer than is longer than it can solve, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Sycning can be diveded into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [puning horizon](pruninghorizon) to current head, as well as every block header from genesis block. +When a new node comes online, loses connection or encounters a chain re-organisation that is longer than it can tolerate, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Syncing can be divided into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [pruning horizon](pruninghorizon) to current head, as well as every block header from genesis block. #### Complete Sync From d3211dea5143eb39bb2ba769ad6fc6b5d16a582f Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:23:49 +0200 Subject: [PATCH 09/22] Update RFC/src/RFC-0140 Syncing and seeding.md Co-Authored-By: SWvheerden --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index e95afbb628..537d6d746b 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -77,7 +77,7 @@ The syncing process is done in the following steps: 2. Request a peer list from that peer. 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. 4. Choose the longest chain based on pow. -5. Download all headers from genesis block up onto [current head](currenthead), an validate the headers as you receive them. +5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. 6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. 7. Validate blocks as if they where just mined and then received, in order of ascending age. From ed6b7b71302e73fad208ef7e910a364dc3b289d7 Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:25:16 +0200 Subject: [PATCH 10/22] Update RFC/src/RFC-0140 Syncing and seeding.md Co-Authored-By: SWvheerden --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 537d6d746b..91f9fb75fd 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -79,7 +79,7 @@ The syncing process is done in the following steps: 4. Choose the longest chain based on pow. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. 6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. -7. Validate blocks as if they where just mined and then received, in order of ascending age. +7. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. From 5328cce58e1970950ce4546dac989dc1ac837dfb Mon Sep 17 00:00:00 2001 From: Cayle Sharrock Date: Tue, 5 Feb 2019 08:26:56 +0200 Subject: [PATCH 11/22] Update RFC/src/RFC-0140 Syncing and seeding.md Co-Authored-By: SWvheerden --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 91f9fb75fd..2f12ffac68 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -87,7 +87,7 @@ After this process the node will be in sync and able to process blocks and trans The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should enclude the current total pow, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will replay with a pong message also including the total pow, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest pow has the responsiblity to ask the peer for syncing information. -If the genesis block hash's dont match the node is removed from its peer list as this node is running a different block chain, we can remove it. +If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different block chain. This will be hanled by the node asking for each block from the [current head](currenthead) going backword untill a known block is found. If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). From 10ee1c2305988c5948f47ed9b14cb51dbbc689e1 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 10:40:04 +0200 Subject: [PATCH 12/22] grammer changes --- RFC/src/RFC-0140 Syncing and seeding.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 2f12ffac68..b8d160ed87 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -58,11 +58,11 @@ This document describes the process of Syncing, seeding, pruning and cut-through ### Syncing -When a new node comes online, loses connection or encounters a chain re-organisation that is longer than it can tolerate, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Syncing can be divided into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [pruning horizon](pruninghorizon) to current head, as well as every block header from genesis block. +When a new node comes online, loses connection or encounters a chain re-organisation that is longer than it can tolerate, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Syncing can be divided into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [pruning horizon](pruninghorizon) to [current head](current head), as well as every block header from genesis block. #### Complete Sync -Complete sync is only available from archive nodes, as these will be the only nodes that will be able to supply the complet history required to sync every block with every transaction from genesis block up onto [current head](currenthead). +Complete sync is only available from archive nodes, as these will be the only nodes that will be able to supply the complete history required to sync every block with every transaction from genesis block up onto [current head](currenthead). @@ -78,22 +78,22 @@ The syncing process is done in the following steps: 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. 4. Choose the longest chain based on pow. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. -6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the pruninghorizon will just be infinite, which means you will download all blocks ever created. +6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. 7. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. #### Keeping in sync -The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should enclude the current total pow, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will replay with a pong message also including the total pow, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest pow has the responsiblity to ask the peer for syncing information. +The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total pow, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will relay with a pong message also including the total pow, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest pow has the responsiblity to ask the peer for syncing information. -If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different block chain. +If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. -This will be hanled by the node asking for each block from the [current head](currenthead) going backword untill a known block is found. If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). +This will be handled by the node asking for each block from the [current head](currenthead) going backward for older blocks until a known block is found. If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). #### Chain forks -Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [utxo]. Only keepinng the current [utxo] will not be possible, because if you encounter a fork where there are two running version op the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [utxo]. Only keeping the current [utxo] will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. @@ -103,7 +103,7 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. -Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens afterthe [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. +Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. When running in pruning mode, [base node]s have the following responsibilities: From d058365dda1f405f4769d8d6bcca8f165a4055bf Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 10:53:33 +0200 Subject: [PATCH 13/22] review comments --- RFC/src/RFC-0140 Syncing and seeding.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index b8d160ed87..440c47ceda 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -76,7 +76,7 @@ The syncing process is done in the following steps: 1. Establish a connection with the peer. 2. Request a peer list from that peer. 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. -4. Choose the longest chain based on pow. +4. Choose the longest chain based on POW. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. 6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. 7. Validate blocks as if they where just mined and then received, in chronological order. @@ -85,7 +85,7 @@ After this process the node will be in sync and able to process blocks and trans #### Keeping in sync -The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total pow, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will relay with a pong message also including the total pow, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest pow has the responsiblity to ask the peer for syncing information. +The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total POW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will relay with a pong message also including the total POW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest POW has the responsiblity to ask the peer for syncing information. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. @@ -93,7 +93,7 @@ This will be handled by the node asking for each block from the [current head](c #### Chain forks -Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [utxo]. Only keeping the current [utxo] will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [UTXO](utxo). Only keeping the current [UTXO](utxo) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. From 599eefc9f0339104d3135137b1188b81ae9980dd Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 10:55:36 +0200 Subject: [PATCH 14/22] added must for BCP 14 --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 440c47ceda..814a3b7a5c 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -68,7 +68,7 @@ Complete sync is only available from archive nodes, as these will be the only no #### Syncing process -The syncing process is done in the following steps: +The syncing process MUST be done in the following steps: 1. Set its [SynchronisationState] to `Synchronising`. 2. Load a bootstrap list of peers from a configuration file, or a cached list. From 7d41e291afd421a5bdc43bca34d88679181e29e6 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 10:56:43 +0200 Subject: [PATCH 15/22] adde must and should for sync --- RFC/src/RFC-0140 Syncing and seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 814a3b7a5c..9505ab6bb6 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -85,7 +85,7 @@ After this process the node will be in sync and able to process blocks and trans #### Keeping in sync -The node should periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total POW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node will relay with a pong message also including the total POW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest POW has the responsiblity to ask the peer for syncing information. +The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total POW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST relay with a pong message also including the total POW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest POW has the responsiblity to ask the peer for syncing information. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. From e350d764b0f3e227d9a0ee107582a881dd21d1b1 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Tue, 5 Feb 2019 11:31:15 +0200 Subject: [PATCH 16/22] Add extra clarification --- RFC/src/RFC-0140 Syncing and seeding.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 9505ab6bb6..684391b9b8 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -78,8 +78,9 @@ The syncing process MUST be done in the following steps: 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. 4. Choose the longest chain based on POW. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. -6. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. -7. Validate blocks as if they where just mined and then received, in chronological order. +6. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). +7. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. +8. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. From 79a4c99515815b1765d7ba99262a13b36317e994 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Wed, 6 Feb 2019 08:10:37 +0200 Subject: [PATCH 17/22] address code review --- RFC/src/Glossary.md | 4 +++- RFC/src/RFC-0140 Syncing and seeding.md | 14 +++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/RFC/src/Glossary.md b/RFC/src/Glossary.md index 47ce4b81dd..9e8f3cd8a5 100644 --- a/RFC/src/Glossary.md +++ b/RFC/src/Glossary.md @@ -251,7 +251,7 @@ A Tari Wallet is responsible for managing key pairs, and for constructing and ne ## Pruning horizon -[pruninghorizon]: #pruninghorizon "Block height at which pruning must commence" +[pruninghorizon]: #pruninghorizon "Block height at which pruning will commence" This is a local setting for each node to help reduce syncing time and bandwidth. This is the number of blocks from the chain tip beyond which a chain will be pruned. @@ -261,6 +261,8 @@ This is a local setting for each node to help reduce syncing time and bandwidth. This is a full history [base node]. It will keep a complete history of every transaction ever received and it will not implement pruning. + + # Disclaimer This document is subject to the [disclaimer](../DISCLAIMER.md). diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 684391b9b8..4d46207cda 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -58,7 +58,7 @@ This document describes the process of Syncing, seeding, pruning and cut-through ### Syncing -When a new node comes online, loses connection or encounters a chain re-organisation that is longer than it can tolerate, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Syncing can be divided into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will involve that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [pruning horizon](pruninghorizon) to [current head](current head), as well as every block header from genesis block. +When a new node comes online, loses connection or encounters a chain re-organisation that is longer than it can tolerate, it must enter syncing mode. This will allow it to recover its state to the newest up to date state. Syncing can be divided into 2 [SynchronisationStrategy]s, complete sync and sync. Complete sync will mean that the node communicates with an archive node to get the complete history of every single block from genesis block. Sync will involve the node getting every block from its [pruning horizon](pruninghorizon) to [current head](current head), as well as every block header from genesis block. #### Complete Sync @@ -70,23 +70,23 @@ Complete sync is only available from archive nodes, as these will be the only no The syncing process MUST be done in the following steps: -1. Set its [SynchronisationState] to `Synchronising`. +1. Set [SynchronisationState] to `Synchronising`. 2. Load a bootstrap list of peers from a configuration file, or a cached list. 3. For each peer in this list: 1. Establish a connection with the peer. 2. Request a peer list from that peer. 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. -4. Choose the longest chain based on POW. +4. Choose the longest chain based on PoW. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. 6. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). -7. Download all blocks from [pruning horizon](pruninghorizon) up onto [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. +7. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. 8. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. #### Keeping in sync -The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total POW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST relay with a pong message also including the total POW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest POW has the responsiblity to ask the peer for syncing information. +The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest PoW has the responsiblity to ask the peer for syncing information. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. @@ -94,7 +94,7 @@ This will be handled by the node asking for each block from the [current head](c #### Chain forks -Chain forks can be a problem since in mimblewimble not all nodes keep the complete history, the design philosophy is more in the lines of only keeping the current [UTXO](utxo). Only keeping the current [UTXO](utxo) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in MimbleWimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [UTXO](utxo). Only keeping the current [UTXO](utxo) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. @@ -102,7 +102,7 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e [Pruning and cut-through]: #Pruning-and-cut-through "Remove already spent outputs from the [utxo]" -In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the inputs and outputs, but will retain the excesses of each [transaction]. +In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the spent inputs and outputs, but will retain the excesses of each [transaction]. Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. From 9ccaa77c436c86cacf9b5f01bae387e3103c7d37 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Wed, 6 Feb 2019 08:13:13 +0200 Subject: [PATCH 18/22] refactored to try and reduce confusion --- RFC/src/Glossary.md | 4 ++++ RFC/src/RFC-0140 Syncing and seeding.md | 5 +++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/RFC/src/Glossary.md b/RFC/src/Glossary.md index 9e8f3cd8a5..2c9c3bc06c 100644 --- a/RFC/src/Glossary.md +++ b/RFC/src/Glossary.md @@ -261,7 +261,11 @@ This is a local setting for each node to help reduce syncing time and bandwidth. This is a full history [base node]. It will keep a complete history of every transaction ever received and it will not implement pruning. +## Blockchain state +[blockchainstate]: #blockchainstate "This is a snapshot of how the blockchain looks" + +This is a complete snapshot of the blockchain at a spesific block height. This means pruned [utxo], complete set of kernals and headers up to that block height from genesis block is known. # Disclaimer diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140 Syncing and seeding.md index 4d46207cda..6fb8366a76 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140 Syncing and seeding.md @@ -94,7 +94,7 @@ This will be handled by the node asking for each block from the [current head](c #### Chain forks -Chain forks can be a problem since in MimbleWimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [UTXO](utxo). Only keeping the current [UTXO](utxo) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in MimbleWimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [Blockchain state](blockchainstate). Only keeping the current [Blockchain state](blockchainstate) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. @@ -113,6 +113,7 @@ When running in pruning mode, [base node]s have the following responsibilities: [archivenode]: Glossary.md#archivenode +[blockchainstate]: Glossary.md#blockchainstate [pruninghorizon]: Glossary.md#pruninghorizon [tari coin]: Glossary.md#tari-coin [blockchain]: Glossary.md#blockchain @@ -129,4 +130,4 @@ When running in pruning mode, [base node]s have the following responsibilities: [SynchronisationStrategy]: Glossary.md#synchronisationstrategy [SynchronisationState]: Glossary.md#synchronisationstate [mining server]: Glossary.md#mining-server -[cut-through]: RFC-0110_BaseNodes.md#Pruning-and-cut-through +[cut-through]: RFC-0110_BaseNodes.md#Pruning-and-cut-through \ No newline at end of file From 5c14aaa00286576aaf55f641cd190de15ca9c409 Mon Sep 17 00:00:00 2001 From: Hansie Odendaal Date: Wed, 6 Feb 2019 08:39:20 +0200 Subject: [PATCH 19/22] Renamed file without spaces for inclusion in SUMMARY.md, otherwise mdbook build gives ERRORS Spelling corrections --- ... and seeding.md => RFC-0140_Syncing_and_seeding.md} | 10 +++++----- RFC/src/SUMMARY.md | 1 + 2 files changed, 6 insertions(+), 5 deletions(-) rename RFC/src/{RFC-0140 Syncing and seeding.md => RFC-0140_Syncing_and_seeding.md} (94%) diff --git a/RFC/src/RFC-0140 Syncing and seeding.md b/RFC/src/RFC-0140_Syncing_and_seeding.md similarity index 94% rename from RFC/src/RFC-0140 Syncing and seeding.md rename to RFC/src/RFC-0140_Syncing_and_seeding.md index 6fb8366a76..56a06b132e 100644 --- a/RFC/src/RFC-0140 Syncing and seeding.md +++ b/RFC/src/RFC-0140_Syncing_and_seeding.md @@ -76,7 +76,7 @@ The syncing process MUST be done in the following steps: 1. Establish a connection with the peer. 2. Request a peer list from that peer. 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. -4. Choose the longest chain based on PoW. +4. Choose the longest chain based on PoW. 5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. 6. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). 7. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. @@ -86,7 +86,7 @@ After this process the node will be in sync and able to process blocks and trans #### Keeping in sync -The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains dont match up, the node with the lowest PoW has the responsiblity to ask the peer for syncing information. +The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains don't match up, the node with the lowest PoW has the responsibility to ask the peer for syncing information. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. @@ -94,7 +94,7 @@ This will be handled by the node asking for each block from the [current head](c #### Chain forks -Chain forks can be a problem since in MimbleWimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [Blockchain state](blockchainstate). Only keeping the current [Blockchain state](blockchainstate) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in Mimblewimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [Blockchain state](blockchainstate). Only keeping the current [Blockchain state](blockchainstate) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. @@ -102,13 +102,13 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e [Pruning and cut-through]: #Pruning-and-cut-through "Remove already spent outputs from the [utxo]" -In MimbleWimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the spent inputs and outputs, but will retain the excesses of each [transaction]. +In Mimblewimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the spent inputs and outputs, but will retain the excesses of each [transaction]. Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. When running in pruning mode, [base node]s have the following responsibilities: -1. MUST remove all spent outputs thats older than the [pruning horizon](pruninghorizon) in it's current stored [UTXO](utxo) when a new block is received from another [base node]. +1. MUST remove all spent outputs that is older than the [pruning horizon](pruninghorizon) in it's current stored [UTXO](utxo) when a new block is received from another [base node]. diff --git a/RFC/src/SUMMARY.md b/RFC/src/SUMMARY.md index d0fee1f33d..3e0384bb1e 100644 --- a/RFC/src/SUMMARY.md +++ b/RFC/src/SUMMARY.md @@ -6,6 +6,7 @@ - [RFC-0100: The Tari Base Layer](RFC-0100_BaseLayer.md) - [RFC-0110: Base nodes](RFC-0110_BaseNodes.md) - [RFC-0130: Mining](RFC-0130_Mining.md) + - [RFC-0140: Sync and Seeding](RFC-0140_Syncing_and_seeding.md) - [RFC-0150: Wallets](RFC-0150_Wallets.md) - [RFC-0200: Tari-specific extensions to Mimblewimble](RFC-0200_BaseLayerExtensions.md) - [RFC-0210: Namespace registration](RFC-0210_NamespaceRegistration.md) From 7a6f6dc18487e9c48ca1fd8373f49c8aa1188c12 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Wed, 6 Feb 2019 09:36:13 +0200 Subject: [PATCH 20/22] review comments --- RFC/src/RFC-0140_Syncing_and_seeding.md | 31 +++++++++++++------------ 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/RFC/src/RFC-0140_Syncing_and_seeding.md b/RFC/src/RFC-0140_Syncing_and_seeding.md index 56a06b132e..3c667df97b 100644 --- a/RFC/src/RFC-0140_Syncing_and_seeding.md +++ b/RFC/src/RFC-0140_Syncing_and_seeding.md @@ -71,30 +71,31 @@ Complete sync is only available from archive nodes, as these will be the only no The syncing process MUST be done in the following steps: 1. Set [SynchronisationState] to `Synchronising`. -2. Load a bootstrap list of peers from a configuration file, or a cached list. -3. For each peer in this list: - 1. Establish a connection with the peer. - 2. Request a peer list from that peer. - 3. Request information about the most recent chain state (total accumulated work, block height, etc.) from the peer. -4. Choose the longest chain based on PoW. -5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. -6. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). -7. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. -8. Validate blocks as if they where just mined and then received, in chronological order. +2. Selects a connected peer to sync from, this is based on the following criteria teria: + 1. Does the peer have a high enough [pruning horizon](pruninghorizon). + 2. Does the peer allow syncing. + 3. Does the peer have a low latency. +3. Choose the longest chain based on PoW. +4. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. +5. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). +6. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. +7. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. #### Keeping in sync -The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains don't match up, the node with the lowest PoW has the responsibility to ask the peer for syncing information. +The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains don't match up, the node with the lowest PoW has the responsibility to ask the peer for syncing information and set [SynchronisationState] to `Synchronising`. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. -This will be handled by the node asking for each block from the [current head](currenthead) going backward for older blocks until a known block is found. If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). +This will be handled by the node asking for each block header from the [current head](currenthead) going backward for older blocks until a known block is found. If a known block is found, and it has missing blocks it MUST set [SynchronisationState] to `Synchronising` while it is busy catching up those blocks. + +If no block is found, the node will enter sync mode and resync. It cannot recover from its state as the fork is older than its [pruning horizon](pruninghorizon). #### Chain forks -Chain forks can be a problem since in Mimblewimble not all nodes keep the complete history, the design philosophy is more along the lines of only keeping the current [Blockchain state](blockchainstate). Only keeping the current [Blockchain state](blockchainstate) will not be possible, because if you encounter a fork where there are two running version on the blockchain, you will not be able to swop without doing sync again. +Chain forks can be a problem since in Mimblewimble not all nodes keep the complete transaction history, the design philosophy is more along the lines of only keeping the current [Blockchain state](blockchainstate). However, if such a node only maintains only the current [Blockchain state](blockchainstate) it is not possible for the node to "rewind" its state to handle forks in the chain history. In this case, a mode must re-sync its chain to recover the necessary transaction history up onto its [pruning horizon](pruninghorizon). To counter this problem we use [pruning horizon](pruninghorizon), this allows every [node](base node) to be a "light" [archival node](archivenode). This in effect means that the node will keep a full history for a short while. If the node encounters a fork it can easily rewind its state to apply the fork. If the fork is longer than the [pruning horizon](pruninghorizon), the node will enter a sync state where it will resync. @@ -102,9 +103,9 @@ To counter this problem we use [pruning horizon](pruninghorizon), this allows e [Pruning and cut-through]: #Pruning-and-cut-through "Remove already spent outputs from the [utxo]" -In Mimblewimble, the state can be completely verified using the current [UTXO](utxo) set, the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the spent inputs and outputs, but will retain the excesses of each [transaction]. +In Mimblewimble, the state can be completely verified using the current [UTXO](utxo) set (which contains the output commitments and range proofs), the set of excess signatures (contained in the transaction kernels) and the proof-of-work. The full block and transaction history is not required. This allows base layer nodes to remove old used inputs from the [blockchain] and or the [mempool]. [Cut-through](cut-through) happens in the [mempool] while pruning happens in the [blockchain] with already confirmed transactions. This will remove the spent inputs and outputs, but will retain the excesses of each [transaction]. -Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it should not prune. +Pruning is only for the benefit of the local base node as it reduces the local blockchain size. Pruning only happens after the block is older than the [pruning horizon](pruninghorizon) height. A Base node will either run in archive mode or prune mode, if the base node is running in archive mode it MUST NOT prune. When running in pruning mode, [base node]s have the following responsibilities: From 61c486e0a8cb0efc694f028ab741b2e2b0cc0bc9 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Thu, 7 Feb 2019 11:58:56 +0200 Subject: [PATCH 21/22] add review comments --- RFC/src/RFC-0140_Syncing_and_seeding.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/RFC/src/RFC-0140_Syncing_and_seeding.md b/RFC/src/RFC-0140_Syncing_and_seeding.md index 3c667df97b..de52f0a779 100644 --- a/RFC/src/RFC-0140_Syncing_and_seeding.md +++ b/RFC/src/RFC-0140_Syncing_and_seeding.md @@ -71,21 +71,22 @@ Complete sync is only available from archive nodes, as these will be the only no The syncing process MUST be done in the following steps: 1. Set [SynchronisationState] to `Synchronising`. -2. Selects a connected peer to sync from, this is based on the following criteria teria: +2. Asks peers for their latest block, so it can get the total proof of work. +3. Choose the longest chain based on total PoW done on that chain. +4. Selects a connected peer with the logest chain to sync from, this is based on the following criteria teria: 1. Does the peer have a high enough [pruning horizon](pruninghorizon). 2. Does the peer allow syncing. 3. Does the peer have a low latency. -3. Choose the longest chain based on PoW. -4. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. -5. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). -6. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. -7. Validate blocks as if they where just mined and then received, in chronological order. +5. Download all headers from genesis block up onto [current head](currenthead), and validate the headers as you receive them. +6. Download [UTXO](utxo) set at [pruning horizon](pruninghorizon). +7. Download all blocks from [pruning horizon](pruninghorizon) up to [current head](currenthead), if the node is doing a complete sync, the [pruning horizon](pruninghorizon) will just be infinite, which means you will download all blocks ever created. +8. Validate blocks as if they where just mined and then received, in chronological order. After this process the node will be in sync and able to process blocks and transaction normally. #### Keeping in sync -The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it should include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains don't match up, the node with the lowest PoW has the responsibility to ask the peer for syncing information and set [SynchronisationState] to `Synchronising`. +The node SHOULD periodically test its peers with ping messages to ensure that they are alive. When a node sends a ping message, it MUST include the current total PoW, hash of the [current head](currenthead) and genesis block hash of its own current longest chain in the ping message. The receiving node MUST reply with a pong message also including the total PoW, [current head](currenthead) and genesis block hash of its longest chain. If the two chains don't match up, the node with the lowest PoW has the responsibility to ask the peer for syncing information and set [SynchronisationState] to `Synchronising`. If the genesis block hashes don't match, the node is removed from its peer list as this node is running a different blockchain. From 803b432ecb2eda1856885a0071b0a9c999afb9f2 Mon Sep 17 00:00:00 2001 From: Schalk van Heerden Date: Thu, 7 Feb 2019 14:06:24 +0200 Subject: [PATCH 22/22] fixed some typos --- RFC/src/RFC-0140_Syncing_and_seeding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/src/RFC-0140_Syncing_and_seeding.md b/RFC/src/RFC-0140_Syncing_and_seeding.md index de52f0a779..8077f8cee0 100644 --- a/RFC/src/RFC-0140_Syncing_and_seeding.md +++ b/RFC/src/RFC-0140_Syncing_and_seeding.md @@ -73,7 +73,7 @@ The syncing process MUST be done in the following steps: 1. Set [SynchronisationState] to `Synchronising`. 2. Asks peers for their latest block, so it can get the total proof of work. 3. Choose the longest chain based on total PoW done on that chain. -4. Selects a connected peer with the logest chain to sync from, this is based on the following criteria teria: +4. Selects a connected peer with the longest chain to sync from, this is based on the following criteria: 1. Does the peer have a high enough [pruning horizon](pruninghorizon). 2. Does the peer allow syncing. 3. Does the peer have a low latency.