feat!: fix horizon sync after smt upgrade #6006

hansieodendaal · 2023-12-04T14:47:14Z

Description

Fixed horizon sync:
- initial sync;
- re-sync after initial sync;
- re-sync after being offline;
- sync from prune node after initial sync.
Added logic to detect genesis block outputs being spent.
Fixed an issue where a tip block body could not be inserted due to the input being a compact input.
Removed unused release mode code.
~~Renamed some struct members and functions related to these changes to depict their use better.~~
Edit: Fixed fn fetch_outputs_in_block_with_spend_state(...) whereby it did not filter out outputs with a spent state at the target header. Updated integration level unit test test_horizon_sync_from_archival_node_happy_path() to verify this behaviour.
Edit: Fixed fn prune_outputs_spent_at_hash(..) whereby it used the wrong key(s) to try and prune outputs.

Note: Initial prune node sync can still be optimized if we can allow it to happen from another prune node, as this PR restricts initial prune node sync from an archival node. That is left for another PR.

Motivation and Context

Horizon sync was not working.
There were no integration-level horizon sync unit tests.

How Has This Been Tested?

Added integration-level horizon sync unit tests.
Added integration-level block sync unit test.
System-level tests [TBD]

What process can a PR reviewer use to test or verify this change?

Code walk-through.
Run the integration-level horizon sync and block sync unit tests.
Selected output of test_horizon_sync_from_archival_node_happy_path() with trace logs is added here to assist reviewers (pr_#6006.log):
- The horizon sync scenario is from blocks 10 to 20, where coinbases in blocks 10 to 15 are spent in block 16.
- Notice log extract where outputs from block 11 to block 15 as well as corresponding inputs in block 16 are not streamed.

[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Starting UTXO stream for peer '4e0e87239540d0b25f401283c5'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Starting stream task with current_header: 14d84dc13934e94a79618fa733de7e877dd5a6e80fd67fc0d26074b6a11510e3, end_header: 0b43c2da6011e57ab95451df2630234b5e3f3674f6584e48d0d357d13a6b5e6f
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #11 (14d84dc13934e94a79618fa733de7e877dd5a6e80fd67fc0d26074b6a11510e3)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 outputs in response for block #11 '14d84dc13934e94a79618fa733de7e877dd5a6e80fd67fc0d26074b6a11510e3'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 inputs in response for block #11 '14d84dc13934e94a79618fa733de7e877dd5a6e80fd67fc0d26074b6a11510e3'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streamed 0 TXOs in 231.60┬╡s (including stream backpressure)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #12 (d8e09ed6b43ad65d64950f804990dc181d3798d1c8d19c6924c0f7ffd0185aa1)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 outputs in response for block #12 'd8e09ed6b43ad65d64950f804990dc181d3798d1c8d19c6924c0f7ffd0185aa1'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 inputs in response for block #12 'd8e09ed6b43ad65d64950f804990dc181d3798d1c8d19c6924c0f7ffd0185aa1'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streamed 0 TXOs in 134.80┬╡s (including stream backpressure)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #13 (b31f587fbd545f6ce1be855b88388e9efbf4bf9f4205aa9088d2f1d372db2826)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 outputs in response for block #13 'b31f587fbd545f6ce1be855b88388e9efbf4bf9f4205aa9088d2f1d372db2826'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 inputs in response for block #13 'b31f587fbd545f6ce1be855b88388e9efbf4bf9f4205aa9088d2f1d372db2826'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streamed 0 TXOs in 118.10┬╡s (including stream backpressure)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #14 (48271b358f9d2f08b406ee6c482f98716b3b03eef44f28610e58cca805837142)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 outputs in response for block #14 '48271b358f9d2f08b406ee6c482f98716b3b03eef44f28610e58cca805837142'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 inputs in response for block #14 '48271b358f9d2f08b406ee6c482f98716b3b03eef44f28610e58cca805837142'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streamed 0 TXOs in 102.40┬╡s (including stream backpressure)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #15 (23e1772f4fd691d7e8ffed59ac8c5ac5d790352130e0c527e1ecd9e70bc9d4d1)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 outputs in response for block #15 '23e1772f4fd691d7e8ffed59ac8c5ac5d790352130e0c527e1ecd9e70bc9d4d1'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 0 inputs in response for block #15 '23e1772f4fd691d7e8ffed59ac8c5ac5d790352130e0c527e1ecd9e70bc9d4d1'
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streamed 0 TXOs in 150.90┬╡s (including stream backpressure)
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Streaming TXO(s) for block #16 (124ab4fd9e06e05a08656d071e75109f319ea856c288fbe0f79e008eb01ced89)
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment 'cab362382fa13da54ae5059eacbc0aedb91d10834569a27c3a0cc95008059f6b') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '96d3323d84ab40ba6c84a3151337cedd682e4c0a7acea2efbafa47ca5edca462') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '1c0498276efd43a04a9850313b93090c0e69fdbdd9e3b0eb1e270fdcac6dc729') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '98ebfe46f098343e40baa03c6820c26a28c39d82347947fe893b48270231df28') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '087ae3cd0c4afd40ed89e96ba4d90e117d7716622724491392eff8e5da2a1243') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '1e6bdc6fd6a443afbcf45a3415589b5f975bb24f1e745a0f1e082558874b8c41') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '381216fd50669b29027e2fd6943b8cf018e4edf4bee33a30fc41d8b62baf9212') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment 'b8887c04b12ff98ff75f9fda031d8dd7f5969f07afd4ce474a252958341ee515') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '9c430777aa2a4a93805e26a20a5320c079f7f0a69458142b9c9cfb61b82c3a55') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '1ca47c3bb33bdcd71be5f52eea9c7c35ab6c2e1a19b6b9d7a5bbbd4c9291506c') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '426891a0a609381ef66da5bdbe8b066bfeaec210b665d141ffc2b5fafc936009') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '44fda12192b185831a60f0850de493885105ee8fe1c68046ae23bb5a6cc1220d') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '68aab326d389391768947ac15c8d41b20e4d75fd35f21be99e1f9814c28f4c51') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment 'ec51f02c1fa611f98562c1199e05fabb7e573dbbd80f051fe7aa911bc5618348') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '8c136433dec73c1719bdb6367810833e17c22c29b752ef9ed1ecbf4af6efe017') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '4a1cf051b5215abb9d1bee55d2a3149bf5988d070d490b306aaa48928c3c7800') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '32867e8efa78f627e8b42585f13ebda587214a6c0a88cff1c322ec61b1009e34') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '8ef63b672ebc557d6134a77b66eb0025d2dbc7e5a99f9c2271139e457266583f') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment 'f040c3b43ab7283b693a05ad3ed07267afe3cbce7ead7f87dc883e7256686d6c') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '388d89c16078f0246016c6b4ef868043e0f35bc56284a17472d314a2b9a8735a') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '1a917dac4d616005d44039ae4cb5de86a793ec89c6ad99beef999272f74c4e3d') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Unspent TXO (commitment '742000c5781ca57a5cf7d87dc1f5c3fdb4043027c04c66bc954d5418438c3411') to peer
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 22 outputs in response for block #16 '124ab4fd9e06e05a08656d071e75109f319ea856c288fbe0f79e008eb01ced89'
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (hash '6c786a9017704c477fe326ea87a605c3c4c59779035677b9d478418008ca8338') not sent to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment 'acbc67b4d304afea0d004058881c4b11571200b5571a2324ca26f3544475d168') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment 'ec49e91354cbef0fdaa599d50ba7db91ff71a77c8292a4d0d6bee8ccba41c32e') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (hash 'c403de95b0caebe0ad0f173c36976201739fecaabdedeae29dce17e71c1b142d') not sent to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '18b6707d0e44537b8fddbc02306ef3f7e1629751bb30fb194b3f3933047dd53a') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (hash 'e16dc3d652d5aeb1f6996b9071f29425d744101f830a56385fff4263e274026b') not sent to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '9ee42164e479a3e2451dea627d4c97e9fc3f2283808a620eb7ef2f36acbb212a') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment 'b26bcee38a6d3c01701b62b679ab31796771e2e03dda0804c04eb5191f98664e') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '1cedd6437174f06a31f610d45b6a199183d4f76af7ad53b902f40ad37c50d306') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '5024d962d4bd92897f9aaa6990f9a30814c896cb9375e11aa121360c6204324d') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '926066f00276815389d512d74316282a520331dfab890b1a49dfc5cc55ebb454') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment 'e43c29015632d5bb50c51d15f5fd669e6b50e44506cd36913d15ae5f785f9e39') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (commitment '2a6ee8813d2203cd4ff4644ca270e4a9b143c973d634ab583fcfbcf46b486405') to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (hash '31ab562c38b188b264421da2d0555aa07227d0965796433a7bc90f3a11b56430') not sent to peer
[2024-01-24T13:14:34Z TRACE tari_core::base_node::sync::rpc::sync_utxos_task] Spent TXO (hash '7bc39d95f57bec3f5baaf58b12dc6ab79fd73ae52e355e88dc2c1b3e0fb787e8') not sent to peer
[2024-01-24T13:14:34Z DEBUG tari_core::base_node::sync::rpc::sync_utxos_task] Adding 10 inputs in response for block #16 '124ab4fd9e06e05a08656d071e75109f319ea856c288fbe0f79e008eb01ced89'

Breaking Changes

None
Requires data directory on base node to be deleted
Requires hard fork
Other - Please specify

BREAKING CHANGE: Sync nodes can only sync from base nodes running the same or later software version

github-actions · 2024-01-12T11:09:17Z

Test Results (CI)

1 268 tests 1 268 ✅ 11m 9s ⏱️
39 suites 0 💤
1 files 0 ❌

Results for commit 658679f.

♻️ This comment has been updated with latest results.

github-actions · 2024-01-12T11:30:30Z

Test Results (Integration tests)

2 files 11 suites 15m 41s ⏱️
29 tests 27 ✅ 0 💤 2 ❌
31 runs 29 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 658679f.

♻️ This comment has been updated with latest results.

brianp

Ack

base_layer/core/src/base_node/sync/horizon_state_sync/synchronizer.rs

hansieodendaal · 2024-01-25T08:17:41Z

WIP notice: The system-level use needs some love, specifically cleaning out the output tables after a failed attempt.

Edit:: System level tests pass doing horizon sync from scratch and form an old or new archival node.

SWvheerden

This looks good, just some comments

base_layer/core/src/base_node/state_machine_service/states/horizon_state_sync.rs

base_layer/core/src/base_node/state_machine_service/states/sync_decide.rs

base_layer/core/src/base_node/sync/horizon_state_sync/synchronizer.rs

base_layer/core/src/base_node/sync/rpc/sync_utxos_task.rs

base_layer/core/src/chain_storage/blockchain_database.rs

base_layer/core/src/base_node/sync/horizon_state_sync/synchronizer.rs

base_layer/core/src/chain_storage/lmdb_db/lmdb_db.rs

SWvheerden

You can fix this in a next PR or this one.

base_layer/core/src/base_node/state_machine_service/states/horizon_state_sync.rs

- Added integration-level horizon sync unit tests. - Fixed horizon sync: initial, re-sync, re-sync after being offline. - Added logic to detect genesys block outputs being spend. - Fixed an issue where a tip block body could not be inserted due to the input being a compact input. - Added integration-level block sync unit tests to verufy ^^.

hansieodendaal mentioned this pull request Jan 10, 2024

Add horizon sync integration-level unit tests #6071

Closed

hansieodendaal force-pushed the ho_horizon_sync branch 2 times, most recently from 1fa90d3 to 0a22fb3 Compare January 12, 2024 10:52

ghpbot-tari-project added P-acks_required Process - Requires more ACKs or utACKs P-reviews_required Process - Requires a review from a lead maintainer to be merged labels Jan 12, 2024

hansieodendaal force-pushed the ho_horizon_sync branch 7 times, most recently from cc526fa to 9bae457 Compare January 19, 2024 12:57

ghpbot-tari-project added the CR-too_long Changes Requested - Your PR is too long label Jan 21, 2024

hansieodendaal requested a review from a team as a code owner January 22, 2024 12:56

hansieodendaal force-pushed the ho_horizon_sync branch 4 times, most recently from a5b7550 to 1c005e1 Compare January 23, 2024 11:07

hansieodendaal changed the title ~~[wip] feat: add horizon sync integration unit tests~~ feat: add horizon sync integration unit tests Jan 23, 2024

hansieodendaal changed the title ~~feat: add horizon sync integration unit tests~~ feat: fix horizon sync after smt upgrade Jan 23, 2024

brianp previously approved these changes Jan 23, 2024

View reviewed changes

base_layer/core/src/base_node/sync/horizon_state_sync/synchronizer.rs Show resolved Hide resolved

ghpbot-tari-project removed P-reviews_required Process - Requires a review from a lead maintainer to be merged labels Jan 23, 2024

hansieodendaal dismissed brianp’s stale review via cc5716a January 24, 2024 13:21

ghpbot-tari-project added the P-reviews_required Process - Requires a review from a lead maintainer to be merged label Jan 24, 2024

hansieodendaal marked this pull request as draft January 25, 2024 08:14

hansieodendaal mentioned this pull request Jan 25, 2024

Horizon syn correct behavior #5771

Closed

hansieodendaal force-pushed the ho_horizon_sync branch from 79d8591 to 7401d33 Compare January 26, 2024 13:13

hansieodendaal marked this pull request as ready for review January 26, 2024 13:14

hansieodendaal mentioned this pull request Jan 26, 2024

Database inconsistant state when pruning #6104

Closed

SWvheerden reviewed Jan 29, 2024

View reviewed changes

hansieodendaal changed the title ~~feat: fix horizon sync after smt upgrade~~ feat!: fix horizon sync after smt upgrade Jan 30, 2024

hansieodendaal force-pushed the ho_horizon_sync branch from 7401d33 to 15a6c18 Compare January 30, 2024 12:02

SWvheerden previously approved these changes Jan 30, 2024

View reviewed changes

ghpbot-tari-project removed the P-reviews_required Process - Requires a review from a lead maintainer to be merged label Jan 30, 2024

SWvheerden reviewed Jan 30, 2024

View reviewed changes

base_layer/core/src/base_node/state_machine_service/states/horizon_state_sync.rs Outdated Show resolved Hide resolved

base_layer/core/src/base_node/state_machine_service/states/horizon_state_sync.rs Outdated Show resolved Hide resolved

hansieodendaal dismissed SWvheerden’s stale review via 9931495 January 31, 2024 09:51

hansieodendaal force-pushed the ho_horizon_sync branch from f7e7136 to 9931495 Compare January 31, 2024 09:51

hansieodendaal force-pushed the ho_horizon_sync branch from 9931495 to 658679f Compare January 31, 2024 11:04

ghpbot-tari-project added the P-reviews_required Process - Requires a review from a lead maintainer to be merged label Jan 31, 2024

SWvheerden approved these changes Jan 31, 2024

View reviewed changes

ghpbot-tari-project removed the P-reviews_required Process - Requires a review from a lead maintainer to be merged label Jan 31, 2024

SWvheerden merged commit b6b80f6 into tari-project:development Jan 31, 2024
12 of 14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: fix horizon sync after smt upgrade #6006

feat!: fix horizon sync after smt upgrade #6006

hansieodendaal commented Dec 4, 2023 •

edited

Loading

github-actions bot commented Jan 12, 2024 •

edited

Loading

github-actions bot commented Jan 12, 2024 •

edited

Loading

brianp left a comment

hansieodendaal commented Jan 25, 2024 •

edited

Loading

SWvheerden left a comment

SWvheerden left a comment

feat!: fix horizon sync after smt upgrade #6006

feat!: fix horizon sync after smt upgrade #6006

Conversation

hansieodendaal commented Dec 4, 2023 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

What process can a PR reviewer use to test or verify this change?

Breaking Changes

github-actions bot commented Jan 12, 2024 • edited Loading

Test Results (CI)

github-actions bot commented Jan 12, 2024 • edited Loading

Test Results (Integration tests)

brianp left a comment

Choose a reason for hiding this comment

hansieodendaal commented Jan 25, 2024 • edited Loading

SWvheerden left a comment

Choose a reason for hiding this comment

SWvheerden left a comment

Choose a reason for hiding this comment

hansieodendaal commented Dec 4, 2023 •

edited

Loading

github-actions bot commented Jan 12, 2024 •

edited

Loading

github-actions bot commented Jan 12, 2024 •

edited

Loading

hansieodendaal commented Jan 25, 2024 •

edited

Loading