fix(core): periodically commit large transaction in prune_to_height #4805
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
ACK |
|
A bit of a leaky abstraction -- it's arguably an LMDB-specific intervention in the More of a note for the future here, because the abstractions are FUBARed everywhere when it comes to the DB. |
CjS77
approved these changes
Oct 18, 2022
sdbondi
added a commit
to sdbondi/tari
that referenced
this pull request
Oct 19, 2022
* development: fix(core): dont request full non-tip block if block is empty (tari-project#4802) feat(comms): adds periodic socket-level liveness checks (tari-project#4819) feat: add multisig script that returns aggregate of signed public keys (tari-project#4742) fix(core): increase sync timeouts (tari-project#4800) fix(comms/rpc): measures client-side latency to first message received (tari-project#4817) fix(core): periodically commit large transaction in prune_to_height (tari-project#4805) feat: add deepsource config v0.38.7 test: remove cucumber tests, simplify others (tari-project#4794) fix(miner): clippy error (tari-project#4793) fix(core): only resize db if migration is required (tari-project#4792) v0.38.6 fix(dht): remove some invalid saf failure cases (tari-project#4787) feat: move nonce to first in sha hash (tari-project#4778) feat: optimize transaction service queries (tari-project#4775) fix(tari-script): use tari script encoding for execution stack serde de/serialization (tari-project#4791) fix(p2p/liveness): remove fallible unwrap (tari-project#4784) fix: fix config.toml bug (tari-project#4780) fix: batch rewind operations (tari-project#4752)
CjS77
added a commit
that referenced
this pull request
Oct 19, 2022
* fix: batch rewind operations (#4752) Description --- Split rewind DbTx into smaller pieces. How Has This Been Tested? --- I did rewind on 20000+ (empty) blocks. * fix: fix config.toml bug (#4780) Description --- The base node errored when reading the `block_sync_trigger = 5` setting ``` ExitError { exit_code: ConfigError, details: Some("Invalid value for `base_node`: unknown field `block_sync_trigger`, expected one of `override_from`, `unconfirmed_pool`, `reorg_pool`, `service`") } ``` Motivation and Context --- Reading default config settings should not cause an error How Has This Been Tested? --- System level testing * fix(p2p/liveness): remove fallible unwrap (#4784) Description --- Removed stray unwrap in liveness service Motivation and Context --- Caused a base node to panic in stress test conditions. ``` thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: DhtOutboundError(RequesterReplyChannelClosed)', base_layer\p2p\src\services\liveness\service.rs:164:71 ``` How Has This Been Tested? --- Tests pass * fix(tari-script): use tari script encoding for execution stack serde de/serialization (#4791) Description --- - Uses tari script encoding (equivalent to consensus encoding) for `ExecutionStack` serde impl - Rename as_bytes to to_bytes as per rust convention. - adds migration to fix execution stack encoding in db Motivation and Context --- Resolves #4790 How Has This Been Tested? --- Added test to alert if breaking changes occur with serde serialization for execution stack. Manual testing in progress * feat: optimize transaction service queries (#4775) Description --- Transaction service sql db queries must handle `DieselError(DatabaseError(__Unknown, "database is locked"))`. This PR attempts to remove situations where that error may occur under highly busy async cirumstances, specifically: - Combine find and update/write type queries into one. - Add sql transactions around complex tasks. _**Note:** Partial resolution for #4731._ Motivation and Context --- See above. How Has This Been Tested? --- - Passed unit tests. - Passed cucumber tests. - ~~**TODO:**~~ System level tests under stress conditions. * feat: move nonce to first in sha hash (#4778) Description --- This moves the nonce to the front of the hashing order when hashing for the sha3 difficulty. This is done so that mining cannot cache part most the header and only load the nonce in. This forces the miner to hash the complete header each time the nonce chances. Motivation and Context --- Fixes: #4767 How Has This Been Tested? --- Unit tests all pass. * fix(dht): remove some invalid saf failure cases (#4787) Description --- - Ignores nanos for `stored_at` field in StoredMessages - Uses direct u32 <-> i32 conversion - Improve error message if attempting to store an expired message - Discard expired messages immediately - Debug log when remote client closes the connection in RPC server Motivation and Context --- - Nano conversion will fail when >= 2_000_000_000, nanos are not important to preserve so we ignore them (set to zero) - u32 to/from i32 conversion does not lose any data as both are 32-bit, only used as i32 in the database - 'The message was not valid for store and forward' occurs if the message has expired, this PR uses a more descriptive error message for this specific case. - Expired messages should be discarded immediately - Early close "errors" on the rpc server simply indicate that the client went away, which is expected and not something that the server controls, and so is logged at debug level How Has This Been Tested? --- Manually, * v0.38.6 * fix(core): only resize db if migration is required (#4792) Description --- Adds conditional to only increase database size if migration is required Motivation and Context --- A new database (cucumber, functional tests) has no inputs and so migration is not required. Ref #4791 How Has This Been Tested? --- * fix(miner): clippy error (#4793) Description --- Removes unused function in miner Motivation and Context --- Clippy How Has This Been Tested? --- No clippy error * test: remove cucumber tests, simplify others (#4794) Description --- * remove auto update tests from cucumber * rename some tests to be prefixed with `test_` * simplified two cucumber tests by removing steps Motivation and Context --- The auto update tests have an external dependency, which makes it hard to test reliably. They were marked as broken, so I rather removed them. There were two steps in the `list_height` and `list_headers` tests that created base nodes. Upon inspection of the logs, these base nodes never synced to the height of 5 and were not checked in the test, so were pretty useless and just slowed the test down How Has This Been Tested? --- npm test * v0.38.7 * feat: add deepsource config * fix(core): periodically commit large transaction in prune_to_height (#4805) * fix(comms/rpc): measures client-side latency to first message received (#4817) * fix(core): increase sync timeouts (#4800) Co-authored-by: Cayle Sharrock <CjS77@users.noreply.github.com> * feat: add multisig script that returns aggregate of signed public keys (#4742) Description --- Added an `m-of-n` multisig TariScript that returns the aggregate public key of the signatories if successful and fails otherwise. This is useful if the aggregate public key of the signatories is also the script public key, where signatories would work together to create an aggregate script signature using their individual script private keys. Motivation and Context --- To enhance the practicality of the `m-of-n` multisig TariScript. How Has This Been Tested? --- Unit tests Co-Authored-By: SW van Heerden swvheerden@gmail.com * feat(comms): adds periodic socket-level liveness checks (#4819) Description --- - adds socket-level liveness checks - adds configuration to enable liveness checks (currently enabled by default in base node, disabled in wallet) - update status line to display liveness status Motivation and Context --- Allows us to gain visibility on the base latency of the transport without including overhead of the noise socket and yamux How Has This Been Tested? --- Manually * fix(core): dont request full non-tip block if block is empty (#4802) Description --- - checks for edge-case which prevents an unnecessary full candidate block request when block is empty. Motivation and Context --- A full block request for empty block is not necessary as we already have all the information required to construct the candidate block. This check was missing from the branch where the candidate block is not the next tip block. How Has This Been Tested? --- Co-authored-by: Martin Stefcek <35243812+Cifko@users.noreply.github.com> Co-authored-by: Hansie Odendaal <39146854+hansieodendaal@users.noreply.github.com> Co-authored-by: SW van Heerden <swvheerden@gmail.com> Co-authored-by: stringhandler <mikethetike@tari.com> Co-authored-by: CjS77 <CjS77@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Commits large (>= 100 ops) transaction in prune_to_height
Motivation and Context
When partially using block sync, then switching to pruned sync, the base node prunes blocks. This typically involves many
database operations and can easily exceed the growth maximum of the chain storage db.
This PR commits each time the transaction gets to >=100 operations.
Fixes #4760
How Has This Been Tested?
Manually: Partially block sync then switch to pruned sync