[GRPC] Implement SearchKernels #2245
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sdbondi
reviewed
Sep 18, 2020
|
agreed on the tests |
SWvheerden
reviewed
Sep 18, 2020
delta1
force-pushed
the
grpc-search-kernel
branch
2 times, most recently
from
3c5a2e8
to
98786bb
Compare
|
The supplied |
This commit adds the ability to search for blocks containing transaction kernels via GRPC.
delta1
force-pushed
the
grpc-search-kernel
branch
from
98786bb
to
69e58e3
Compare
delta1
pushed a commit
to delta1/blockchain-explorer-api
that referenced
this pull request
Sep 21, 2020
This commit implements the SearchKernels call from GRPC in the explorer API. Requires tari-project/tari#2245
stringhandler
approved these changes
Sep 22, 2020
stringhandler
added a commit
that referenced
this pull request
Oct 12, 2020
Major features since v0.5.5 * Propagate block hashes to all connected nodes (#2232) * Fixed difficulty calculation on Monero proof of work blocks (#2250) * Remove floating point calculation from achieved difficulty (#2190) * Introduce peer syncing and network discovery to improve a node's overall knowledge of the network (#2252) * Improve Closest broadcast strategy and use it for SAF transaction broadcast (#2290) * Implement variation of block percentage between proof of work algorithms (#2231) * Add integer based emission curve to replace floating point emission curve after testnet block 120000 (#2246) Minor changes since v0.5.5 * Allow client application to control when transaction broadcast protocols are restarted (#2221) * Expose kernel excess, public nonce, and signature via wallet FFI (#2230) * Fix possible inflation bug (#2251) * Only update mempool after block sync is complete (#2242) * Add the ability to search for blocks containing transaction kernels via GRPC (#2245) * Refactored to get the actual reason a tx failed to be logged (#2260) * Reduce ban time for sync peers (#2249) * Decouple Mempool and BlockchainDatabase (#2257) * Optimisation for MMR + clippy fixes (#2265) * Delay UTXO validation process until after initial blockchain sync has been achieved (#2243) * Implement tools to produce network graphs in memory net examples (#2258) * Add daily binary build for Tari Base Node with S3 upload (#2270) * Allow fee to be larger than the amount per configuration setting, to allow spending dust in some edge cases (#2262) * Fix Name collision in `types.proto` (#2266) * Add reason peer is banned (#2272, #2285) * Implement STXO validation in the wallet’s Output Manager Service (#2264) * Handle runtime error by removing an unnecessary dial-peer command, otherwise, when sometimes it is not successful, the code panics (#2282) * Add examples for Service Framework (#2276) * Fix target difficulty for block 1 on rincewind (#2284) * Cache orphan pool results in memory (#2287) * Refactor basenode/wallet/p2p initialization (#2280) * Add the Mempool RPC service (#2301) * Add `unban-all-peers` command (#2310) * Add list-banned-peers command to base node (#2312) * DHT RPC get_peers returns all peers and "on connect" mode for NetworkDiscovery (#2306) * Add recovery mode (#2214) * Add chain tip tracking of all peers (#2318) * Allow the chain meta service to react to banned peers by removing them from its internal collection (#2314) * Base node shuts down cleanly
delta1
pushed a commit
to delta1/blockchain-explorer-api
that referenced
this pull request
Oct 18, 2020
This commit implements the SearchKernels call from GRPC in the explorer API. Requires tari-project/tari#2245
CjS77
added a commit
that referenced
this pull request
Nov 9, 2020
* Preventing fee to be larger than the amount will also prevent spending dust in some cases. * Add reason peer needs to be banned * docs: add cargo test requirement to PR checklist * Remove sqrt from achieved difficulty - Change to using the product of Monero and Blake difficulties to determine the strongest chain. Previously this was the square root of the product. - Refactor chain strength out of validators, since it was not a validator - Add test blockchain for cleaner tests - Add limit to find_header_with_achieved_difficulty - Add test blockchain for cleaner tests - Add fetch all orphans * DHT network discovery (peer syncing) The PR introduces peer syncing and network discovery to improve a node's overall knowedge of the network. - Added a state machine that is responsible for syncing peers in rounds according to the current node state and previous round. - If the number of peers in the peer manager is below a threshold, the sync will attempt to sync (streamed by RPC) many peers at once. Once the threshold has been reached, a small number of peers are periodically requested from random peers. - Event notifies other components when a peer sync round has completed. - The DHT connectivity actor responds to this event by "refreshing" the neighbouring and random peer pools. - Created base for mocking out RPC calls - Created DHT mock - Added some basic tests for network discovery * Add migration to peer db * Quick fix for compile error caused by complex feature flag usage The MergeMiningError variant on the PowError enum is only compliled in for `base_node` and `transactions` features. * Add examples for Service Framework This PR provides a minimal example of how the service framework is used. * Implement STXO validation in the wallet’s Output Manager Service Currently the Output Manager Service in the wallet has 2 types of Output validation: 1. Validation Unspent outputs to see if they are still valid spendable outputs 2. Revalidate outputs marked as Invalid (from the previous validation This PR adds a third type of validation which checks the wallet’s Spent outputs to see if any of them have become Unspent since the last time the wallet started up. This is done because it is possible that during a Reorg a transaction that the wallet thought was successful could be rolled back. In that case currently the wallet will never restore those funds. This validation will be able to restore the outputs spent in such a situation to spendable, returning them to the wallets Available balance. A third kind TxoValidationType is added to the Txo Validation Protocol and it is started at the same time the Invalid UTXO revalidation protocol is started (i.e. on startup for the Base Node Wallet and when the Base Node Public key is set for the Wallet FFI library). This PR does not do anything to the transactions associated with the outputs so the Wallets will still show the affected transaction as successful but at least the funds will become available again. Fixing the presentation of the affect Transaction will be the focus of a subsequent PR. * Fix block sync * Compile failure fixes Fixes some compilation errors on development * fix for quotes in key in block chain * Update `test_transaction_cancellation_sqlite_db` to track down CI flakiness Added a check for the cancellation event which will help nail down where the test fails on CI. The test is not using a real comms stack and the operation its conducting is not time intensive so I cannot understand what is different on CI compared to local. * Handle runtime error by retrying dial peer. ``` ---- test::test_wallet_ffi stdout ---- thread 'test::test_wallet_ffi' panicked at 'called `Result::unwrap()` on an `Err` value: DialCancelled', base_layer\wallet_ffi\src\lib.rs:5812:13 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` * Fix target difficulty for block 1 on rincewind * add peer banned reason to display * cache orphan pool results in memory * Fix stack overflow on `test_store_and_forward_send_tx` test This test was overflowing its stack. Looks like running 3 wallets in a single thread is too big for the stack so to fix each wallet gets its own runtime. * Refactor basenode/wallet/p2p initialization The main thrust of ths PR is to allow services assembled using the StackBuilder to add protocols to comms before comms is spawned. These protocols will typically be RPC services. There existed a _chicken-and-egg_ problem between initializing the `StackBuilder` services and RPC services which may depend on those services, which in turn may depend on comms services. The solution in this PR is to initialize comms itself using the stack builder (`tari_p2p::P2pInitializer`) allowing objects to be registered on the `StackBuilder` before comms has spawned. A `SharedCommsContext` is provided to services that wish to register comms protocols and some comms components which are available during early intialization (e.g. PeerManager). - Created `P2pInitializer` and added it to the base node and wallet `StackBuilder` - Refactored generics out of the `CommsBuilder` and `UnspawnedCommsNode` - Greatly improved ergonomics of `StackBuilder` - Added `spawn_when_ready` convinience function to `ServiceInitializer` context that allows a service to wait for all handles to become ready before starting. - Comms no longer provides `ShutdownSignals` but has to have a shutdown signal provided to it. This allows for a single shutdown signal per app. - Cleanup of `tari_base_node::builder` module - Removed redundant `Shutdown` trigger. A single shutdown trigger is used throughout - Refactored tests and examples as necessary * Clean up all the remaining clippy warnings This PR cleans up all the remaining clips warnings and compiler warnings. From this point onwards any new warnings and clippy warnings are caused by you and your shame will be known to all! * Improve Closest broadcast strategy and use it for saf txn broadcast Closest broadcast strategy changed as follow: 1. A predefined number of peers is used instead of an arbitrary `n` (broadcast_factor) 1. A minimum of 2 peers are dialed that are closer to the message destination. This is in addition to closer peers with existing connections. - Transaction protocols use the new closest broadcast strategy - The orders of the store and forward layers are switched as forward depends on a message flag from store to determine how to propagate the message. * New commit, added specific log targets for specific messages * Improve TXO validation and broadcast and transaction protocol restart logic at base node startup. * Prevent tari_app_grpc rebuild on evvery build `tari_app_grpc` rebuilds each time because the tonic output is set to the `src` directory in `build.rs` and the generated file is written to on each build, causing rust to recompile. This PR builds to `OUT_DIR` which avoids the rebuild. * Improve display of Difficulty * Fix max block interval Adding trace to block sync * Mempool RPC service Added the Mempool RPC service. This service exposes: `get_stats`, `get_state`, `get_tx_state_by_excess_sig`, `submit_transaction`. The RPC service calls the `MempoolService` which performs the necessary DB calls. * Put in temp fix for monero big_endian-little_endian issue * Added algo split for pow Co-authored-by: Mike the Tike <stringhandler@gmail.com> Add doc comments * Added new regression tests for `Invalid coinbase in body` * DHT RPC get_peers returns all peers and "on connect" mode for NetworkDiscovery DHT RPC changes: - `get_peers` now returns all peers - `get_closest_peers` returns `n` peers closer to the given node id - Added `include_clients` bool flag to indicate if client nodes should be returned - New / updated unit tests Network discovery changes: - Added "on connect" mode. Initially, network discovery is "active" insofar as it actively connects to and seeks peers closer to the node's neighbourhood. Once no more neighbouring nodes can be found, the network discovery enters "on connect" mode which syncs whenever a (non-client) peer connects. - `DiscoveryReady` emits `OnConnectMode` event if no more peers are left to sync * Removed pretty table dev dependency pretty table was used in the memorynet exampe and is no longer needed * Allow for versions greater than 9 What happens when you want to push it to 11? * Integrate connectivity manager for Liveness service Liveness service drives the chain metadata service which in turn allows the base node state machine to determine if it needs to transition to sync mode. This change uses connectivity manager to select a random subset of active connections for each ping/pong round. Selecting active connections is extremely cheap and can be done often. Banned peers cannot be selected as connectivity immediately disconnects and removes the peer from active connections once banned. Liveness service changes: - Use connectivity manager to select a random subset of connections - Removed unused "monitored node" code P2pInitializer changes: - `SharedCommsContext` removed in favour of adding seperate comms components. This is to make testing easier so that there is no need to setup for e.g. a PeerManager if it's not needed by the component being tested. - Updated dependent components * Added `unban-all-peers` command `unban-all-peers` removes bans on all banned peers. * Add list-banned-peers command to base node * Adds in a trait for use by emission curve adds integer emission curve to consensus_manager Adds swap over point for hot swapping across points * Add sha3 as pow option Rename blake_miner to more accurately reflect role Update Tari_crypto to 0.9 Update Sha3 Pow to digest 0.9 * Remove banned or disconnected peers from chain metadata round This PR allows the chain meta service to react to banned peers by removing them from its internal collection. This ensures that newly banned peers cannot make it into a chain metadata round result, which in turn causes the base node state machine to attempt to sync from a banned peer. * Add meta_data field to peer * Add recovery mode Add recovery mode to re-add blocks to the db one by one Co-authored-by: Stan Bondi <sdbondi@users.noreply.github.com> * Add chain tip tracking of all peers * Add missing connecivity handle for state machine tests Fixes `tari_core::node_state_machine` tests by registering a missing handle in the test node builder. * Added DNS seed resolver module Added a DNS seed resolver that parses TXT records to obtain seed node details (PublicKey, Addresses) tuple. - A TXT record MAY contain text as per [IETF RFC-1464](https://tools.ietf.org/html/rfc1464) A valid Tari seed node TXT record: - MUST be in the format `[public key]::[[addresses]::+]`. The dellimiter is `::`. - `public key` MUST be a Tari public key represented as a hex string - `addresses` MUST be a valid multiaddr format - At least one address MUST be included - Every address part MUST NOT be empty - Multiple addresses MAY be included by appending `::[address]` * Detecting bootstrap based on accumulated difficulty comparison is not always successful, as peers can supply a different accumulated difficulty for the same block height as part of their blockchain metadata. This detection will always yield a positive result whenever the `Synchronizing missing blocks` are added to the database successfully the first time round. * Fix incorrect handle await in base_node state machine * Base node shuts down cleanly There were three issues preventing a "clean" shutdown from occurring: 1. GRPC server shutdown was not attached to the shutdown signal 1. If the watch channel closed (as it does on shutdown), the `start_transaction_protocols_and_txo_validation` would block its future's poll function (by going into an infinite loop), preventing the select from polling the shutdown signal. 1. There seems to be a bug with `Runtime::shutdown_timeout(n)` which causes it to wait `n` even if all tasks have completed. * Improvements for tie breaking comms, connection manager cleanup Deduplicate the connection state management within comms. Previously, `ConnectionManager` and `ConnectivityManager` both held `PeerConnection` handles. This was because the `Connectivity` concept came after the connection manager and was not refactored at the time. Now the connection state is contained only within `ConnectivityManager` and `ConnectionManager` is soley responsible for establishing outbound connections and handling inbound connections. Connections closed by tie breaking are now delayed by 2s and the `PeerConnectionWillClose` emitted to allow components to react to a connection that will close. The prime use case is the messaging protocol which can finish sending remaining messages and cut off further messages from being lost before the connection is closed. This should increase messaging reliability and possibly a fix for #2295 * Peer db migrations must be incremental Each step in the peer db migration must be incremental. Migration 2 broke this by adding metadata before v3 meaning that when v3 is run it cannot read the v3 peer. This PR fixes this by only adding banned_reason in v2 and metadata in v3. * This log message only needs to be written once to the log file, not with every non-bootstrapped event from the status watch. * Release v0.5.6, v0.16.6, v0.2.7 Major features since v0.5.5 * Propagate block hashes to all connected nodes (#2232) * Fixed difficulty calculation on Monero proof of work blocks (#2250) * Remove floating point calculation from achieved difficulty (#2190) * Introduce peer syncing and network discovery to improve a node's overall knowledge of the network (#2252) * Improve Closest broadcast strategy and use it for SAF transaction broadcast (#2290) * Implement variation of block percentage between proof of work algorithms (#2231) * Add integer based emission curve to replace floating point emission curve after testnet block 120000 (#2246) Minor changes since v0.5.5 * Allow client application to control when transaction broadcast protocols are restarted (#2221) * Expose kernel excess, public nonce, and signature via wallet FFI (#2230) * Fix possible inflation bug (#2251) * Only update mempool after block sync is complete (#2242) * Add the ability to search for blocks containing transaction kernels via GRPC (#2245) * Refactored to get the actual reason a tx failed to be logged (#2260) * Reduce ban time for sync peers (#2249) * Decouple Mempool and BlockchainDatabase (#2257) * Optimisation for MMR + clippy fixes (#2265) * Delay UTXO validation process until after initial blockchain sync has been achieved (#2243) * Implement tools to produce network graphs in memory net examples (#2258) * Add daily binary build for Tari Base Node with S3 upload (#2270) * Allow fee to be larger than the amount per configuration setting, to allow spending dust in some edge cases (#2262) * Fix Name collision in `types.proto` (#2266) * Add reason peer is banned (#2272, #2285) * Implement STXO validation in the wallet’s Output Manager Service (#2264) * Handle runtime error by removing an unnecessary dial-peer command, otherwise, when sometimes it is not successful, the code panics (#2282) * Add examples for Service Framework (#2276) * Fix target difficulty for block 1 on rincewind (#2284) * Cache orphan pool results in memory (#2287) * Refactor basenode/wallet/p2p initialization (#2280) * Add the Mempool RPC service (#2301) * Add `unban-all-peers` command (#2310) * Add list-banned-peers command to base node (#2312) * DHT RPC get_peers returns all peers and "on connect" mode for NetworkDiscovery (#2306) * Add recovery mode (#2214) * Add chain tip tracking of all peers (#2318) * Allow the chain meta service to react to banned peers by removing them from its internal collection (#2314) * Base node shuts down cleanly * Fix off by one height error in target difficulty Fixes a subtle bug where the target difficulty would be validated against the current height, before a block was added, instead of the height at the block * Fix target_diff Calculation * Revert "Fix off by one height error in target difficulty" This reverts commit 7939876. * Check that advertised difficulty matches actual chain difficulty Added a check after block sync that the difficulty advertised by peers is at least as high as the actual difficulty of the synced chain. If this check fails, the peer is temporarily banned. - Use effective pruned height to determine if the sync peer is able to supply blocks above a certain height. - Saved some sync peer clones when using `ban_sync_peer` * Release v0.5.7, v0.2.7 Major Fixes: * Fix target_diff Calculation * Console Wallet Automation for send and make-it-rain Add command parsing and executing logic to enable automation of the console wallet. By passing `--input path/to/file`, the lines of a text file can be parsed and executed as wallet commands. A single command can be executed by using `--command <command to execute>`. Currently only `get-balance`, `send-tari`, and `make-it-rain` are implemented. In script mode, all commands are parsed before being executed. If any commands fail to parse, the wallet exits immediately with an appropriate error message. New configuration options `command_send_wait_timeout` and `command_send_wait_stage` are added. These allow the command/script mode to wait until the sent transactions are detected at the configured "wait stage" within a given timeout value. Defaults to "Broadcast" and 300 second timeout. - "DirectSendOrSaf" - The transaction was initiated and was accepted via Direct Send or Store And Forward. - "Negotiated" - The recipient replied and the transaction was negotiated. - "Broadcast" - The transaction was broadcast to the base node mempool. - "Mined" - The transaction was successfully detected as mined on the blockchain. * Clean out orphan database at startup and/or sync Optionally clean out the orphan database at startup and before a block sync if the database has grown above a configurable threshold. Searching through the orphan database for every block to be added during a large block sync can be inefficient. Reorgs will still happen as per original design as the orphans db will only be cleared once before missing blocks are requested and added to the db. * Small refactor of `fetch_headers` `fetch_headers` takes a `start` and `end_inclusive` numbers instead of `Vec<u64>`. In all usages of `fetch_headers` the array was sequential. This limits the API to only what is needed and so allows more optimal implementations to exist (e.g. `SELECT * from headers where height >= x AND height <= y`). However, the main reason for this is not wanting to include this refactor in a future block sync implementation PR. * Expose latency for last call for client RPC connections Latency of the last call comes for free in RPC. This PR exposes the latency to users of the RPC client. * Add node.js integration tests * Refactoring to remove Arc<RwLock<>> from Wallet * Change Tor archive location for Windows install The Tor download URL was not pointing to the archives and had to be corrected for the Windows installer. * Update TUI Transaction listing and Contact management This PR firstly cleans up the TUI implementation a bit further from the initial refactor now that we are working with real data from the backend rather than dummy data. This involves a little bit or reorganisation. The main contributions of this PR are an updated StatefulList that can provide a windowed view into the list data and contact management. The update to the StatefulList implementation was required because the naive implementation put forward by tui-rs will parse and format the entire list for every draw operation. This was a problem on a wallet with thousands of transactions which made the UI very laggy. This PR adds functionality to the Stateful list which will produce a windowed view into the list data based on what items are selected and the provided height of the list widget that will be rendered so only the minimum required items are formatted for rendering. To avoid rewriting the whole List widget from tui-rs this is achieved by outputting the windowed slice of the data and a generated ListState object that the native tui-rs render method expects. The second part of this PR is adding the implementation to allow for managing Contacts in the Send/Receive tab. This involved completing the UI elements for listing contacts and adding, editing or deleting contacts. The initial UI logic for selecting a contact to send to, adding an amount and hitting Send Transaction is added though only the validation is done and not the actual sending yet. Errors are also displayed using an Error popup dialog. Tested: Tests are provided for the updates to the StatefulList and the windowed view. * Add shutdown triggers after wallet refactoring Somebody (me) broke the tests, wallet shutdowns were not triggered as they should be in the tests. Mistakes were made. * Migrate to dirs-next * Implement transaction sending and event based updating to TUI app This PR implements transaction sending in the TUI. In addition to actually sending the Tx the PR adds monitoring of the send progress and updates to the transaction status in the transaction lists as the process progresses. In order to allow events from the backend to update the state that is displayed by the TUI while also being able to update the state based on TUI input events the AppState has been refactored into a thread-safe pattern that manages its own RwLock and provides a rudimentary cached copy of the backend data for the front-end to be able to reference during every draw without cloning. The backend version of the data can be updated asynchronously by events from the backend and the frontend and only when the state has changed will the backend state be cloned to the cached copy. The update of the stale cache data is triggered by the frontend Tick event. The update of the stale cache is currently quite naive and there are a number of potential optimisations possible here to reduce the amount of data being cloned. To update the state based on asynchronous events coming from the Wallet backend this PR introduces an Event Monitor which is a spawned task that listened for events that mean that transaction data has been updated. When an event occurs for a specific TxId the monitor triggers an update of just that transaction’s state in the AppState backend to minimise database reads. For transaction sending a message field was added to the Send/Receive tab and popup dialogs were implemented to show success and errors that can be dismissed with a keypress. A “wait” dialog is also added that block further activity on that specific tab while a long operation is in progress (in this case the transaction send) but still allows the user to move and interact with the other tabs. In order to update the “wait” dialog with the current status of the send operation a Tokyo::watch channel is created and sent to a task which initiates the transaction send and then monitors the event bus for updates on its sending operation. The current stage of the operation is then broadcast on the watch channel for the wait dialog to monitor and use to update itself. Other small improvements in this PR: - Added the Excess Sig hex to the Detailed transaction view - Transaction Status now shows if the transaction has been Cancelled * v0.5.8, v0.2.9, v0.16.7 ## Major changes since v0.5.7 - Console Wallet Automation for send and make-it-rain [PR](#2291) - Added a check after block sync that the difficulty advertised by peers is at least as high as the actual difficulty of the synced chain. [PR](#2334) - Implement transaction sending in the TUI and introduce an asynchronous event monitor to update TUI state based on wallet backend events. [PR](#2351) ## Minor changes since v0.5.7 - Added a DNS seed resolver that parses TXT records to obtain seed node details (PublicKey, Addresses) tuple. This has not been integrated with any Tari applications yet. [PR](#2319) - Small refactor of `fetch_headers` [PR](#2338) - Change Tor archive location for Windows install [PR](#2345) - Add node.js integration tests [PR](#2339) - Expose latency for last call for client RPC connections [PR](#2340) - Update TUI Transaction listing and Contact management [PR](#2337) - Migrate the now deprecated and unmaintained dirs dependency to dirs-next [PR](#2350) * Fix transaction service bug with tx send/receive cancellation Fix a bug where the result of the cancellation oneshot in the send and receive protocols was not checked if it was Ok(()). This meant that on a shutdown when the sender side of that oneshot channel is dropped the receiver side resolves to a Err(_) and triggered an attempt to cancel the transaction. Generally the database was already broken down during the shutdown but depending on when stuff is dropped this could mistakenly cancel a pending transaction on wallet shutdown. * Display connected peers in TUI This PR adds monitoring of the Connectivity Manager event stream and display of the currently connected Peers in the Network Tab of the TUI. When a connectivity event is detected that would affect the connected peer list the list is updated in the backend. * Return errors from inbound base node handler Instead of swallowing errors, return them and log them * Set port + 1 for base node wallet in TCP mode This "temporary fix" only applies to TCP and SOCKS transport mode and was refactored out at some point. This PR re-adds it. * Fix features to build FFI * Remove DhtJoin reply and only send join when node comes online - Node will no longer reply to join messages as this method is deprecated in favor of #2306. Until #2306 is released to mobile, mobile wallets will fail to discover neighbouring nodes which include this PR. - Dht connectivity actor only sends a join when online. Join is more like an 'announce' that anounces to the neighbourhood of the node that the node is online. `Join` could be renamed to `Anounce` in future. - Only publish ConnectivityDegraded event only if the number of peers has changed - Updated _memorynet_ and a test * Add a client key-value store to the wallet and wallet_ffi The mobile team that uses the WalletFFI library requested a feature to be able to store arbitrary Key-Values in the Wallet’s database and make use of the Wallet’s encryption for these values. To enable this the functionality to set, get and clear client key-value pairs was added to the Wallet database as a new table. If an encryption cipher is set for the wallet the Values will be encrypted and decrypted by the database backend. Three FFI interface functions were added to the FFI library to allow a client to access this functionality. How it is tested: Tests are provided for the functions in the database backend and in the FFI library. * Revert change to fetch_mmr_nodes from #2357 The change to `fetch_mmr_nodes` introduced in #2357 caused the `request_and_response_fetch_mmr_node_and_count` test to timeout and fail. This commit reverts that changes, and makes the previous `debug` log into `warn` for greater visibility. This really depends on desired behaviour - the test indicates that an empty set should be returned if a peer requests "out of bounds" MMR node(s) - instead of an error result. Either this specific change introduced in PR #2357 needs to be reverted to maintain the test invariant, or the test itself needs to be changed. * cargo-fmt * Reduce base node service request timeout * Simplify and improve thread config - Remove `base_node.{network}.blocking_threads` config item - Add `base_node.{network}.max_threads` config item - If these configs are not set, the tokio defaults are used. i.e `max_threads = 512, core_threads = <number of cores>` - Update sample configs and docs * Removed floating point curve Add 60-40 split Add Ridcully network Add Ridcully genesis block Add Sha3 remove regression tests remove regression * remove genesis offset * Removed difficulty hack in monero_rx * remove linked_kernel field * Removed unused variable. * remove metadata field from kernel * Increase short term ban duration * Various improvements to network discovery - `get_peers` RPC call no longer sends the requesting peer back to itself. - Peer sync no longer accepts itself as a peer - "on connect" peer sync mode no longer syncs as aggressively (n=100, nodes only) - Propagate join messages using the "directed propagate" strategy. This strategy is similar to "closest connected" used previously except it stops propagating if there are no closer known nodes to the destination. - Readability improvements for `list-peers` command - Allow network discovery to enter "on connect" mode sooner even if many peer syncs don't work due to peers being unavailable. - Mark a peer as offline after 3 attempt(s) instead of 6 (`max_failures_mark_offline` = 1 and `max_dial_attempts` = 3) so that it takes less time to connect to available peers. - Set last seen for the dialled addresses when connecting. * Clean up errors in DHT pipeline Using anyhow instead of a custom PipelineError in DHT messaging pipelines. * V0.5.9, v0.2.10, v0.16.8 * Implement pending transaction cancellation in the TUI app This PR implements the ability to cancel a pending transaction from the TUI app. If you have pending transactions you can select a transaction and hit C to cancel it. The PR also adds in Confirmation dialogs to the TUI app and implements from the tx cancellation, transaction sending and contact deletion. This PR also fixes a bug in the WindowedListState logic where if an item in a list was deleted while the last element was selected it broke the List State. * Remove cyclic ban if online * Bug fix for offline connectivity state Connectivity should only be considered `Offline` if there are zero connections. Previously, the node could be considered offline if there if there were no _node_ connections, but some _client_ connections were active. This PR fixes that. * Fix for network discovery state machine spin loop If the node is unable to obtain many peers it continues to aggresively ask for peers. On a small localhost network this can cause high CPU usage as it repeatedly queries for peers in a state loop. This fix adds checks in the `Ready` state that allow a transition to `Idle` if: there are no sync peers, there are no sync peers other than the ones used last time, or there have been 10 rounds in a row where the node has not been able to obtain "enough" peers to enter "on connect" mode. * Add faucet kernel Add faucet value to constants * Add config option to disable wallet in Base Node This PR adds the ability to specify the “enable_wallet” boolean config option, which when set too false will not startup the Wallet instance embedded in the Base node. All commands that used the embedded wallet instance were updated to take this into account. This is done so that it is possible to run our Seed nodes without the Wallet instance which was cluttering the logs. * change accumulated difficulty to use the difficulty as just squared * Add ridcully config * Base node block sync RPC service RPC service for faster block sync. Allows streaming of full blocks, headers and a method for chain split detection. The client-side is next up. - Added BlockchainStateService. This service should eventually be the only service which provides a query interface to the blockchain database for other services. - Added BaseNodeSyncRpcService. This RPC service exposes `sync_blocks`, `sync_headers`, `get_header_by_height` and `find_chain_split`. These functions will be used in the improved block sync. - Added `NotFound` status to RpcStatus - Added `RpcStatus::log_internal_error()` convenience function to make dealing with misc errors in RPC handler functions easier. - Added `spawn_until_shutdown` method to the service initialiser context that spawns a task until the shutdown is triggered. This removes the need for implementers to worry about how to implement shutdown in initialisers. - Cleaned up some test helpers - Added unit tests for the Base node sync RPC handlers - Added unit tests for the BlockchainStateService Ref #2113 * Revert change to fetch blocks utxos txos (#2357) - The change to fetch_utxos, fetch_txos and fetch_blocks introduced in #2357 caused the those calls to timeout and fail in the client, as nothing was returned if matching data could not be found, contrary to the original design. This commit reverts that change, and renames the affected `NodeCommsRequest` functions so their intent can be clear. - The change to `NodeCommsRequest::fetch_mmr_nodes` was reverted in #2364; this PR also renames that function aptly. * Add more integration tests * Add max difficulty to enable quick testing * Add more integration tests * Add node wrapper for starting base node * Fix unit tests for `get_target_difficulty` - A recent PR increased parameters for `pub fn get_target_difficulty` without changing the unit tests. - Expanded `inbound_fetch_utxos` unit test to request only a subset plus 1 fake UTXOs. - Changed log level in transaction_service/service.rs to be consistent; not finding a UTXO is not an error in this case. * merge conflicts apply 0 as max difficulty on error. * Fix thread panic when getting non u128 total_diff * Add ridcully config as default fix data dir * add temp timestamp * Upgrade aes-gcm dependency in wallet * Remove rincewind from default toml files Remove all mention of rincewind * Remove crossbeam Crossbeam was used in the redundant pingpong example. This example has largely outlived it's usefullness, requires cursive which is not supported on windows, so it has been removed here. Criterion depends on rayon which depends on crossbeam. Criterion is now behind a feature flag in tari_mmr. * Update peer seed * Fix cargo fmt and clippy * Add automated stress test to base node wallet This change adds the ability to perform an automated stress test that uses a command file as input. The command file accommodates one optional `coin-split` command and one or more `make-it-rain` commands. The number and amount of test UTXOs will be assessed, and if found to be inadequate to perform the entire test, the `coin-split` will be attempted if requested. In the event of a `coin-split`, the wallet will be monitored for up to 30 minutes to determine if a sufficient number of UTXOs became available, and when that is achieved, the stress test will start. The `make-it-rain` commands can be scheduled to start at a specific time, thus the test can be started in advance. * increase comms version Fix clippy warnings remove rincewind sanity check fn * v0.6.0, v0.2.11, v0.16.9 * Combine validation code to use same function in pruned and archive mode. Fix bug in validation * Hook up mempool sync (Regression fix) Correctly hook up MempoolSyncProtocol to comms Also shortened the protocol identifier to conform with other protocol identifiers. * INFO log messages must be on a single line Logging policy is that DEBUG and higher messages fit on one line. Printing the entire block must be trace level. * Add configurable BN service request timeouts This change adds configurable base node service request timeouts, notably differentiating between general service requests, requests to fetch blocks for block sync and requests for a complete current UTXO set included in the blockchain. * Implement wallet base node service Adds a base node service to the wallet, to start tracking a connected base node's state. This will be required for the wallet to know the base node status, chain height, and when last the information was updated. The service has a configurable refresh interval to call `GetChainMetadata` on the base node. * Add exclusive file locks to Wallet, Chain and Peer db’s This PR adds an exclusive file locking mechanism to the wallet databases, the chain storage LMDB database and the Peer database. This is done to prevent two processes from simultaneously accessing this databases which can cause data corruption. This was done using the fs2 crate which contains a cross platform method for acquiring an OS level write lock on a file. For all these database the first step in initialization is to attempt to acquire a firelock on the lock file in the path where the database is located. If the file is not locked then a lock is acquired and the db initialisation proceeds as normal. If the file cannot be locked because the lock is already held by another instance an error is thrown. If the process holding the lock panics the OS level lock is lost so it is not needed to clean up the file explicitly for this method to work. During testing this mechanism revealed that the wallet services were not being explicitly shutdown properly, we also plumb the shutdown signals into the services and their sub protocols. The `test_store_and_forward_send_tx` test was being particularly flakey in the PR so I have opted to ignore it for now and come back and examine it properly in the future. How was this tested: Tests were written for the wallet and chain storage db’s but it was not trivial to test the peer_manager addition as it is located in the p2p initialiser. The code was manually tested on: Mac OS X Windows 10 Ubuntu 20.01 Android * Update README.md * [ci] split build and test tasks on circle-ci shaves some time off our circle ci, `test` is the longer task and now `build` can run independently. * Update README.md * Update README.md * Implement daemon-mode in tari_base_node * Review Comments * cargo-fmt * Fix wallet conversion error for a valid tx status Fixes a subtle error where a valid `TransactionStatus` of 5 was mistakenly reported as a conversion error. This caused the wallet to fail on startup if coinbase transactions were in teh database. The `TransactionStatus::try_from` method was missing the value. Also added some additional info to conversion error, in case it appears again it will be easier to narrow down. * Simplify automated stress test Simplified the automated stress test so that it will also automatically perform any required coin split beforehand. If not enough large valued UTXOs are available for the coin split, it will create what is possible and let the user know afterwards what the status is, without performing the stress test. * Fix preset config files Some information was outdated in these files and a large portion of the example file was deleted by accident recently. * merge conflicts Co-authored-by: Hansie Odendaal <pluto@tari.com> Co-authored-by: Mike Berry <mikethetike@tari.com> Co-authored-by: Byron Hambly <bizzle@tari.com> Co-authored-by: Stanimal <sdbondi@users.noreply.github.com> Co-authored-by: Philip Robinson <simian@tari.com> Co-authored-by: Keith Simon <41575888+Krakaw@users.noreply.github.com> Co-authored-by: StriderDM <51991544+StriderDM@users.noreply.github.com> Co-authored-by: CjS77 <CjS77@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This commit adds the ability to search for blocks containing transaction kernels via GRPC.
Motivation and Context
As part of #2201 and tari-project/block-explorer-frontend#107 we need a way to search for blocks containing specific transaction kernels.
How Has This Been Tested?
Existing tests pass, new GRPC call tested manually
Types of changes
Checklist:
developmentbranch.cargo-fmt --allbefore pushing.