/monero

Faster wallet refreshes #3716

Merged
merged 17 commits into from Jun 27, 2018
+757 −490

Conversation

Reviewers

@stoffu stoffu

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
5 participants
@moneromooo-monero @glv2 @stoffu @SamsungGalaxyPlayer @luigi1111
@moneromooo-monero
Contributor

moneromooo-monero commented Apr 27, 2018

This has random stuff in it, because I need random stuff that's not merged, so all this has to be plonked in this tree for me to hack on it. I'll rebase once those things are merged.

Warning:
If you run this, your blockchain DB will upgrade to v2, which means you won't be able to run any daemon that does not have this patch (I'll get it merged once 0.12.1.0 is out of the way).

Speeds up between 2x and 5x. Speed tests welcome.

If your wallet is not a mining wallet (ie, you never got any coinbase output on it), setting refresh-type=no-coinbase now gives a pretty good speed boost.

vtnerd's wallet asm patch will give more speedup, as the wallet is using 90% of its time in the fe API.
@glv2

This comment has been minimized.

Show comment
Hide comment
@glv2

glv2 May 21, 2018

Contributor

Here are some test results of a wallet scanning the whole blockchain:

computer

  • OS: GNU/Linux x86-64
  • CPU: Intel Core i7-3630QM @ 2.40GHz
  • Storage: Samsung SSD 850 PRO

master branch (4b728d7)

  • blocks: 1577488
  • time: 31.3 minutes

master branch + PR3716

  • blocks: 1577521
  • time: 8.6 minutes

master branch + PR3716 + refresh-type=no-coinbase

  • blocks: 1577526
  • time: 7.2 minutes
Contributor

glv2 commented May 21, 2018

Here are some test results of a wallet scanning the whole blockchain:

computer

  • OS: GNU/Linux x86-64
  • CPU: Intel Core i7-3630QM @ 2.40GHz
  • Storage: Samsung SSD 850 PRO

master branch (4b728d7)

  • blocks: 1577488
  • time: 31.3 minutes

master branch + PR3716

  • blocks: 1577521
  • time: 8.6 minutes

master branch + PR3716 + refresh-type=no-coinbase

  • blocks: 1577526
  • time: 7.2 minutes

@moneromooo-monero moneromooo-monero changed the title from [DO NOT MERGE] Faster wallet refreshes, plus random stuff to Faster wallet refreshes Jun 10, 2018

@moneromooo-monero

This comment has been minimized.

Show comment
Hide comment
@moneromooo-monero

moneromooo-monero Jun 10, 2018

Contributor

Now ready for review.
Contributor

moneromooo-monero commented Jun 10, 2018

Now ready for review.

@moneromooo-monero moneromooo-monero referenced this pull request Jun 10, 2018

Closed

Any special reason why Blockchain::get_transactions is template member function? #3967

@stoffu

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 11, 2018

Contributor

Computer:

  • OS: Windows 7 Professional, 64bit
  • CPU: Intel Core i7-5690X @ 3.00GHz
  • Storage: Samsung SSD 850 PRO

Master (25e7a7d):

  • optimize-coinbase -> 26m 46s (1592555 blocks)
  • no-coinbase -> 24m 24s (1592573 blocks)

PR-3716 (c4fda6d):

  • optimize-coinbase -> 5m 31s (1592557 blocks)
  • no-coinbase -> 4m 52s (1592578 blocks)

I can do two more tests on OSX machines with SSD and HDD.
Contributor

stoffu commented Jun 11, 2018 •
edited
Edited 1 time
  • @stoffu stoffu edited Jun 11, 2018 (most recent)
  • @stoffu stoffu created Jun 11, 2018

Computer:

  • OS: Windows 7 Professional, 64bit
  • CPU: Intel Core i7-5690X @ 3.00GHz
  • Storage: Samsung SSD 850 PRO

Master (25e7a7d):

  • optimize-coinbase -> 26m 46s (1592555 blocks)
  • no-coinbase -> 24m 24s (1592573 blocks)

PR-3716 (c4fda6d):

  • optimize-coinbase -> 5m 31s (1592557 blocks)
  • no-coinbase -> 4m 52s (1592578 blocks)

I can do two more tests on OSX machines with SSD and HDD.
@stoffu

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 11, 2018

Contributor

Note for testers: if you are to measure the time of scanning the entire blockchain from genesis, not only the restore height has to be set to 0, but also the wallet creation time must not be set to current. That is, if you create a new wallet and set refresh-from-block-height 0 and then rescan_bc, the wallet will NOT do the full scan of the blockchain (instead it skips old blocks based on timestamps). The correct way is to restore the wallet from seed and setting the restore height to 0.
Contributor

stoffu commented Jun 11, 2018 •
edited
Edited 2 times
  • @stoffu stoffu edited Jun 11, 2018 (most recent)
  • @stoffu stoffu edited Jun 11, 2018
  • @stoffu stoffu created Jun 11, 2018

Note for testers: if you are to measure the time of scanning the entire blockchain from genesis, not only the restore height has to be set to 0, but also the wallet creation time must not be set to current. That is, if you create a new wallet and set refresh-from-block-height 0 and then rescan_bc, the wallet will NOT do the full scan of the blockchain (instead it skips old blocks based on timestamps). The correct way is to restore the wallet from seed and setting the restore height to 0.
@stoffu

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 11, 2018

Contributor

Computer:

  • OS: Mac OSX 10.12.6
  • CPU: 2.9 GHz Intel Core i7
  • Storage: APPLE SSD SM2048L

Master:

  • optimize-coinbase -> 23m 12s (1592682 blocks)
  • no-coinbase -> 21m 31s (1592694 blocks)

PR-3716:

  • optimize-coinbase -> 11m 9s (1592697 blocks)
  • no-coinbase -> 9m 17s (1592702 blocks)
Contributor

stoffu commented Jun 11, 2018

Computer:

  • OS: Mac OSX 10.12.6
  • CPU: 2.9 GHz Intel Core i7
  • Storage: APPLE SSD SM2048L

Master:

  • optimize-coinbase -> 23m 12s (1592682 blocks)
  • no-coinbase -> 21m 31s (1592694 blocks)

PR-3716:

  • optimize-coinbase -> 11m 9s (1592697 blocks)
  • no-coinbase -> 9m 17s (1592702 blocks)
@stoffu

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 13, 2018

Contributor

Computer:

  • OS: macOS Sierra 10.12.6
  • CPU: 2.3 GHz Intel Core i7
  • Storage: APPLE HDD HTS541010A9E662

Master:

  • optimize-coinbase -> 1h 58m 24s (1592635 blocks)
  • no-coinbase -> 1h 56m 42s (1593176 blocks)

PR-3716:

  • optimize-coinbase -> 2h 0m 18s (1593251 blocks)
  • no-coinbase -> 1h 57m 50s (1593310 blocks)
Contributor

stoffu commented Jun 13, 2018

Computer:

  • OS: macOS Sierra 10.12.6
  • CPU: 2.3 GHz Intel Core i7
  • Storage: APPLE HDD HTS541010A9E662

Master:

  • optimize-coinbase -> 1h 58m 24s (1592635 blocks)
  • no-coinbase -> 1h 56m 42s (1593176 blocks)

PR-3716:

  • optimize-coinbase -> 2h 0m 18s (1593251 blocks)
  • no-coinbase -> 1h 57m 50s (1593310 blocks)
@moneromooo-monero

This comment has been minimized.

Show comment
Hide comment
@moneromooo-monero

moneromooo-monero Jun 13, 2018

Contributor

That one seems odd since HDD means I/O is a bottleneck, and the new version reads much less from the database, since it stores prunable and unprunable data separately... Nevermind, the new db format patch is already merged in master.
Contributor

moneromooo-monero commented Jun 13, 2018 •
edited
Edited 1 time
  • @moneromooo-monero moneromooo-monero edited Jun 13, 2018 (most recent)
  • @moneromooo-monero moneromooo-monero created Jun 13, 2018

That one seems odd since HDD means I/O is a bottleneck, and the new version reads much less from the database, since it stores prunable and unprunable data separately... Nevermind, the new db format patch is already merged in master.
@SamsungGalaxyPlayer

This comment has been minimized.

Show comment
Hide comment
@SamsungGalaxyPlayer

SamsungGalaxyPlayer Jun 14, 2018

Computer: Purism Librem 13
OS: PureOS GNU/Linux 8
Processor: Intel Core i7-6500U CPU @ 2.50 GHz
OS Hard Drive: 120 GB SATA SSD (unknown type)
Attached Hard Drive: 1TB Samsung 970 EVO

Master:

  • optimize-coinbase
    • 970 EVO -> 0h 28m 01s; 1598776 blocks (951.1 blocks per second)
  • no-coinbase
    • 970 EVO -> 0h 24m 20s; 1598750 blocks (1095.0 blocks per second)

PR-3716:

  • optimize-coinbase
    • 970 EVO -> 0h 20m 43s (limited by CPU); 1596508 blocks (1284.4 blocks per second)
  • no-coinbase
    • 970 EVO -> 0h 17m 25s (limited by CPU); 1597181 blocks (1528.4 blocks per second)

SamsungGalaxyPlayer commented Jun 14, 2018 •
edited
Edited 5 times
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer edited Jun 20, 2018 (most recent)
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer edited Jun 17, 2018
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer edited Jun 17, 2018
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer edited Jun 17, 2018
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer edited Jun 16, 2018
  • @SamsungGalaxyPlayer SamsungGalaxyPlayer created Jun 14, 2018

Computer: Purism Librem 13
OS: PureOS GNU/Linux 8
Processor: Intel Core i7-6500U CPU @ 2.50 GHz
OS Hard Drive: 120 GB SATA SSD (unknown type)
Attached Hard Drive: 1TB Samsung 970 EVO

Master:

  • optimize-coinbase
    • 970 EVO -> 0h 28m 01s; 1598776 blocks (951.1 blocks per second)
  • no-coinbase
    • 970 EVO -> 0h 24m 20s; 1598750 blocks (1095.0 blocks per second)

PR-3716:

  • optimize-coinbase
    • 970 EVO -> 0h 20m 43s (limited by CPU); 1596508 blocks (1284.4 blocks per second)
  • no-coinbase
    • 970 EVO -> 0h 17m 25s (limited by CPU); 1597181 blocks (1528.4 blocks per second)
src/rpc/core_rpc_server.cpp
@@ -593,7 +597,7 @@ namespace cryptonote
return true;
}
sorted_txs.push_back(std::move(txs.front()));
- txs.pop_front();
+ txs.erase(txs.begin());

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Isn't this an unneeded burden which can be avoided by having a counter and doing sorted_txs.push_back(std::move(txs[counter]));?
@stoffu

stoffu Jun 26, 2018

Contributor

Isn't this an unneeded burden which can be avoided by having a counter and doing sorted_txs.push_back(std::move(txs[counter]));?
src/wallet/wallet2.cpp
{
error = false;
try
{
drop_from_short_history(short_chain_history, 3);
- // prepend the last 3 blocks, should be enough to guard against a block or two's reorg

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

This comment seems still relevant. Why remove?
@stoffu

stoffu Jun 26, 2018

Contributor

This comment seems still relevant. Why remove?
src/wallet/wallet2.cpp
+ hw::reset_mode rst(hwdev);
+
+ hwdev.set_mode(hw::device::TRANSACTION_PARSE);
+ if (!hwdev.generate_key_derivation(tx_pub_key, keys.m_view_secret_key, derivation))

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Since the line 1133(before) tx_pub_key = pub_key_field.pub_key; was deleted, where does tx_pub_key get its value?
@stoffu

stoffu Jun 26, 2018

Contributor

Since the line 1133(before) tx_pub_key = pub_key_field.pub_key; was deleted, where does tx_pub_key get its value?
src/wallet/wallet2.cpp
+ hwdev.set_mode(hw::device::TRANSACTION_PARSE);
+ const cryptonote::account_keys &keys = m_account.get_keys();
+
+ auto gender = [&](wallet2::is_out_data &iod) {

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Had to wonder for a while what this name means :D
@stoffu

stoffu Jun 26, 2018

Contributor

Had to wonder for a while what this name means :D
src/wallet/wallet2.h
@@ -1282,6 +1295,8 @@ namespace tools
std::string m_ring_database;
bool m_ring_history_saved;
std::unique_ptr<ringdb> m_ringdb;
+ boost::optional<crypto::chacha_key> m_ringdb_key;
+ unsigned int m_ringdb_key_refs;

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Unused?
@stoffu

stoffu Jun 26, 2018

Contributor

Unused?
src/wallet/wallet2.cpp
@@ -5791,6 +5810,7 @@ bool wallet2::get_ring(const crypto::chacha_key &key, const crypto::key_image &k
bool wallet2::get_rings(const crypto::hash &txid, std::vector<std::pair<crypto::key_image, std::vector<uint64_t>>> &outs)
{
+ key_ref kref(*this);

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Is this needed?
@stoffu

stoffu Jun 26, 2018

Contributor

Is this needed?
src/wallet/wallet2.cpp
@@ -2332,6 +2333,8 @@ bool wallet2::delete_address_book_row(std::size_t row_id) {
//----------------------------------------------------------------------------------------------------
void wallet2::refresh(uint64_t start_height, uint64_t & blocks_fetched, bool& received_money)
{
+ key_ref kref(*this);

This comment has been minimized.

Show comment
Hide comment
@stoffu

stoffu Jun 26, 2018

Contributor

Is this needed?
@stoffu

stoffu Jun 26, 2018

Contributor

Is this needed?

This comment has been minimized.

Show comment
Hide comment
@moneromooo-monero

moneromooo-monero Jun 26, 2018

Contributor

No, but it speeds up saving rings from incoming txes if there are several.
@moneromooo-monero

moneromooo-monero Jun 26, 2018

Contributor

No, but it speeds up saving rings from incoming txes if there are several.

moneromooo-monero added some commits Apr 15, 2018

@moneromooo-monero
wallet2: use std::move on containers where appropriate
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
a2b557f
@moneromooo-monero
rpc: add option to skip coinbase info in get_blocks.bin
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
6795bd0
@moneromooo-monero
rpc: sanity check on number of txes in a block
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
209ec96
@moneromooo-monero
replace std::list with std::vector on some hot paths 
also use reserve where appropriate
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
ed2c81e
@moneromooo-monero
rpc: rework to avoid repeated calculations in get_blocks.bin
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
a830db2
@moneromooo-monero
wallet: don't process miner tx if we're refreshing with no-coinbase
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
57ea902
@moneromooo-monero
wallet2: simplify/speedup process_blocks
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
31a895e
@moneromooo-monero
wallet2: parse blocks in the RPC thread, not the processing thread 
Processing typically is the bottleneck
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
ba8331c
@moneromooo-monero
wallet2: remove useless device locking
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
f7f1917
@moneromooo-monero
wallet2: avoid duplicate parsing of tx extra
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
41be339
@moneromooo-monero
wallet2: avoid re-parsing blocks when refreshing in a loop
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
f025ae9
@moneromooo-monero
blockchain: simplify/speedup handle_get_objects
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
ef2cb63
@moneromooo-monero
wallet2: speedup refresh 
key derivation and checking for incoming outputs are threaded
in batch before adding blocks to the local blockchain. Other
minor bits and bobs are also cached.
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
dcfd299
@moneromooo-monero
wallet2: remove old blockchain size cache hack
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
5d3e702
@moneromooo-monero
wallet2: remove unneeded divisions
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
2704624
@moneromooo-monero
threadpool: allow leaf functions to run concurrently 
Decrease the number of worker threads by one to account
for the fact the calling thread acts as a worker thread now
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
2771a18
@moneromooo-monero
wallet2: cache ringdb key while refreshing 
Speeds up syncing with a lot of outgoing outputs as key generation
runs Cryptonight.
Unverified
This user has not uploaded their public key yet.
GPG key ID: 686F07454D6CEFC3 Learn about signing commits
Loading status checks…
0e4c7d0
@stoffu

stoffu approved these changes Jun 27, 2018
View changes

@luigi1111 luigi1111 merged commit 0e4c7d0 into monero-project:master Jun 27, 2018
0 of 7 checks passed

0 of 7 checks passed

buildbot/monero-linux-armv7 Build done.
Details
buildbot/monero-static-osx-10.11 Build done.
Details
buildbot/monero-static-osx-10.13 Build done.
Details
buildbot/monero-static-ubuntu-amd64 Build done.
Details
buildbot/monero-static-ubuntu-i686 Build done.
Details
buildbot/monero-static-win32 Build done.
Details
buildbot/monero-static-win64 Build done.
Details

luigi1111 added a commit that referenced this pull request Jun 27, 2018

@luigi1111
Merge pull request #3716 
a2b557f 6795bd0 209ec96 ed2c81e a830db2 57ea902 31a895e ba8331c f7f1917 41be339 f025ae9 ef2cb63 dcfd299 5d3e702 2704624 2771a18 0e4c7d0 (moneromooo-monero)
Verified
This commit was signed with a verified signature.
@luigi1111 luigi1111
GPG key ID: F4ACA0183641E010 Learn about signing commits
Loading status checks…
a844844
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment