Faster wallet refreshes #3716

moneromooo-monero · 2018-04-27T16:08:23Z

This has random stuff in it, because I need random stuff that's not merged, so all this has to be plonked in this tree for me to hack on it. I'll rebase once those things are merged.

Warning:
If you run this, your blockchain DB will upgrade to v2, which means you won't be able to run any daemon that does not have this patch (I'll get it merged once 0.12.1.0 is out of the way).

Speeds up between 2x and 5x. Speed tests welcome.

If your wallet is not a mining wallet (ie, you never got any coinbase output on it), setting refresh-type=no-coinbase now gives a pretty good speed boost.

vtnerd's wallet asm patch will give more speedup, as the wallet is using 90% of its time in the fe API.

glv2 · 2018-05-21T09:08:35Z

Here are some test results of a wallet scanning the whole blockchain:

computer

OS: GNU/Linux x86-64
CPU: Intel Core i7-3630QM @ 2.40GHz
Storage: Samsung SSD 850 PRO

master branch (4b728d7)

blocks: 1577488
time: 31.3 minutes

master branch + PR3716

blocks: 1577521
time: 8.6 minutes

master branch + PR3716 + refresh-type=no-coinbase

blocks: 1577526
time: 7.2 minutes

moneromooo-monero · 2018-06-10T10:45:03Z

Now ready for review.

stoffu · 2018-06-11T09:21:27Z

Computer:

OS: Windows 7 Professional, 64bit
CPU: Intel Core i7-5690X @ 3.00GHz
Storage: Samsung SSD 850 PRO

Master (25e7a7d):

optimize-coinbase -> 26m 46s (1592555 blocks)
no-coinbase -> 24m 24s (1592573 blocks)

PR-3716 (c4fda6d):

optimize-coinbase -> 5m 31s (1592557 blocks)
no-coinbase -> 4m 52s (1592578 blocks)

I can do two more tests on OSX machines with SSD and HDD.

stoffu · 2018-06-11T09:39:49Z

Note for testers: if you are to measure the time of scanning the entire blockchain from genesis, not only the restore height has to be set to 0, but also the wallet creation time must not be set to current. That is, if you create a new wallet and set refresh-from-block-height 0 and then rescan_bc, the wallet will NOT do the full scan of the blockchain (instead it skips old blocks based on timestamps). The correct way is to restore the wallet from seed and setting the restore height to 0.

stoffu · 2018-06-11T14:07:50Z

Computer:

OS: Mac OSX 10.12.6
CPU: 2.9 GHz Intel Core i7
Storage: APPLE SSD SM2048L

Master:

optimize-coinbase -> 23m 12s (1592682 blocks)
no-coinbase -> 21m 31s (1592694 blocks)

PR-3716:

optimize-coinbase -> 11m 9s (1592697 blocks)
no-coinbase -> 9m 17s (1592702 blocks)

stoffu · 2018-06-13T09:41:05Z

Computer:

OS: macOS Sierra 10.12.6
CPU: 2.3 GHz Intel Core i7
Storage: APPLE HDD HTS541010A9E662

Master:

optimize-coinbase -> 1h 58m 24s (1592635 blocks)
no-coinbase -> 1h 56m 42s (1593176 blocks)

PR-3716:

optimize-coinbase -> 2h 0m 18s (1593251 blocks)
no-coinbase -> 1h 57m 50s (1593310 blocks)

moneromooo-monero · 2018-06-13T13:35:34Z

~~That one seems odd since HDD means I/O is a bottleneck, and the new version reads much less from the database, since it stores prunable and unprunable data separately...~~ Nevermind, the new db format patch is already merged in master.

SamsungGalaxyPlayer · 2018-06-14T16:01:14Z

Computer: Purism Librem 13
OS: PureOS GNU/Linux 8
Processor: Intel Core i7-6500U CPU @ 2.50 GHz
OS Hard Drive: 120 GB SATA SSD (unknown type)
Attached Hard Drive: 1TB Samsung 970 EVO

Master:

optimize-coinbase
- 970 EVO -> 0h 28m 01s; 1598776 blocks (951.1 blocks per second)
no-coinbase
- 970 EVO -> 0h 24m 20s; 1598750 blocks (1095.0 blocks per second)

PR-3716:

optimize-coinbase
- 970 EVO -> 0h 20m 43s (limited by CPU); 1596508 blocks (1284.4 blocks per second)
no-coinbase
- 970 EVO -> 0h 17m 25s (limited by CPU); 1597181 blocks (1528.4 blocks per second)

stoffu · 2018-06-26T04:03:47Z

src/rpc/core_rpc_server.cpp

@@ -593,7 +597,7 @@ namespace cryptonote
              return true;
            }
            sorted_txs.push_back(std::move(txs.front()));
-            txs.pop_front();
+            txs.erase(txs.begin());


Isn't this an unneeded burden which can be avoided by having a counter and doing sorted_txs.push_back(std::move(txs[counter]));?

stoffu · 2018-06-26T05:06:06Z

src/wallet/wallet2.cpp

 {
  error = false;

  try
  {
    drop_from_short_history(short_chain_history, 3);

-    // prepend the last 3 blocks, should be enough to guard against a block or two's reorg


This comment seems still relevant. Why remove?

stoffu · 2018-06-26T05:33:59Z

src/wallet/wallet2.cpp

+      hw::reset_mode rst(hwdev);
+
+      hwdev.set_mode(hw::device::TRANSACTION_PARSE);
+      if (!hwdev.generate_key_derivation(tx_pub_key, keys.m_view_secret_key, derivation))


Since the line 1133(before) tx_pub_key = pub_key_field.pub_key; was deleted, where does tx_pub_key get its value?

stoffu · 2018-06-26T05:49:29Z

src/wallet/wallet2.cpp

+  hwdev.set_mode(hw::device::TRANSACTION_PARSE);
+  const cryptonote::account_keys &keys = m_account.get_keys();
+
+  auto gender = [&](wallet2::is_out_data &iod) {


Had to wonder for a while what this name means :D

stoffu · 2018-06-26T06:09:37Z

src/wallet/wallet2.h

@@ -1282,6 +1295,8 @@ namespace tools
    std::string m_ring_database;
    bool m_ring_history_saved;
    std::unique_ptr<ringdb> m_ringdb;
+    boost::optional<crypto::chacha_key> m_ringdb_key;
+    unsigned int m_ringdb_key_refs;


stoffu · 2018-06-26T06:12:04Z

src/wallet/wallet2.cpp

@@ -5791,6 +5810,7 @@ bool wallet2::get_ring(const crypto::chacha_key &key, const crypto::key_image &k

 bool wallet2::get_rings(const crypto::hash &txid, std::vector<std::pair<crypto::key_image, std::vector<uint64_t>>> &outs)
 {
+  key_ref kref(*this);


Is this needed?

stoffu · 2018-06-26T06:16:11Z

src/wallet/wallet2.cpp

@@ -2332,6 +2333,8 @@ bool wallet2::delete_address_book_row(std::size_t row_id) {
 //----------------------------------------------------------------------------------------------------
 void wallet2::refresh(uint64_t start_height, uint64_t & blocks_fetched, bool& received_money)
 {
+  key_ref kref(*this);


Is this needed?

No, but it speeds up saving rings from incoming txes if there are several.

also use reserve where appropriate

Processing typically is the bottleneck

key derivation and checking for incoming outputs are threaded in batch before adding blocks to the local blockchain. Other minor bits and bobs are also cached.

Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now

Speeds up syncing with a lot of outgoing outputs as key generation runs Cryptonight.

a2b557f 6795bd0 209ec96 ed2c81e a830db2 57ea902 31a895e ba8331c f7f1917 41be339 f025ae9 ef2cb63 dcfd299 5d3e702 2704624 2771a18 0e4c7d0 (moneromooo-monero)

Motivated by https://monero.stackexchange.com/questions/10483 Some exchanges appear to have customized the wallet software in an inappropriate way, making the tx extra field partially unreadable. PR monero-project#3716 changed the wallet behavior disallowing such partially valid tx extra. An example tx reported by the user is e87c675a85f34ecac58a8846613d25062f1813e1023c552b705afad32b972c38 where the normal tx pubkey appears again with the aditional tx pubkeys tag `04` which is inappropriate.

moneromooo-monero force-pushed the mswip2 branch from 28f42cd to 9875f12 Compare April 28, 2018 11:35

moneromooo-monero force-pushed the mswip2 branch from 9875f12 to c4fda6d Compare June 10, 2018 10:44

moneromooo-monero changed the title ~~[DO NOT MERGE] Faster wallet refreshes, plus random stuff~~ Faster wallet refreshes Jun 10, 2018

moneromooo-monero mentioned this pull request Jun 10, 2018

Any special reason why Blockchain::get_transactions is template member function? #3967

Closed

moneromooo-monero force-pushed the mswip2 branch from c4fda6d to 286d4ff Compare June 20, 2018 20:56

stoffu reviewed Jun 26, 2018

View reviewed changes

moneromooo-monero added 16 commits June 26, 2018 22:14

wallet2: use std::move on containers where appropriate

a2b557f

rpc: add option to skip coinbase info in get_blocks.bin

6795bd0

rpc: sanity check on number of txes in a block

209ec96

replace std::list with std::vector on some hot paths

ed2c81e

also use reserve where appropriate

rpc: rework to avoid repeated calculations in get_blocks.bin

a830db2

wallet: don't process miner tx if we're refreshing with no-coinbase

57ea902

wallet2: simplify/speedup process_blocks

31a895e

wallet2: parse blocks in the RPC thread, not the processing thread

ba8331c

Processing typically is the bottleneck

wallet2: remove useless device locking

f7f1917

wallet2: avoid duplicate parsing of tx extra

41be339

wallet2: avoid re-parsing blocks when refreshing in a loop

f025ae9

blockchain: simplify/speedup handle_get_objects

ef2cb63

wallet2: speedup refresh

dcfd299

key derivation and checking for incoming outputs are threaded in batch before adding blocks to the local blockchain. Other minor bits and bobs are also cached.

wallet2: remove old blockchain size cache hack

5d3e702

wallet2: remove unneeded divisions

2704624

threadpool: allow leaf functions to run concurrently

2771a18

Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now

wallet2: cache ringdb key while refreshing

0e4c7d0

Speeds up syncing with a lot of outgoing outputs as key generation runs Cryptonight.

moneromooo-monero force-pushed the mswip2 branch from 286d4ff to 0e4c7d0 Compare June 26, 2018 21:28

stoffu approved these changes Jun 27, 2018

View reviewed changes

luigi1111 merged commit 0e4c7d0 into monero-project:master Jun 27, 2018

stoffu mentioned this pull request Aug 6, 2018

launch aeon-wallet-rpc failed! aeonix/aeon#48

Closed

stoffu mentioned this pull request Nov 6, 2018

Upstream merge 2 aeonix/aeon#77

Merged

stoffu mentioned this pull request Nov 22, 2018

wallet2: resume processing when tx extra is partially broken #4885

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Faster wallet refreshes #3716

Faster wallet refreshes #3716

moneromooo-monero commented Apr 27, 2018

glv2 commented May 21, 2018

moneromooo-monero commented Jun 10, 2018

stoffu commented Jun 11, 2018 •

edited

stoffu commented Jun 11, 2018 •

edited

stoffu commented Jun 11, 2018

stoffu commented Jun 13, 2018

moneromooo-monero commented Jun 13, 2018 •

edited

SamsungGalaxyPlayer commented Jun 14, 2018 •

edited

stoffu Jun 26, 2018

stoffu Jun 26, 2018

stoffu Jun 26, 2018

stoffu Jun 26, 2018

stoffu Jun 26, 2018

stoffu Jun 26, 2018

stoffu Jun 26, 2018

moneromooo-monero Jun 26, 2018

Faster wallet refreshes #3716

Faster wallet refreshes #3716

Conversation

moneromooo-monero commented Apr 27, 2018

glv2 commented May 21, 2018

moneromooo-monero commented Jun 10, 2018

stoffu commented Jun 11, 2018 • edited

stoffu commented Jun 11, 2018 • edited

stoffu commented Jun 11, 2018

stoffu commented Jun 13, 2018

moneromooo-monero commented Jun 13, 2018 • edited

SamsungGalaxyPlayer commented Jun 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stoffu commented Jun 11, 2018 •

edited

stoffu commented Jun 11, 2018 •

edited

moneromooo-monero commented Jun 13, 2018 •

edited

SamsungGalaxyPlayer commented Jun 14, 2018 •

edited