wallet: let Listwalletdir do not iterate through our blocksdata. #19419

Saibato · 2020-06-30T07:12:56Z

Fix: While fiddling with wallets, I noticed that if datadir == walltetsdir an unsynced node can become quite unresponsive.

if no "wallets" dir is in datadir ( i.e. exiting old node )
we would have iterate trough our own node files
to find wallets, that consumes time and could cause an unresponsive node.

DrahtBot · 2020-06-30T13:00:19Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

rpc, wallet: Expose database format in getwalletinfo #20125 (rpc, wallet: Expose database format in getwalletinfo by promag)
wallet: bugfix; if datadir has a trailing '/' listwalletdir would strip lead char of walletname #19933 (wallet: bugfix; if datadir has a trailing '/' listwalletdir would strip lead char of walletname by Saibato)
Bugfix: Wallet: Soft-fail exceptions within ListWalletDir file checks #19502 (Bugfix: Wallet: Soft-fail exceptions within ListWalletDir file checks by luke-jr)
wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077 (wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets by achow101)
Fix crashes and infinite loop in ListWalletDir() #18095 (Fix crashes and infinite loop in ListWalletDir() by uhliksk)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

promag · 2020-07-01T09:40:11Z

No strong opinion regarding excluding some paths. Current patch needs to be updated (format) and maybe change implementation to use a list of ignore/exclude paths.

This can improve GUI by reducing freezes but it's not the right fix for that. When ListWalletDir is triggered from the GUI, it should be called on a background thread. Alternatively we could consider caching ListWalletDir result and invalidate it when wallets are created or loaded.

Saibato · 2020-07-01T12:22:24Z

No strong opinion regarding excluding some paths. Current patch needs to be updated (format) and maybe change implementation to use a list of ignore/exclude paths.

This can improve GUI by reducing freezes but it's not the right fix for that. When ListWalletDir is triggered from the GUI, it should be called on a background thread. Alternatively we could consider caching ListWalletDir result and invalidate it when wallets are created or loaded.

yup, ugly quick fix and background thread seams reasonable.. but fix proves the point and lets be honest why should we waste time and risk locks, when traverse trough known node dir?.. . also other GUI freezes should be tackled.
Before moon scale there is a long list of things in my backlog, that need to be fixed before bam..
.
There might be soon real users who use the gui, and lets be real, gui is bitcoin for them.

promag · 2020-07-01T21:40:32Z

src/wallet/walletutil.cpp

+    fs::path sub_path = GetDataDir();
+
+    bool we_have_wallets_dir = (wallet_dir != sub_path);
+    const fs::path fblocks = GetBlocksDir();


Instead of multiple variables, you can declare something like const std::vector ignore_paths{ GetBlocksDir(), sub_path / "testnet3", ... } and then compare against this.

yup 👍 , btw strange how long those bare strings survived refactor, still no global #define BLOCKS_DIR or TESTNET_STR etc.

Sjors

Concept ACK

This seems orthogonal to using a background thread. We shouldn't be looking for wallets inside the blocks and chainstate dir even in the background.

Sjors · 2020-07-09T18:23:12Z

src/wallet/walletutil.cpp

-        const fs::path path = it->path().string().substr(offset);
+        // We dont want to iterate trough those special node dirs
+        if (it->status().type() == fs::directory_file) {
+            skip_path = false;


better to define skip_path here

Note to other reviewers, the enum file_type: https://www.boost.org/doc/libs/1_66_0/libs/filesystem/doc/reference.html#Enum-file_type

src/wallet/walletutil.cpp

Sjors · 2020-07-09T18:28:53Z

src/wallet/walletutil.cpp

+                it.disable_recursion_pending();
+                continue;
+            }
+        } else it.disable_recursion_pending();


nit: just use brackets and newline

Also, I think we should follow symlinks? symlink_file

Following symlinks could create an infinite loop.

src/wallet/walletutil.cpp

luke-jr · 2020-07-12T18:59:17Z

src/wallet/walletutil.cpp

+            skip_path = false;
+            for(const auto& fpath: ignore_paths) {
+                if (!skip_path && it->path().string().find(fpath.string()) == 0) {
+                    skip_path = true;
+                }
+            }
+            if (skip_path) {
+                it.disable_recursion_pending();
+                continue;
+            }


Suggested change

skip_path = false;

for(const auto& fpath: ignore_paths) {

if (!skip_path && it->path().string().find(fpath.string()) == 0) {

skip_path = true;

}

}

if (skip_path) {

it.disable_recursion_pending();

continue;

}

if (ignore_paths.count(it->path()) {

it.disable_recursion_pending();

continue;

}

👍 Though I choose
if (std::count(ignore_paths.begin(), ignore_paths.end(), it->path())) {
not sure ignore_paths has a count?

It will once changed to an unordered_set as suggested

meshcollider · 2020-08-13T21:47:05Z

Concept ACK, looks good to me

achow101 · 2020-08-14T16:41:48Z

Concept ACK, please squash your commits.

src/wallet/walletutil.cpp

hebasto · 2020-10-04T14:04:21Z

src/wallet/walletutil.cpp

            }
+        } catch (const std::exception& e) {


I personally prefer "throwing" versions of the Boost.Filesystem API. But I don't think that combining "throwing" and "non-throwing" versions of the Boost.Filesystem API in the same function is a good design.

with catch boost::exception
and dynamic_cast<std.exception i run with a recursive path in an non catched
EXCEPTION: N5boost10filesystem16filesystem_errorE
so unless u have a hint, i leave it that way, for now?

When WalletDir == DataDir we would have iterate trough our own node files to find wallets, that consumes time and could cause an unresponsive node.

Saibato · 2020-10-08T10:59:06Z

Rebased da6eb3c and addressed reviews.

hebasto

Approach ACK da6eb3c.

Mind adding #include <algorithm> in separate header group?

src/wallet/walletutil.cpp

Designed to be used on ranges of elements.

Saibato · 2020-10-12T12:20:45Z

Add #include <algorithm> to header.

Tyi @hebasto while dev this i encountered and ugly wallet and filesystem edge case. that is only partly fixed by #18095 but with fewer changes and final here and in #19502 u might reach out to @luke-jr or @laanwj for more info.

luke-jr · 2020-10-13T19:27:06Z

src/wallet/walletutil.cpp

+                    // software will never create these files but will allow them to be
+                    // opened in a shared database environment for backwards compatibility.
+                    // Add it to the list of available wallets.
+                    paths.emplace_back(it->path().filename());


Why did you change this?

its not changed just the correct indent.

If u mean the whole logic itself, its because there where an other wallet access issue not mentioned here fixed also, could be no longer valid since we had some grave changes in master, i have to check, but since the PR its closed yet, it makes sense to let all as is and open two new and or wait until its reopened and then help out with the review.

luke-jr · 2020-10-13T19:32:42Z

src/wallet/walletutil.h

@@ -5,6 +5,7 @@
 #ifndef BITCOIN_WALLET_WALLETUTIL_H
 #define BITCOIN_WALLET_WALLETUTIL_H

+#include <algorithm>


Why did you add this here? If it's needed in the .cpp file, put it there...

;was keen to see some compiler warnings, i though that was the hint from @hebasto to add <algorithm> addressed and changed back to .cpp that other bug we can fix later.

When WalletDir == DataDir we would have iterate trough our own node files to find wallets, that consumes time and could cause an unresponsive node. Diff-minimised Github-Pull: bitcoin#19419 Rebased-From: da6eb3c

luke-jr · 2020-10-13T19:45:15Z

Please make these changes:

DrahtBot · 2020-10-15T11:09:41Z

🐙 This pull request conflicts with the target branch and needs rebase.

_{Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".}

laanwj · 2020-10-16T04:57:12Z

Marking this as up for grabs. The concept to not iterate needlessly into directories that should not contain wallets definitely makes sense, but I think this makes unnecessary changes that are not well motivated.

Saibato · 2020-10-17T20:58:52Z

Rebased and addressed 5a8f19e 4fb4fb7 an comments, changed by this a lot so pls review again

@luke-jr could not use ur code by merge and lost by fixup ur commit entrys, i hope u don;t mind.

https://github.com/Saibato/bitcoin/tree/wallet_351

Since this is closed here the rebase and changes are just in my tree, since github does not update closed pr tyi

Saibato · 2020-10-25T10:08:26Z

to find wallets, that consumes time and could cause an unresponsive node.

@laanwj "... cause an unresponsive node" in commit message was a hint, this PR is not about to fix something and must not be merged anyway but to point to some lines of code and its implications to gave others a chance to have a look in this and guess by themselfs.
So my question is now, will have we have a fix and sure its not done if we just skip the boost traversing?

in #19502 and #18095 we see some progress, but will that be enough and why have I to write this even, if it is to u so obvious?

luke-jr · 2020-11-13T14:23:21Z

@Saibato GitHub won't let anyone reopen this since your branch changed. Can you git push ??? 3f9cc0cd736e79c563dee3573f0dab01dac10622:wallet_351 -f temporarily (you can re-push your new version after it's reopened; put your repo name in place of ??? of course).

Don't forget to rebase!

Also: You need to include <set> instead of <algorithm> ;)

When WalletDir == DataDir we would have iterate trough our own node files to find wallets, that consumes time and could cause an unresponsive node. Github-Pull: bitcoin#19419 Rebased-From: c730c7a

luke-jr · 2020-11-13T14:28:53Z

Here's a rebase for later: master...luke-jr:listwalletdir_skip_data

Saibato · 2020-11-27T09:11:31Z

@luke-jr tyi was the last week just counting my sats and had not looked in this, thx. for the reopen hint.
btw look now why some wallets are missing when using the 0.21 rc1 tag, maybe that was the reason i changed also the scan for wallets logic here , but just forgot to mention that sggh...

When WalletDir == DataDir we would have iterate trough our own node files to find wallets, that consumes time and could cause an unresponsive node. Github-Pull: bitcoin#19419 Rebased-From: c730c7a

fanquake added the Wallet label Jun 30, 2020

Saibato force-pushed the wallet_351 branch 2 times, most recently from 530c3a2 to 24d2044 Compare June 30, 2020 08:24

Saibato changed the title ~~wallet: Listwallets do not iterate trough our blocksdata.~~ wallet: Listwalletdir do not iterate trough our blocksdata. Jun 30, 2020

Saibato mentioned this pull request Jun 30, 2020

[Open Wallet] click freeze system, when in sync node bitcoin-core/gui#16

Open

DrahtBot mentioned this pull request Jun 30, 2020

Fix crashes and infinite loop in ListWalletDir() #18095

Closed

Saibato changed the title ~~wallet: Listwalletdir do not iterate trough our blocksdata.~~ [WIP} wallet: Listwalletdir do not iterate trough our blocksdata. Jul 1, 2020

Saibato changed the title ~~[WIP} wallet: Listwalletdir do not iterate trough our blocksdata.~~ [WIP] wallet: Listwalletdir do not iterate trough our blocksdata. Jul 1, 2020

Saibato force-pushed the wallet_351 branch 3 times, most recently from 55e5460 to 6e5b87a Compare July 1, 2020 17:58

DrahtBot mentioned this pull request Jul 1, 2020

wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077

Merged

Saibato force-pushed the wallet_351 branch 3 times, most recently from 0a58f3f to 3d69e4c Compare July 1, 2020 21:17

promag reviewed Jul 1, 2020

View reviewed changes

Saibato force-pushed the wallet_351 branch 2 times, most recently from c49d922 to 4aa7bc5 Compare July 2, 2020 16:02

Saibato changed the title ~~[WIP] wallet: Listwalletdir do not iterate trough our blocksdata.~~ wallet: let Listwalletdir do not iterate trough our blocksdata. Jul 2, 2020

DrahtBot mentioned this pull request Jul 5, 2020

Replace boost::filesystem with std::filesystem #19245

Closed

5 tasks

Sjors reviewed Jul 9, 2020

View reviewed changes

luke-jr reviewed Jul 12, 2020

View reviewed changes

DrahtBot mentioned this pull request Jul 13, 2020

Bugfix: Wallet: Soft-fail exceptions within ListWalletDir file checks #19502

Merged

Saibato force-pushed the wallet_351 branch from 4aa7bc5 to 320e94e Compare July 13, 2020 13:33

Saibato mentioned this pull request Jul 14, 2020

Per-Peer Message Capture #19509

Merged

hebasto reviewed Oct 4, 2020

View reviewed changes

src/wallet/walletutil.cpp Outdated Show resolved Hide resolved

hebasto reviewed Oct 4, 2020

View reviewed changes

src/wallet/walletutil.cpp Outdated Show resolved Hide resolved

hebasto reviewed Oct 4, 2020

View reviewed changes

Saibato mentioned this pull request Oct 5, 2020

wallet: Can't Access System Volume Information #20081

Closed

wallet: let ListWalletDir do not iterate trough our blocksdata.

da6eb3c

When WalletDir == DataDir we would have iterate trough our own node files to find wallets, that consumes time and could cause an unresponsive node.

Saibato force-pushed the wallet_351 branch from caa4184 to da6eb3c Compare October 8, 2020 10:50

Saibato changed the title ~~wallet: let Listwalletdir do not iterate trough our blocksdata.~~ wallet: let Listwalletdir do not iterate through our blocksdata. Oct 8, 2020

hebasto requested changes Oct 11, 2020

View reviewed changes

src/wallet/walletutil.cpp Show resolved Hide resolved

DrahtBot mentioned this pull request Oct 12, 2020

rpc, wallet: Expose database format in getwalletinfo #20125

Merged

add #include <algorithm> to src/wallet/walletutil.h

3f9cc0c

Designed to be used on ranges of elements.

luke-jr reviewed Oct 13, 2020

View reviewed changes

DrahtBot added the Needs rebase label Oct 15, 2020

laanwj added the Up for grabs label Oct 16, 2020

laanwj closed this Oct 16, 2020

Saibato mentioned this pull request Feb 10, 2021

UI Hangs when running as std user #21136

Closed

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wallet: let Listwalletdir do not iterate through our blocksdata. #19419

wallet: let Listwalletdir do not iterate through our blocksdata. #19419

Saibato commented Jun 30, 2020

DrahtBot commented Jun 30, 2020 •

edited

Loading

promag commented Jul 1, 2020

Saibato commented Jul 1, 2020

promag Jul 1, 2020

Saibato Jul 2, 2020

Sjors left a comment

Sjors Jul 9, 2020

Sjors Jul 9, 2020

luke-jr Jul 12, 2020

luke-jr Jul 12, 2020

Saibato Jul 13, 2020 •

edited

Loading

luke-jr Jul 13, 2020

meshcollider commented Aug 13, 2020

achow101 commented Aug 14, 2020

hebasto Oct 4, 2020

Saibato Oct 8, 2020

Saibato commented Oct 8, 2020

hebasto left a comment

Saibato commented Oct 12, 2020

luke-jr Oct 13, 2020

Saibato Oct 17, 2020

Saibato Oct 17, 2020

luke-jr Oct 13, 2020

Saibato Oct 17, 2020

luke-jr commented Oct 13, 2020

DrahtBot commented Oct 15, 2020

laanwj commented Oct 16, 2020

Saibato commented Oct 17, 2020 •

edited

Loading

Saibato commented Oct 25, 2020

luke-jr commented Nov 13, 2020 •

edited

Loading

luke-jr commented Nov 13, 2020

Saibato commented Nov 27, 2020

wallet: let Listwalletdir do not iterate through our blocksdata. #19419

wallet: let Listwalletdir do not iterate through our blocksdata. #19419

Conversation

Saibato commented Jun 30, 2020

DrahtBot commented Jun 30, 2020 • edited Loading

Conflicts

promag commented Jul 1, 2020

Saibato commented Jul 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saibato Jul 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meshcollider commented Aug 13, 2020

achow101 commented Aug 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saibato commented Oct 8, 2020

hebasto left a comment

Choose a reason for hiding this comment

Saibato commented Oct 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luke-jr commented Oct 13, 2020

DrahtBot commented Oct 15, 2020

laanwj commented Oct 16, 2020

Saibato commented Oct 17, 2020 • edited Loading

Saibato commented Oct 25, 2020

luke-jr commented Nov 13, 2020 • edited Loading

luke-jr commented Nov 13, 2020

Saibato commented Nov 27, 2020

DrahtBot commented Jun 30, 2020 •

edited

Loading

Saibato Jul 13, 2020 •

edited

Loading

Saibato commented Oct 17, 2020 •

edited

Loading

luke-jr commented Nov 13, 2020 •

edited

Loading