Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wallet] Reopen CDBEnv after encryption instead of shutting down #12493

Merged
merged 4 commits into from Sep 14, 2018

Conversation

@achow101
Copy link
Member

@achow101 achow101 commented Feb 20, 2018

This is the replacement for #11678 which implements @ryanofsky's suggestion.

Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted here. This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote this script to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

cc @ryanofsky

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch to b56e661 Feb 20, 2018
}
// Close the individual Db's
for (std::string filename : filenames) {
CloseDb(filename);

This comment has been minimized.

@ryanofsky

ryanofsky Feb 21, 2018
Contributor

This is a great start! I don't think think it is thread safe yet, though, because in parallel with this thread, a block connected notification could be coming in, or another another RPC could be being made that is using one of the Db* or DbEnv* pointers that this closes.

I think all you need to do to make this thread safe is wait for the mapFileUseCount entries to go to zero. You could do this with a condition variable. For example, if you added a std::condition_variable m_cv_in_use; member to CDBEnv you could trigger it in CDB::Close:

m_cv_in_use.notify_all()

and wait for it at the top of CDBEnv::ReloadDbEnv:

WAIT_LOCK(cs_db, lock);
m_cv_in_use.wait(lock, [this](){
    for (count : mapFileUseCount) {
        if (count.second > 0) return false;
    }
    return true;
});

This is one possible approach. Other approaches may be simpler or better. One drawback of this approach is that if there are a lot of background writes happening in different wallets, ReloadDbEnv could get starved out waiting for all wallets to be simultaneously not in use.

@achow101
Copy link
Member Author

@achow101 achow101 commented Feb 21, 2018

@ryanofsky I've implemented something which is basically what you described. It seems to work, although I'm not sure how to test the thread safe-ness of it. Let me know what you think.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

It seems to work, although I'm not sure how to test the thread safe-ness of it.

It'd be a little tricky to test the threadsafeness in a unit test. I think you'd need to insert hooks that would allow the test to block threads at particular points. If you want to test in a more ad-hoc way, though, I think you could do this by inserting a sleep in CDB::Write and starting bitcoin with two wallets loaded. You could then call an RPC that triggers a write in one wallet and hits the sleep. Then call an RPC on the other wallet to trigger ReloadDbEnv. If the condition variable is working correctly, the ReloadDbEnv call in the second wallet should get stuck until the sleep is over and the first wallet completes its write.

src/wallet/db.h Outdated
@@ -35,10 +35,11 @@ class CDBEnv
void EnvShutdown();

public:
mutable CCriticalSection cs_db;
std::recursive_mutex cs_db;

This comment has been minimized.

@ryanofsky

ryanofsky Feb 21, 2018
Contributor

In commit "Replace cs_db with a recursive mutex use with a conditional_variable_any"

cs_db is actually already a recursive mutex (it inherits from std::recursive_mutex). I think you could just leave it unchanged, and continue using the LOCK macro instead of std::lock_guard everywhere. This would make the commit smaller & simpler.

This comment has been minimized.

@achow101

achow101 Feb 21, 2018
Author Member

I tried that, but it kept giving me a compiler error.

This comment has been minimized.

@ryanofsky

ryanofsky Feb 21, 2018
Contributor

#12493 (comment)

I tried that, but it kept giving me a compiler error.

What is the compiler error? The following compiles for me: c271de2 (fetchable with git fetch https://github.com/ryanofsky/bitcoin pr/lreload).

I didn't do any real testing with it but it passes python & unit tests.

This comment has been minimized.

@achow101

achow101 Feb 21, 2018
Author Member

Ah, I was doing something with the macros and CCriticalSection. It was an error about not having an unlock method.

I'll test this out, but from testing I just did with the current code, it looks like it does work, so this should work too.

This comment has been minimized.

@achow101

achow101 Feb 21, 2018
Author Member

@ryanofsky I've used your commit.

src/wallet/db.cpp Outdated
@@ -511,6 +512,15 @@ void CDBEnv::CloseDb(const std::string& strFile)

void CDBEnv::ReloadDbEnv()
{
// Make sure that no Db's are in use
std::unique_lock<std::recursive_mutex> lock(cs_db);

This comment has been minimized.

@ryanofsky

ryanofsky Feb 21, 2018
Contributor

In commit "Replace cs_db with a recursive mutex use with a conditional_variable_any"

Would be good to add an AssertLockNotHeld(cs_db); above this, since it would be a bug if the mutex were held recursively when this was called (wait could hang if the lock wasn't released).

This comment has been minimized.

@achow101

achow101 Feb 21, 2018
Author Member

Done

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch 2 times, most recently Feb 21, 2018
Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK 437d2dd56f81a6258243aca6ed137cbca9255b32

@@ -690,6 +690,11 @@ bool CWallet::EncryptWallet(const SecureString& strWalletPassphrase)
// bits of the unencrypted private key in slack space in the database file.
dbw->Rewrite();

// BDB seems to have a bad habit of writing old data into

This comment has been minimized.

@ryanofsky

ryanofsky Feb 22, 2018
Contributor

In commit "After encrypting the wallet, reload the database environment"

Delayed flushing seems more like a legitimate design tradeoff than a bad habit. Maybe just say something like "flush and reload the database environment here to clear out any data in memory that might be left behind after the rewrite above."

This comment has been minimized.

@achow101

achow101 Feb 22, 2018
Author Member

The comment was just copied over from rpcwallet.cpp.

@@ -690,6 +690,11 @@ bool CWallet::EncryptWallet(const SecureString& strWalletPassphrase)
// bits of the unencrypted private key in slack space in the database file.
dbw->Rewrite();

This comment has been minimized.

@ryanofsky

ryanofsky Feb 22, 2018
Contributor

In commit "After encrypting the wallet, reload the database environment"

Looking at the CDB::Rewrite implementation I noticed that it already has a while loop that sleeps until mapFileUseCount (for just one wallet) is 0. You could take advantage of this if you wanted by tweaking the while loop there to wait for all use counts in the environment to be 0 and then calling ReloadDbEnv inside CDB::Rewrite. This would be a simplification since it would let you get rid of the new condition variable, and it would also make the new waiting code more consistent with previous code.

This is just a suggestion, though. Your current implementation seems fine, too.

This comment has been minimized.

@achow101

achow101 Feb 22, 2018
Author Member

I don't want to do anything to CDB::Rewrite since it is used in places other than encryptwallet.

This comment has been minimized.

@ryanofsky

ryanofsky Feb 22, 2018
Contributor

#12493 (comment)

I don't want to do anything to CDB::Rewrite since it is used in places other than encryptwallet

I didn't realize Rewrite was called other places, and the current approach does seem fine. But if you did want to unify the UseCount waiting logic, I think you could do it in a pretty clean way by adding a bool reload_env or similar option to CDB::Rewrite. I think in terms code clarity, having the option would actually be an improvement over always reloading.

This comment has been minimized.

@achow101

achow101 Feb 22, 2018
Author Member

I think I'll leave it as it is now.

Copy link
Member

@promag promag left a comment

Tested ACK 437d2dd.

Tested with multiple wallets and with different -walletdir.

src/qt/askpassphrasedialog.cpp Outdated
@@ -123,7 +123,7 @@ void AskPassphraseDialog::accept()
{
QMessageBox::warning(this, tr("Wallet encrypted"),
"<qt>" +
tr("%1 will close now to finish the encryption process. "
tr("Your wallet is now encrypted. "
"Remember that encrypting your wallet cannot fully protect "
"your bitcoins from being stolen by malware infecting your computer.").arg(tr(PACKAGE_NAME)) +

This comment has been minimized.

@promag

promag Feb 26, 2018
Member

Remove .arg() since %1 was removed.

This comment has been minimized.

@achow101

achow101 Feb 26, 2018
Author Member

Done

@@ -123,7 +123,7 @@ void AskPassphraseDialog::accept()
{
QMessageBox::warning(this, tr("Wallet encrypted"),

This comment has been minimized.

@promag

promag Feb 26, 2018
Member

QMessageBox::information?

This comment has been minimized.

@achow101

achow101 Feb 26, 2018
Author Member

I don't think it needs to change.

});

std::vector<std::string> filenames;
for (auto it : mapDb) {

This comment has been minimized.

@promag

promag Feb 26, 2018
Member

Remove filenames and just iterate once, something like:

while (!mapDb.empty()) {
  std::string filename = mapDb.begin().first;
  ...
}

This comment has been minimized.

@achow101

achow101 Feb 26, 2018
Author Member

This doesn't work (causes test failures with encryptwallet). I think it's becauese mapDb is modified by CloseDb too.

src/wallet/db.cpp Outdated

void CWalletDBWrapper::ReloadDbEnv()
{
if(!IsDummy()) {

This comment has been minimized.

@promag

promag Feb 26, 2018
Member

Nit, space after if.

This comment has been minimized.

@achow101

achow101 Feb 26, 2018
Author Member

Done

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch Feb 26, 2018
@achow101
Copy link
Member Author

@achow101 achow101 commented Mar 28, 2018

Rebased

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch 2 times, most recently Mar 28, 2018
@achow101
Copy link
Member Author

@achow101 achow101 commented Mar 31, 2018

Fixed travis and rebased again.

@achow101
Copy link
Member Author

@achow101 achow101 commented Apr 12, 2018

Rebased

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch 2 times, most recently Apr 12, 2018
@achow101
Copy link
Member Author

@achow101 achow101 commented May 5, 2018

Review beg

@achow101
Copy link
Member Author

@achow101 achow101 commented Jun 12, 2018

Review Beg?

src/wallet/db.cpp Outdated
@@ -570,6 +571,16 @@ void BerkeleyEnvironment::CloseDb(const std::string& strFile)

void BerkeleyEnvironment::ReloadDbEnv()
{
// Make sure that no Db's are in use
AssertLockNotHeld(cs_db);
std::unique_lock<std::recursive_mutex> lock(cs_db);

This comment has been minimized.

@sipa

sipa Jul 14, 2018
Member

It would be slightly cleaner to use std::unique_lock<CCriticalSection> here.

Also, please squash this into the first commit - currently your PR introduces a bug and then fixes it later.

We should probably also work on making the sync.h primitives more C++11 compatible, but let's leave that for later.

This comment has been minimized.

@achow101

achow101 Jul 14, 2018
Author Member

Done

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch Jul 14, 2018
Copy link
Member

@sipa sipa left a comment

utACK 22c1c367062c960a28ca9b3b463b6ab6d96ebd02, just nits.

src/wallet/db.cpp Outdated
@@ -551,6 +551,7 @@ void BerkeleyBatch::Close()
{
LOCK(cs_db);
--env->mapFileUseCount[strFile];
env->m_db_in_use.notify_all();

This comment has been minimized.

@sipa

sipa Jul 14, 2018
Member

Performance nit: notifying while not holding the lock is slightly faster (currently you'll wake the waiting thread, which then immediately needs to sleep again because the lock is still held).

This comment has been minimized.

@achow101

achow101 Jul 14, 2018
Author Member

Done

src/wallet/db.cpp Outdated
filenames.push_back(it.first);
}
// Close the individual Db's
for (std::string filename : filenames) {

This comment has been minimized.

@sipa

sipa Jul 14, 2018
Member

Nit: const std::string& filename.

This comment has been minimized.

@achow101

achow101 Jul 14, 2018
Author Member

Done

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch Jul 14, 2018
@sipa
Copy link
Member

@sipa sipa commented Jul 14, 2018

utACK 06f17267ddafa089f093cfffc181d8c834d346e7

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch Jul 14, 2018
@achow101
Copy link
Member Author

@achow101 achow101 commented Jul 14, 2018

Rebased and added a commit to fix the silent merge conflict (test failure).

@achow101 achow101 force-pushed the achow101:dbenv-reopen branch Jul 14, 2018
@sipa
Copy link
Member

@sipa sipa commented Jul 14, 2018

utACK bdad9c40595c1e4959ebce1b55894060b2d7c146

Copy link
Member

@instagibbs instagibbs left a comment

utACK

provided the locking behavior is as I expect, where the lock will be acquired without fail by the .wait. The documentation isn't super clear and I'm definitely not an expert.

promag added a commit to promag/bitcoin that referenced this pull request Dec 6, 2018
0.17 branch doesn't include bitcoin#12493 which changed encryptwallet behavior. For that
reason the test is adjusted.

Github-Pull: bitcoin#14453
Rebased-From: 8907df9
promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019
Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac
promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019
Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5
promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019
Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461
promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019
Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a
@promag promag mentioned this pull request Mar 11, 2019
laanwj added a commit that referenced this pull request Mar 20, 2019
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - #12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - #14350 Add WalletLocation class
   - #14320 [bugfix] wallet: Fix duplicate fileid detection
   - #14552 wallet: detecting duplicate wallet by comparing the db filename.
   - #11911 Free BerkeleyEnvironment instances when not in use
   - #15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14
sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019
Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac
sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019
Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5
sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019
Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461
sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019
Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a
sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14
uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019
Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac
uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019
Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5
uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019
Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461
uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019
Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a
Rishabh42 added a commit to Rishabh42/syscoin that referenced this pull request Apr 22, 2019
* Revert "more dbg"

This reverts commit c114420.

* Revert "more dbg"

This reverts commit 20d7fa1.

* Revert "more dbg"

This reverts commit f2a4621.

* Revert "logging"

This reverts commit 067d813.

* Revert "more dbg"

This reverts commit f2a4621.

* remove dbg

* remove dbg

* add 500 byte check in rpc prepare

* wip on new ban policy

* rpc: Make unloadwallet wait for complete wallet unload

Github-Pull: bitcoin#14941
Rebased-From: c37851d

* Remove errant past from walletcreatefundedpsbt for nLocktime replaceability

Github-Pull: bitcoin#15213
Rebased-From: 85f0ca9

* Update zmq to 4.3.1

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6250

Github-Pull: bitcoin#15188
Rebased-From: 3046e5f

* [0.17] [Doc] Backport release note about PSBT doc

* add pingretries to cmasternode

* start masternode preenabled and switch to enabled on ping check from broadcast

* assetallocationbalance rpc call

* have aamany vs just aa rpc

* display retries

* reset ping retries once successful ping and also ensure mn is in winnerslist

if mn is in winnerlist + 6 signatures a ping is accepted, if mn is in winners list between 0 and 6 signatures it wont be valid but it will not DOS ban peer, if  you are outside of the list entirely you should ban the peer.

The idea is that once you get into the winners list and have enough signatures its ok to start pinging. This avoids the race condition of pingers being banned and being accepted because thre is a delay between having 0 and 6 signatures, outside of the winners list

* change to 2 to test

* remove unused vars

* accept first ping on new broadcast, dont check winners list for that

* dont expire if lastPing is empty

* move enabled after checks and fix fSentinelIsCurrent

* move force check after enabled set

* reset retries only if mn enabled

* remove ping expired/expired states, either your enabled or not

* remove calls to isexpired and issentinelpingexpired

* idb check removed from miner

* rmove status print

* remove sentinel ping activation checks

* comment code out TODO put it back after test

* dont send ping unless in winners list with 6 sigs

* comment out ping

* enabled by default until theres a ping

* add sentinel ping expired state back

* refactor

* move preenabled check above

* relay if sentinel expired

* brackets around logic

* remove dbg

* ping from sentinelping for test

* some dbg

* allow pinging during preenabled state

* Revert "some dbg"

This reverts commit e6b06fe.

* dont increment npingretries if ourmn

* fix sentinelping call, call check every 10 min

* fix common lib

* move corewrite to server

* run mnchecks before calls that use the information

* is mn is not synced set to preenabled

* ensure we lock main first in istransactionvalid

* move pingretries to masternode info

* set pingretries to mnb list

* store ping retries in setmnping

* check retries from broadcast

* use references not copy

* some helpful debug

* dbg

* remove dbg

* allow force check on masternodemanager::Check

pass false by default but pass in true in the rpc calls and istransactionvalid. Don't update npingretries if passed in true, only when its not forced do we update that variable to avoid adding it multiple times

* set sentinel valid by default on broadcast

* set max retries to 60 and maintenance to every min

* only check ping in winners list if winners list is synced

* check winnerslist on sending ping only if list is synced

* serialize ping retries

* manage local ping retries

* check entire list is synced not just mn list

* startup preenabled for 10s

* remove dbg

* send ping only if not preenabled and is synced

* only check masternodes if they are in the winnerslist (or not enabled)

* output for not being in winners list in check

* use adjusted time instead of lastping for preenabled check

* use function already created IsBroadcastedWithin

* future sig time check fix

* give atleast MASTERNODE_SENTINEL_PING_MAX_SECONDS/2 seconds to propogate

* update to MASTERNODE_MIN_MNB_SECONDS

* also adjust broadcast check

* flip logic

* fix switching from pre to enabled state

* const references

* remove mn cache's

* update proto and hrp

* set sigs to 6 again

* update get to 1.8.23

* Updated relayer to 1.0.4

* geth default data dir in syscoin data dir

* pull out boot nodes

* Updated relayer to 1.0.4

* add net specific to data dir

* remove non-witness enforcement in mempool

this was taken out before but we missed this check, non-witness should be allowed

* update min proto

* skip check on empty winners list

* show active time from gettime if no ping has been processed yet

* check for multiple instances of daemon properly

* fix result check on pidof syscoin call result

* check asset consensus checks during mempool validation

* fix tests

* rmv no wallet build

* fix build

* remove graph from lint

* Removed graph and regex from libboost

* fix build

* fix build

* fix build

* typo

* remove main lock on istransactionvalid

the main lock in masternode::check is avoided from istransactionvalid because fForce=true and we only check isoutpoint which locks main in check() is fForce==false

* compile

* expire masternode sanity check if ping retries >= max

* compile

* dbg for getblockchaininfo

* remove potential for deadlock scenarios

* remove retry param

* process mncheck first time on startup

* better process killing

* implement our asset index spec part 1

* Bumped Masternode major version

* finish asset index spec

* add listassetindexassets RPC

* 3.8gb min

* flipped logic suppoed to end firsttime

* add block hash txid mapping to checksyscoininputs

* asset index configuration parameters

* check index on listasset rpc call

* check assets rpc call aswell

* check for assetindex before using it to init db

* default to page 0

* compile

* read txids regardless if doesnt exist

* compile

* write to asset page when changing or initializing

* remove double booking of asset index, refactor into standard functions and reuse

fix missing assetindextxid in the mintsyscoin indexing routine

* fix listassetallocation logic

* re-org txid structures so they don't double up during forks

* order LIFO on results

* refactor reorg of asset index

1) only erase payload
2) when scanning if payload doesn't exist, continue
this means that some pages may not be full (that is the tradeoff but its a performance based tradeoff). The alternatve is to remove TXIDs but that is more intensive because you have to remove for every receiver/sender which requires one to scan back pages to find txid.

* add erase txid on disconnect

we still have to rollback txid's on disconnect, may be slow but for now its required

* remove unneeded code

* remove potential deadlocking scenario with updatelastpaid/check

* remove potential deadlocks from mn rpc calls

updatelastpaid has cs_main locking inside but locks another cs before that, so we have to ensure we lock main first

* minor code cleanup

* dec page if empty

* ensure page doesnt overflow

* skip check() if not in winners list instead of enabled

also ping retries only if not preenabled to avoid cases where preenabled mn's start to count up since we are allowing non enabled checks now to happen in check() we need to ensure that the mn is in the right state before ping retries is incremented.

* fix while loop issue by changing page to int64_t

* only increase ping retries if votes >= 6 and on winners list, otherwise check every mn every 60 seconds

* remove def versions not needed

* fixed typo

* check for >= 0 votes not >= 6 for pings

* fix serialization of daemon version

* remove ban scenario

* Revert "check for >= 0 votes not >= 6 for pings"

This reverts commit a6c2d79.

* update major mn version

* Revert "remove ban scenario"

This reverts commit ff53b31.

* update version

* make new mn qualify after 4 rounds

* update version and genesis hash

* reset mn version

* masternode payment updates

1) on istransactionvalid, where before it would have failed and halted processing block, we also let it go if the paid masternode is NSR
2) don't allow any NSR votes to be accepted (which helps 1 from people trying to vote for themselves even though they are NSR)
3) in check() allow up to 49 ping retries while 6 or more votes are required in the winners list for a masternode and no pings detected in last 10 minutes. Once it gets to 50 or more, it wont check for the winners list and will update every minute if no pings are detected for a masternode until 60, then NSR state ensues

* non enabled mn cannot ping

* remove node1

* update genesis and proto

* increae sleep between kills

* message for signing

* start in preenabled state

* process msgs while geth isn't synced

* accept connections unless mn and not geth synced

* watch for first time geth is synced

* Patched relayer

* Revert "Patched relayer"

This reverts commit aea436f.

* remove min proto

* update versions

* dont do auxpow for genesis

* fix collateral proposal tx check by using new gettx function with blockindex

removes need for txindex, previous version using accessbytxid wasn't working properly because for opreturn outputs it wouln't return true and if any change outputs were spent then accessbytxid on a burn tx would actually return "cant find collateral tx"

* strip out syscoin stuff from scriptPubKey before adding to vin

* fix witness related functionality

syscointxfund wouldnt add the owner output if it found a witness output since the tx fee calc would have been satisfied, instead we start our calculation from 0 everytime so it adds the outputs we need from the owner of the asset (the address passed in).

We also dont need to do any stripping as scantxoutset doesn't give us the syscoin outputs

* remove log output

* update chain params

* refactor custom outputs in favor of tx versioning

remove custom outputs and use tx versioning to understand syscoin txs. This removes an output for every syscoin tx so its more efficient and also more compatible with bitcoin related integrations like electrum which parses transactions to sign, because no custom outputs it wont need a handler to parse syscoin transactions.

Also refactored burning of sys/assets. For assets do not use cassetallocation serialized into opreturn but put the data needed to validate the burn in opreturn manually and recreate cassetallocation when the transaction is deserialized.

* Updated relayer to 1.0.5

* fix bugs

* fix assetsend/assetallocationsend and burn issues

* Add function to close all Db's and reload the databae environment

Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac

* After encrypting the wallet, reload the database environment

Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5

* Move BerkeleyEnvironment deletion from internal method to callsite

Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461

* No longer shutdown after encrypting the wallet

Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a

* wallet: Add WalletLocation utility class

Github-Pull: bitcoin#14350
Rebased-From: 01a4c09

* wallet: Refactor to use WalletLocation

Github-Pull: bitcoin#14350
Rebased-From: 65f3672

* wallet: Fix duplicate fileid

Github-Pull: bitcoin#14320
Rebased-From: 2d796fa

* tests: add test case for loading copied wallet twice

Github-Pull: bitcoin#14320
Rebased-From: 4ea7732

* wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory.

Github-Pull: bitcoin#14552
Rebased-From: 15c93f0

* Refactor: Move m_db pointers into BerkeleyDatabase

This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

* wallet: Create IsDatabaseLoaded function

Github-Pull: bitcoin#14552
Rebased-From: 5912031

* Free BerkeleyEnvironment instances when not in use

Instead of adding BerkeleyEnvironment objects permanently to the g_dbenvs map,
use reference counted shared pointers and remove map entries when the last
BerkeleyEnvironment reference goes out of scope.

This change was requested by Matt Corallo <git@bluematt.me> and makes code that
sets up mock databases cleaner. The mock database environment will now go out
of scope and be reset on destruction so there is no need to call
BerkeleyEnvironment::Reset() during wallet construction to clear out prior
state.

This change does affect bitcoin behavior slightly. On startup, instead of same
wallet environments staying open throughout VerifyWallets() and OpenWallets()
calls, VerifyWallets() will open and close an environment once for each wallet,
and OpenWallets() will create its own environment(s) later.

Github-Pull: bitcoin#11911
Rebased-From: f1f4bb7

* Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment

Github-Pull: bitcoin#11911
Rebased-From: 14bc2a1

* Tests: add unit tests for GetWalletEnv

Github-Pull: bitcoin#11911
Rebased-From: 88b1d95

* wallet: Close dbenv error file db.log

The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

* wallet: Close wallet env lock file

Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

* qa: Test .walletlock file is closed

Github-Pull: bitcoin#15297
Rebased-From: d3bf3b9

* Merge bitcoin#15575: 0.17: Backport 15297

fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

* depends: Enable unicode support on dbd for Windows

* Log progress while verifying blocks at level 4.

When verifying blocks at startup, the progress is printed in 10%
increments to logs.  When -checklevel=4, however, the second half
of the verification (connecting the blocks again) does not log the
progress anymore.  (It is still computed and shown in the UI, but
not printed to logs.)

This change makes the behaviour consistent, by adding the missing
progress logging also for level-4 checks.

* Hide spendable label if priveate key is disabled

* index: Fix for indexers skipping genesis block.

* compile

* remove files

* fix parsing to support current contract format

* use blockindex to index blocks per txid

* tx version based assets

* restructure spv proof call to give list of tx's and index of tx because the internal proof isnt a merkle proof, used bitcoin-proof instead

* reorg strategy around burn sender in mint asset tx

on mint, the sender is set to burn address, remove from burn when minting and ensuring balance > 0

on disconnect, the sender is set to burn address, add to burn when disconnecting

* minor bug fixes

dont send mn related sync commands to older clients.
isstd for burn txs
serialize tx through syscoingetspvproof as non witness

* allow for easy way to call assetssend and allocationsend without object

* fix assetupdate stripping contract and fix assettxtojson

* update contract

* remove burn sig + fix ethereum abi parsing

burn signature not needed as the function signature is always the same.
ethereum abi parsing was broken waiting for final form, now that we know what that is we hook it up and fix the parsing once and for all.

* dont adjust total sipply on mints/burns

* remove powf

* Removed regtest test files for node2 and node3

* fix order of params

* fix rlp data decode to bytes on parseethmethod check

also set max headers to 1 week based on light chain syncing roughly 1 week usually
prune headers on every set call as well to avoid big db

* working spv proofs

* Fix make install

* remove wallet dependency on sys rpc functions

everything except syscointxfund

* miner fixes

* wait till geth is synced before validating

* remove availablecointypes

we don't need to worry about specific masternode amount types since we now have multi wallet support, masternodes should be set up on a seperate wallet if they want to keep their utxos intact and not spend accidentily

* revert unintentional change

* check if geth synced before mint in rpc

* dont write to indexes if miner mode or just checking

since miner does a sanity run around syscoin consensus on txs  we actually want to make sure no database writes happen during this time, just the normal balance maps which are thrown away after checksyscoininputs is finished.

* renamed bins

* move wallet masternode stuff out of cwallet

* move sys consensus code to seperate file

* update header

* remove multiple outputs and wallet check for small sys inputs

* Fixed win64 bin names

* move static decl's out of header, causing issues

put in cpp files it seems it was declaring multiple times causing a test to fail regularily

convert sorted_vector to unsorted_set for standards

* only do concurrent processing for zdag transactions (allocation send)

* save 3*MAX_ETHEREUM_TX_ROOTS before pruning and check txroot if exists

we want to validate txroot if it exists, likely your node will have up to 120k txroots (last 3 weeks approx) so when you turn node off and on it should process all txroots of last 3 weeks if it finds them.

once you are synced up it should enforce mint transactions be a week(40k  eth blocks) or under otherwise its not allowed.

this fixes the case where a txroot may be invalid and then you turn node off and on and it skips validation, it should not proceed if txroot is invalid. When syncing up pull the txroot from db and validate it if it exists, if it doesn't exist AND your already synced then it should return an error.

* rmv files

* use blockindex to find collateral tx in governance proposal validation code

this means we can use existing sys3 process for proposals not breaking syshub

* rework blockindex

-blockindex is done by default, not a config option
   - this is because governance propsoal validator code needs it now, might as well make it default
-put it in its own db
-write to it using it batch processing for performance since its usually multiple writes at a time

* remove user syscointxfund requirement

this is now built into the rpc calls (except syscoinburn where it doesnt know what address to fund it from).

removed all 3000 satoshi hardcoded dust outputs in various calls, it will find the first usable output and spend that for proof of ownership

* remove txfund req from syscoinburn

we pass in funding address into syscoinburn and remove syscointxfund step for user
uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019
Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac
uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019
Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5
uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019
Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461
uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019
Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Oct 25, 2019
Summary:
There was a bug discovered by the addition of D4255 affecting ASAN
builds.  See core issue:
bitcoin/bitcoin#14163.  The exact solution
for this bug is unclear, but will likely require the backporting of at
least the following PRs and their dependencies from Core:

bitcoin/bitcoin#13161
bitcoin/bitcoin#12493
bitcoin/bitcoin#14320
bitcoin/bitcoin#14552
bitcoin/bitcoin#14760
bitcoin/bitcoin#11911

Suppression of the error is a temporary fix until the above backports
are complete.  This is the same approach that Core took.  See
https://github.com/bitcoin/bitcoin/pull/14794/files#diff-354f30a63fb0907d4ad57269548329e3R111
for the initial suppression and
https://github.com/bitcoin/bitcoin/pull/15303/files#diff-354f30a63fb0907d4ad57269548329e3L128
for its removal.

Test Plan:
From the `build` dir:
  export ABC_BUILD_NAME=build-asan
  ../contrib/teamcity/build-configurations.sh

ASAN build through TeamCity

Reviewers: deadalnix, Fabien, jasonbcox, O1 Bitcoin ABC, #bitcoin_abc

Reviewed By: Fabien, jasonbcox, O1 Bitcoin ABC, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4281
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jan 19, 2020
Summary:
2d471636eb9160ab51b08e491e3f003f57adbc36 wallet: Remove trailing separators from -walletdir arg (Pierre Rochard)
ea3009ee942188750480ca6cc273b2b91cf77ded wallet: Add walletdir arg unit tests (Pierre Rochard)

Pull request description:

  If a user passes in a path with a trailing separator as the `walletdir`, multiple BerkeleyEnvironments may be created in the same directory which can lead to data corruption.

  Discovered while reviewing bitcoin/bitcoin#12493 (comment)

Tree-SHA512: f2bbf1749d904fd3f326b88f2ead58c8386034355910906d7faea155d518642e9cd4ceb3cae272f2d9d8feb61f126523e1c97502799d24e4315bb53e49fd7c09

Backport of Core PR14146
https://github.com/bitcoin/bitcoin/pull/14146/files

Test Plan:
`ninja check`
Autotools build on CI

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D5004
xdustinface added a commit to xdustinface/dash that referenced this pull request Apr 4, 2021
…of shutting down

c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
xdustinface added a commit to xdustinface/dash that referenced this pull request Apr 4, 2021
…of shutting down

c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
xdustinface added a commit to xdustinface/dash that referenced this pull request Apr 4, 2021
…of shutting down

c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
xdustinface added a commit to xdustinface/dash that referenced this pull request Apr 5, 2021
…of shutting down

c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
xdustinface added a commit to xdustinface/dash that referenced this pull request Apr 13, 2021
…of shutting down

c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet