Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wallet: Releases dangling files on BerkeleyEnvironment::Close #15297

Merged
merged 3 commits into from Feb 5, 2019

Conversation

8 participants
@promag
Copy link
Member

commented Jan 31, 2019

This PR closes db.log and removes .walletlock files when BerkeleyEnvironment is closed.

Fixes #15291 (comment).

@gmaxwell

This comment has been minimized.

Copy link
Member

commented Jan 31, 2019

Concept ACK

@fanquake fanquake added the Wallet label Jan 31, 2019

@promag

This comment has been minimized.

Copy link
Member Author

commented Jan 31, 2019

Please add to 0.18 milestone.

@fanquake fanquake added this to the 0.18.0 milestone Jan 31, 2019

Show resolved Hide resolved src/wallet/db.cpp Outdated

@promag promag force-pushed the promag:2019-01-close-dbenv-files branch from 03b47cb to cb132bd Jan 31, 2019

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Jan 31, 2019

Needs rebase

@promag promag force-pushed the promag:2019-01-close-dbenv-files branch from cb132bd to bdbb3b5 Jan 31, 2019

@promag

This comment has been minimized.

Copy link
Member Author

commented Jan 31, 2019

Rebased.

@fanquake fanquake added this to In progress in Multiwallet support Feb 1, 2019

@practicalswift

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

Concept ACK

@promag promag force-pushed the promag:2019-01-close-dbenv-files branch from bdbb3b5 to 2acfb19 Feb 3, 2019

promag added some commits Jan 31, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.
wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

@promag promag force-pushed the promag:2019-01-close-dbenv-files branch from 2acfb19 to 2f8b8f4 Feb 4, 2019

@ryanofsky
Copy link
Contributor

left a comment

utACK 2f8b8f4. Nice cleanup!

@promag

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

Thanks @ryanofsky, pushed a test that fails without this PR.

@ryanofsky
Copy link
Contributor

left a comment

utACK d3bf3b9. Just added test since the previous review.

@laanwj

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

utACK 2f8b8f4

@fanquake

This comment has been minimized.

Copy link
Member

commented Feb 5, 2019

utACK d3bf3b9

master (fc21bb4):

src/qt/bitcoin-qt

lsof -p 56700 | grep wallet
bitcoin-q 56700 michael   24u     REG                1,4        0  4556196 /Users/michael/Library/Application Support/Bitcoin/wallets/.walletlock
bitcoin-q 56700 michael   25w     REG                1,4        0  4556198 /Users/michael/Library/Application Support/Bitcoin/wallets/db.log
bitcoin-q 56700 michael   33u     REG                1,4  1048576 26847564 /Users/michael/Library/Application Support/Bitcoin/wallets/database/log.0000000001
bitcoin-q 56700 michael   39w     REG                1,4        0  4556198 /Users/michael/Library/Application Support/Bitcoin/wallets/db.log

src/bitcoin-cli unloadwallet ""

lsof -p 56700 | grep wallet
bitcoin-q 56700 michael   24u     REG                1,4        0  4556196 /Users/michael/Library/Application Support/Bitcoin/wallets/.walletlock
bitcoin-q 56700 michael   25w     REG                1,4        0  4556198 /Users/michael/Library/Application Support/Bitcoin/wallets/db.log
bitcoin-q 56700 michael   39w     REG                1,4        0  4556198 /Users/michael/Library/Application Support/Bitcoin/wallets/db.log

This PR:

src/qt/bitcoin-qt

lsof -p 54793 | grep wallet
bitcoin-q 54793 michael   36u      REG                1,4        0  4556196 /Users/michael/Library/Application Support/Bitcoin/wallets/.walletlock
bitcoin-q 54793 michael   37w      REG                1,4        0  4556198 /Users/michael/Library/Application Support/Bitcoin/wallets/db.log
bitcoin-q 54793 michael   38u      REG                1,4  1048576 26847564 /Users/michael/Library/Application Support/Bitcoin/wallets/database/log.0000000001

src/bitcoin-cli unloadwallet ""

lsof -p 54793 | grep wallet
@meshcollider

This comment has been minimized.

Copy link
Member

commented Feb 5, 2019

utACK d3bf3b9

@meshcollider meshcollider merged commit d3bf3b9 into bitcoin:master Feb 5, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

meshcollider added a commit that referenced this pull request Feb 5, 2019

Merge #15297: wallet: Releases dangling files on BerkeleyEnvironment:…
…:Close

d3bf3b9 qa: Test .walletlock file is closed (João Barbosa)
2f8b8f4 wallet: Close wallet env lock file (João Barbosa)
8602a1e wallet: Close dbenv error file db.log (João Barbosa)

Pull request description:

  This PR closes `db.log` and removes `.walletlock` files when `BerkeleyEnvironment` is closed.

  Fixes #15291 (comment).

Tree-SHA512: 05d8b027feea914e0ba873e75d117857473d1fd7b400e41bd473d638171fa39d5be048990bf685dc0807f7d92418579b763056dc2a6dcf6b96777d5688ddee04

@jnewbery jnewbery moved this from In progress to Done in Multiwallet support Feb 5, 2019

@promag promag deleted the promag:2019-01-close-dbenv-files branch Feb 11, 2019

@MarcoFalke MarcoFalke modified the milestones: 0.18.0, 0.17.2 Mar 6, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 10, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

promag added a commit to promag/bitcoin that referenced this pull request Mar 10, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

promag added a commit to promag/bitcoin that referenced this pull request Mar 10, 2019

@promag promag referenced this pull request Mar 10, 2019

Merged

0.17: Backport 15297 #15575

@fanquake

This comment has been minimized.

Copy link
Member

commented Mar 11, 2019

Being backported in #15575.

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 12, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

promag added a commit to promag/bitcoin that referenced this pull request Mar 12, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

promag added a commit to promag/bitcoin that referenced this pull request Mar 12, 2019

LitecoinZ added a commit to litecoinz-core/devrepo that referenced this pull request Mar 17, 2019

Backport bitcoin#15297
- bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
- bitcoin#14350 Add WalletLocation class
- bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
- bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
- bitcoin#11911 Free BerkeleyEnvironment instances when not in use
- bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

laanwj added a commit that referenced this pull request Mar 20, 2019

Merge #15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - #12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - #14350 Add WalletLocation class
   - #14320 [bugfix] wallet: Fix duplicate fileid detection
   - #14552 wallet: detecting duplicate wallet by comparing the db filename.
   - #11911 Free BerkeleyEnvironment instances when not in use
   - #15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

Merge bitcoin#15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

Rishabh42 added a commit to Rishabh42/syscoin that referenced this pull request Apr 22, 2019

Dev 4.x (#8)
* Revert "more dbg"

This reverts commit c114420.

* Revert "more dbg"

This reverts commit 20d7fa1.

* Revert "more dbg"

This reverts commit f2a4621.

* Revert "logging"

This reverts commit 067d813.

* Revert "more dbg"

This reverts commit f2a4621.

* remove dbg

* remove dbg

* add 500 byte check in rpc prepare

* wip on new ban policy

* rpc: Make unloadwallet wait for complete wallet unload

Github-Pull: bitcoin#14941
Rebased-From: c37851d

* Remove errant past from walletcreatefundedpsbt for nLocktime replaceability

Github-Pull: bitcoin#15213
Rebased-From: 85f0ca9

* Update zmq to 4.3.1

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6250

Github-Pull: bitcoin#15188
Rebased-From: 3046e5f

* [0.17] [Doc] Backport release note about PSBT doc

* add pingretries to cmasternode

* start masternode preenabled and switch to enabled on ping check from broadcast

* assetallocationbalance rpc call

* have aamany vs just aa rpc

* display retries

* reset ping retries once successful ping and also ensure mn is in winnerslist

if mn is in winnerlist + 6 signatures a ping is accepted, if mn is in winners list between 0 and 6 signatures it wont be valid but it will not DOS ban peer, if  you are outside of the list entirely you should ban the peer.

The idea is that once you get into the winners list and have enough signatures its ok to start pinging. This avoids the race condition of pingers being banned and being accepted because thre is a delay between having 0 and 6 signatures, outside of the winners list

* change to 2 to test

* remove unused vars

* accept first ping on new broadcast, dont check winners list for that

* dont expire if lastPing is empty

* move enabled after checks and fix fSentinelIsCurrent

* move force check after enabled set

* reset retries only if mn enabled

* remove ping expired/expired states, either your enabled or not

* remove calls to isexpired and issentinelpingexpired

* idb check removed from miner

* rmove status print

* remove sentinel ping activation checks

* comment code out TODO put it back after test

* dont send ping unless in winners list with 6 sigs

* comment out ping

* enabled by default until theres a ping

* add sentinel ping expired state back

* refactor

* move preenabled check above

* relay if sentinel expired

* brackets around logic

* remove dbg

* ping from sentinelping for test

* some dbg

* allow pinging during preenabled state

* Revert "some dbg"

This reverts commit e6b06fe.

* dont increment npingretries if ourmn

* fix sentinelping call, call check every 10 min

* fix common lib

* move corewrite to server

* run mnchecks before calls that use the information

* is mn is not synced set to preenabled

* ensure we lock main first in istransactionvalid

* move pingretries to masternode info

* set pingretries to mnb list

* store ping retries in setmnping

* check retries from broadcast

* use references not copy

* some helpful debug

* dbg

* remove dbg

* allow force check on masternodemanager::Check

pass false by default but pass in true in the rpc calls and istransactionvalid. Don't update npingretries if passed in true, only when its not forced do we update that variable to avoid adding it multiple times

* set sentinel valid by default on broadcast

* set max retries to 60 and maintenance to every min

* only check ping in winners list if winners list is synced

* check winnerslist on sending ping only if list is synced

* serialize ping retries

* manage local ping retries

* check entire list is synced not just mn list

* startup preenabled for 10s

* remove dbg

* send ping only if not preenabled and is synced

* only check masternodes if they are in the winnerslist (or not enabled)

* output for not being in winners list in check

* use adjusted time instead of lastping for preenabled check

* use function already created IsBroadcastedWithin

* future sig time check fix

* give atleast MASTERNODE_SENTINEL_PING_MAX_SECONDS/2 seconds to propogate

* update to MASTERNODE_MIN_MNB_SECONDS

* also adjust broadcast check

* flip logic

* fix switching from pre to enabled state

* const references

* remove mn cache's

* update proto and hrp

* set sigs to 6 again

* update get to 1.8.23

* Updated relayer to 1.0.4

* geth default data dir in syscoin data dir

* pull out boot nodes

* Updated relayer to 1.0.4

* add net specific to data dir

* remove non-witness enforcement in mempool

this was taken out before but we missed this check, non-witness should be allowed

* update min proto

* skip check on empty winners list

* show active time from gettime if no ping has been processed yet

* check for multiple instances of daemon properly

* fix result check on pidof syscoin call result

* check asset consensus checks during mempool validation

* fix tests

* rmv no wallet build

* fix build

* remove graph from lint

* Removed graph and regex from libboost

* fix build

* fix build

* fix build

* typo

* remove main lock on istransactionvalid

the main lock in masternode::check is avoided from istransactionvalid because fForce=true and we only check isoutpoint which locks main in check() is fForce==false

* compile

* expire masternode sanity check if ping retries >= max

* compile

* dbg for getblockchaininfo

* remove potential for deadlock scenarios

* remove retry param

* process mncheck first time on startup

* better process killing

* implement our asset index spec part 1

* Bumped Masternode major version

* finish asset index spec

* add listassetindexassets RPC

* 3.8gb min

* flipped logic suppoed to end firsttime

* add block hash txid mapping to checksyscoininputs

* asset index configuration parameters

* check index on listasset rpc call

* check assets rpc call aswell

* check for assetindex before using it to init db

* default to page 0

* compile

* read txids regardless if doesnt exist

* compile

* write to asset page when changing or initializing

* remove double booking of asset index, refactor into standard functions and reuse

fix missing assetindextxid in the mintsyscoin indexing routine

* fix listassetallocation logic

* re-org txid structures so they don't double up during forks

* order LIFO on results

* refactor reorg of asset index

1) only erase payload
2) when scanning if payload doesn't exist, continue
this means that some pages may not be full (that is the tradeoff but its a performance based tradeoff). The alternatve is to remove TXIDs but that is more intensive because you have to remove for every receiver/sender which requires one to scan back pages to find txid.

* add erase txid on disconnect

we still have to rollback txid's on disconnect, may be slow but for now its required

* remove unneeded code

* remove potential deadlocking scenario with updatelastpaid/check

* remove potential deadlocks from mn rpc calls

updatelastpaid has cs_main locking inside but locks another cs before that, so we have to ensure we lock main first

* minor code cleanup

* dec page if empty

* ensure page doesnt overflow

* skip check() if not in winners list instead of enabled

also ping retries only if not preenabled to avoid cases where preenabled mn's start to count up since we are allowing non enabled checks now to happen in check() we need to ensure that the mn is in the right state before ping retries is incremented.

* fix while loop issue by changing page to int64_t

* only increase ping retries if votes >= 6 and on winners list, otherwise check every mn every 60 seconds

* remove def versions not needed

* fixed typo

* check for >= 0 votes not >= 6 for pings

* fix serialization of daemon version

* remove ban scenario

* Revert "check for >= 0 votes not >= 6 for pings"

This reverts commit a6c2d79.

* update major mn version

* Revert "remove ban scenario"

This reverts commit ff53b31.

* update version

* make new mn qualify after 4 rounds

* update version and genesis hash

* reset mn version

* masternode payment updates

1) on istransactionvalid, where before it would have failed and halted processing block, we also let it go if the paid masternode is NSR
2) don't allow any NSR votes to be accepted (which helps 1 from people trying to vote for themselves even though they are NSR)
3) in check() allow up to 49 ping retries while 6 or more votes are required in the winners list for a masternode and no pings detected in last 10 minutes. Once it gets to 50 or more, it wont check for the winners list and will update every minute if no pings are detected for a masternode until 60, then NSR state ensues

* non enabled mn cannot ping

* remove node1

* update genesis and proto

* increae sleep between kills

* message for signing

* start in preenabled state

* process msgs while geth isn't synced

* accept connections unless mn and not geth synced

* watch for first time geth is synced

* Patched relayer

* Revert "Patched relayer"

This reverts commit aea436f.

* remove min proto

* update versions

* dont do auxpow for genesis

* fix collateral proposal tx check by using new gettx function with blockindex

removes need for txindex, previous version using accessbytxid wasn't working properly because for opreturn outputs it wouln't return true and if any change outputs were spent then accessbytxid on a burn tx would actually return "cant find collateral tx"

* strip out syscoin stuff from scriptPubKey before adding to vin

* fix witness related functionality

syscointxfund wouldnt add the owner output if it found a witness output since the tx fee calc would have been satisfied, instead we start our calculation from 0 everytime so it adds the outputs we need from the owner of the asset (the address passed in).

We also dont need to do any stripping as scantxoutset doesn't give us the syscoin outputs

* remove log output

* update chain params

* refactor custom outputs in favor of tx versioning

remove custom outputs and use tx versioning to understand syscoin txs. This removes an output for every syscoin tx so its more efficient and also more compatible with bitcoin related integrations like electrum which parses transactions to sign, because no custom outputs it wont need a handler to parse syscoin transactions.

Also refactored burning of sys/assets. For assets do not use cassetallocation serialized into opreturn but put the data needed to validate the burn in opreturn manually and recreate cassetallocation when the transaction is deserialized.

* Updated relayer to 1.0.5

* fix bugs

* fix assetsend/assetallocationsend and burn issues

* Add function to close all Db's and reload the databae environment

Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac

* After encrypting the wallet, reload the database environment

Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5

* Move BerkeleyEnvironment deletion from internal method to callsite

Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461

* No longer shutdown after encrypting the wallet

Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a

* wallet: Add WalletLocation utility class

Github-Pull: bitcoin#14350
Rebased-From: 01a4c09

* wallet: Refactor to use WalletLocation

Github-Pull: bitcoin#14350
Rebased-From: 65f3672

* wallet: Fix duplicate fileid

Github-Pull: bitcoin#14320
Rebased-From: 2d796fa

* tests: add test case for loading copied wallet twice

Github-Pull: bitcoin#14320
Rebased-From: 4ea7732

* wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory.

Github-Pull: bitcoin#14552
Rebased-From: 15c93f0

* Refactor: Move m_db pointers into BerkeleyDatabase

This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

* wallet: Create IsDatabaseLoaded function

Github-Pull: bitcoin#14552
Rebased-From: 5912031

* Free BerkeleyEnvironment instances when not in use

Instead of adding BerkeleyEnvironment objects permanently to the g_dbenvs map,
use reference counted shared pointers and remove map entries when the last
BerkeleyEnvironment reference goes out of scope.

This change was requested by Matt Corallo <git@bluematt.me> and makes code that
sets up mock databases cleaner. The mock database environment will now go out
of scope and be reset on destruction so there is no need to call
BerkeleyEnvironment::Reset() during wallet construction to clear out prior
state.

This change does affect bitcoin behavior slightly. On startup, instead of same
wallet environments staying open throughout VerifyWallets() and OpenWallets()
calls, VerifyWallets() will open and close an environment once for each wallet,
and OpenWallets() will create its own environment(s) later.

Github-Pull: bitcoin#11911
Rebased-From: f1f4bb7

* Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment

Github-Pull: bitcoin#11911
Rebased-From: 14bc2a1

* Tests: add unit tests for GetWalletEnv

Github-Pull: bitcoin#11911
Rebased-From: 88b1d95

* wallet: Close dbenv error file db.log

The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

* wallet: Close wallet env lock file

Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

* qa: Test .walletlock file is closed

Github-Pull: bitcoin#15297
Rebased-From: d3bf3b9

* Merge bitcoin#15575: 0.17: Backport 15297

fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

* depends: Enable unicode support on dbd for Windows

* Log progress while verifying blocks at level 4.

When verifying blocks at startup, the progress is printed in 10%
increments to logs.  When -checklevel=4, however, the second half
of the verification (connecting the blocks again) does not log the
progress anymore.  (It is still computed and shown in the UI, but
not printed to logs.)

This change makes the behaviour consistent, by adding the missing
progress logging also for level-4 checks.

* Hide spendable label if priveate key is disabled

* index: Fix for indexers skipping genesis block.

* compile

* remove files

* fix parsing to support current contract format

* use blockindex to index blocks per txid

* tx version based assets

* restructure spv proof call to give list of tx's and index of tx because the internal proof isnt a merkle proof, used bitcoin-proof instead

* reorg strategy around burn sender in mint asset tx

on mint, the sender is set to burn address, remove from burn when minting and ensuring balance > 0

on disconnect, the sender is set to burn address, add to burn when disconnecting

* minor bug fixes

dont send mn related sync commands to older clients.
isstd for burn txs
serialize tx through syscoingetspvproof as non witness

* allow for easy way to call assetssend and allocationsend without object

* fix assetupdate stripping contract and fix assettxtojson

* update contract

* remove burn sig + fix ethereum abi parsing

burn signature not needed as the function signature is always the same.
ethereum abi parsing was broken waiting for final form, now that we know what that is we hook it up and fix the parsing once and for all.

* dont adjust total sipply on mints/burns

* remove powf

* Removed regtest test files for node2 and node3

* fix order of params

* fix rlp data decode to bytes on parseethmethod check

also set max headers to 1 week based on light chain syncing roughly 1 week usually
prune headers on every set call as well to avoid big db

* working spv proofs

* Fix make install

* remove wallet dependency on sys rpc functions

everything except syscointxfund

* miner fixes

* wait till geth is synced before validating

* remove availablecointypes

we don't need to worry about specific masternode amount types since we now have multi wallet support, masternodes should be set up on a seperate wallet if they want to keep their utxos intact and not spend accidentily

* revert unintentional change

* check if geth synced before mint in rpc

* dont write to indexes if miner mode or just checking

since miner does a sanity run around syscoin consensus on txs  we actually want to make sure no database writes happen during this time, just the normal balance maps which are thrown away after checksyscoininputs is finished.

* renamed bins

* move wallet masternode stuff out of cwallet

* move sys consensus code to seperate file

* update header

* remove multiple outputs and wallet check for small sys inputs

* Fixed win64 bin names

* move static decl's out of header, causing issues

put in cpp files it seems it was declaring multiple times causing a test to fail regularily

convert sorted_vector to unsorted_set for standards

* only do concurrent processing for zdag transactions (allocation send)

* save 3*MAX_ETHEREUM_TX_ROOTS before pruning and check txroot if exists

we want to validate txroot if it exists, likely your node will have up to 120k txroots (last 3 weeks approx) so when you turn node off and on it should process all txroots of last 3 weeks if it finds them.

once you are synced up it should enforce mint transactions be a week(40k  eth blocks) or under otherwise its not allowed.

this fixes the case where a txroot may be invalid and then you turn node off and on and it skips validation, it should not proceed if txroot is invalid. When syncing up pull the txroot from db and validate it if it exists, if it doesn't exist AND your already synced then it should return an error.

* rmv files

* use blockindex to find collateral tx in governance proposal validation code

this means we can use existing sys3 process for proposals not breaking syshub

* rework blockindex

-blockindex is done by default, not a config option
   - this is because governance propsoal validator code needs it now, might as well make it default
-put it in its own db
-write to it using it batch processing for performance since its usually multiple writes at a time

* remove user syscointxfund requirement

this is now built into the rpc calls (except syscoinburn where it doesnt know what address to fund it from).

removed all 3000 satoshi hardcoded dust outputs in various calls, it will find the first usable output and spend that for proof of ownership

* remove txfund req from syscoinburn

we pass in funding address into syscoinburn and remove syscointxfund step for user

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

wallet: Close dbenv error file db.log
The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

wallet: Close wallet env lock file
Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.