Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wallet: detecting duplicate wallet by comparing the db filename. #14552

Merged
merged 3 commits into from Nov 20, 2018

Conversation

Projects
None yet
8 participants
@ken2812221
Copy link
Member

commented Oct 23, 2018

Fix #14538

Fix crash attempting to load the same wallet with different path strings that resolve to the same absolute path. The primary check which prevents loading the same wallet twice is:

throw std::runtime_error(strprintf("BerkeleyBatch: Can't open database %s (duplicates fileid %s from %s)", filename,

But this check is skipped if both wallet paths resolve to the same absolute path, due to caching here:

if (pdb == nullptr) {

Meanwhile a secondary check for duplicate wallets is not reliable because it based on a literal comparison, instead of comparison using absolute paths:

error_string = strprintf("Error loading wallet %s. Duplicate -wallet filename specified.", location.GetName());

This PR fixes the latter check to compare the absolute path of a new wallet being loaded to absolute paths of wallets already loaded, so there should no longer be any way to load the same wallet more than once.

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch Oct 23, 2018

@fanquake fanquake added the Wallet label Oct 23, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch Oct 23, 2018

@ken2812221 ken2812221 changed the title wallet: throw an error if user load the wallet file by different ways wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. Oct 23, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch 2 times, most recently Oct 23, 2018

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 23, 2018

Would prefer to check this at database level, rather than in higher level wallet code. I think it's better if high level code just passes along wallet paths and isn't concerned with how data is stored in them.

I posted a tweaked version of this PR at 40a6475 which moves the check. @ken2812221, could you incorporate 40a6475 into this PR if you think it makes sense?

@ken2812221

This comment has been minimized.

Copy link
Member Author

commented Oct 23, 2018

@ryanofsky The element in mapDb is not always exist.

For example:

  1. Load wallet dir/w1
  2. Load wallet dir/w2
  3. Unload wallet dir/w1

Now mapDb contains neither dir/w1 nor dir/w2 because it would flush all wallet files that share same BerkeleyEnvironment

@DrahtBot

This comment has been minimized.

Copy link
Contributor

commented Oct 23, 2018

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #11911 (Free BerkeleyEnvironment instances when not in use by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2018

@ryanofsky The element in mapDb is not always exist.

Good catch. I think a small refactoring could fix this: 9c945d0. And then the fix would change slightly to aef19be.

I pushed a branch with these two commits to: https://github.com/ryanofsky/bitcoin/commits/pr/bloaded

I think this is a better than checking for wallet.dat files outside the database layer, but let me know what you think, and feel free to use the code or commits in this PR if you think this approach makes sense.

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch to aef19be Oct 25, 2018

@ryanofsky
Copy link
Contributor

left a comment

utACK aef19be

@ken2812221 ken2812221 changed the title wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. wallet: detecting duplicate wallet by comparing the db file. Oct 26, 2018

@ken2812221 ken2812221 changed the title wallet: detecting duplicate wallet by comparing the db file. wallet: detecting duplicate wallet by comparing the db filename. Oct 30, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch 2 times, most recently Nov 5, 2018

@DrahtBot DrahtBot removed the Needs rebase label Nov 5, 2018

@ken2812221

This comment has been minimized.

Copy link
Member Author

commented Nov 5, 2018

Rebased

@ryanofsky
Copy link
Contributor

left a comment

utACK b0375538375eb0c419b308d1015723ab64d53294. Only change since last review is rebase.

@promag, this might be an easy PR for to you review. First commit is just refactoring that doesn't change behavior. Second commit is the actual fix.

@ken2812221, it might help for this to have a more complete PR description.


Suggested PR description:

Fix crash attempting to load the same wallet with different path strings that resolve to the same absolute path. The primary check which prevents loading the same wallet twice is:

throw std::runtime_error(strprintf("BerkeleyBatch: Can't open database %s (duplicates fileid %s from %s)", filename,

But this check is skipped if both wallet paths resolve to the same absolute path, due to caching here:

if (pdb == nullptr) {

Meanwhile a secondary check for duplicate wallets is not reliable because it based on a literal comparison, instead of comparison using absolute paths:

error_string = strprintf("Error loading wallet %s. Duplicate -wallet filename specified.", location.GetName());

This PR fixes the latter check to compare the absolute path of a new wallet being loaded to absolute paths of wallets already loaded, so there should no longer be any way to load the same wallet more than once.

@ken2812221

This comment has been minimized.

Copy link
Member Author

commented Nov 5, 2018

@ryanofsky Thanks for your review. I would take your suggestion.

src/wallet/db.cpp Outdated
@@ -463,7 +463,8 @@ BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase& database, const char* pszMode, bo
if (!env->Open(false /* retry */))
throw std::runtime_error("BerkeleyBatch: Failed to open database environment.");

pdb = env->mapDb[strFilename];
BerkeleyDatabase& database = env->m_databases.at(strFilename).get();

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

Some comments here:

  • there's a slight change when replacing map::operator[] with map::at, maybe worth a comemnt or an assertion?
  • also, isn't this equivalent to the above database argument?
  • if this is really necessary then you could avoid shadowing database.

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Nov 5, 2018

Contributor

re: #14552 (comment)

Oh, I didn't even realize there was a database argument being shadowed above. Should just delete this line and use the existing database, which points to the same thing.

if (mock) {
env->Close();
env->Reset();
env->MakeMock();
}
}

~BerkeleyDatabase() {

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

nit, could add final to class since this is not virtual.

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Nov 5, 2018

Author Member

final can only be added to virtual method.

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

@ken2812221 I mean class BerkeleyDatabase final.

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Nov 5, 2018

Contributor

Change would be unrelated to this PR, and IMO, it's better only to use final when you have virtual methods and actual optimizations this would allow or bugs it would prevent. There's a core guideline that touches on this: http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rh-final

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

Nice thanks.

src/wallet/db.cpp Outdated
@@ -56,9 +56,8 @@ bool WalletDatabaseFileId::operator==(const WalletDatabaseFileId& rhs) const
return memcmp(value, &rhs.value, sizeof(value)) == 0;
}

BerkeleyEnvironment* GetWalletEnv(const fs::path& wallet_path, std::string& database_filename)
void SplitWalletPath(const fs::path& wallet_path, fs::path& env_directory, std::string& database_filename)

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

Could be static?

src/wallet/db.cpp Outdated
@@ -563,12 +581,11 @@ void BerkeleyEnvironment::CloseDb(const std::string& strFile)
{
{
LOCK(cs_db);
if (mapDb[strFile] != nullptr) {
BerkeleyDatabase& database = m_databases.at(strFile).get();

This comment has been minimized.

Copy link
@promag

promag Nov 5, 2018

Member

Same as above, there's a slight change when replacing map::operator[] with map::at, maybe worth a comment or an assertion?

@ken2812221 ken2812221 force-pushed the ken2812221:default-wallet-fix branch 2 times, most recently Nov 5, 2018

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Nov 15, 2018

Is this for backport?

@MarcoFalke MarcoFalke added this to the 0.18.0 milestone Nov 15, 2018

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2018

Is this for backport?

I don't have a great sense for which changes get backported and which changes don't. I would guess it could be backported, but I don't think it's important to backport. Original bug #14538 reporter @chernyshev might have an opinion.

Note: related fix #14320 (comment) does not appear to have been backported.

@laanwj laanwj added this to Blockers in High-priority for review Nov 15, 2018

@@ -43,7 +45,7 @@ class BerkeleyEnvironment
public:
std::unique_ptr<DbEnv> dbenv;
std::map<std::string, int> mapFileUseCount;
std::map<std::string, Db*> mapDb;
std::map<std::string, std::reference_wrapper<BerkeleyDatabase>> m_databases;

This comment has been minimized.

Copy link
@promag

promag Nov 19, 2018

Member

Shouldn't this be guarded by cs_db?

@laanwj laanwj merged commit 5912031 into bitcoin:master Nov 20, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Nov 20, 2018

Merge #14552: wallet: detecting duplicate wallet by comparing the db …
…filename.

5912031 wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
15c93f0 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
c456fbd Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)

Pull request description:

  Fix #14538

  Fix crash attempting to load the same wallet with different path strings that resolve to the same absolute path. The primary check which prevents loading the same wallet twice is:

  https://github.com/bitcoin/bitcoin/blob/6b8d0a2164b30eab76e7bccb1ffb056a10fba406/src/wallet/db.cpp#L44

  But this check is skipped if both wallet paths resolve to the same absolute path, due to caching here:

  https://github.com/bitcoin/bitcoin/blob/6b8d0a2164b30eab76e7bccb1ffb056a10fba406/src/wallet/db.cpp#L467

  Meanwhile a secondary check for duplicate wallets is not reliable because it based on a literal comparison, instead of comparison using absolute paths:

  https://github.com/bitcoin/bitcoin/blob/6b8d0a2164b30eab76e7bccb1ffb056a10fba406/src/wallet/wallet.cpp#L3853

  This PR fixes the latter check to compare the absolute path of a new wallet being loaded to absolute paths of wallets already loaded, so there should no longer be any way to load the same wallet more than once.

Tree-SHA512: 2fa01811c160b57be3b76c6b4983556a04bbce71a3f8202429987ec020664a062e897deedcd9248bc04e9baaa2fc7b464e2595dcaeff2af0818387bf1fcdbf6f

@ken2812221 ken2812221 deleted the ken2812221:default-wallet-fix branch Nov 20, 2018

@fanquake fanquake removed this from Blockers in High-priority for review Nov 21, 2018

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

Refactor: Move m_db pointers into BerkeleyDatabase
This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

@promag promag referenced this pull request Mar 11, 2019

Merged

0.17: Backport 15297 #15575

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

Refactor: Move m_db pointers into BerkeleyDatabase
This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

LitecoinZ added a commit to litecoinz-core/devrepo that referenced this pull request Mar 17, 2019

Backport bitcoin#15297
- bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
- bitcoin#14350 Add WalletLocation class
- bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
- bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
- bitcoin#11911 Free BerkeleyEnvironment instances when not in use
- bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

laanwj added a commit that referenced this pull request Mar 20, 2019

Merge #15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - #12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - #14350 Add WalletLocation class
   - #14320 [bugfix] wallet: Fix duplicate fileid detection
   - #14552 wallet: detecting duplicate wallet by comparing the db filename.
   - #11911 Free BerkeleyEnvironment instances when not in use
   - #15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

Refactor: Move m_db pointers into BerkeleyDatabase
This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

Merge bitcoin#15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

Refactor: Move m_db pointers into BerkeleyDatabase
This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

Rishabh42 added a commit to Rishabh42/syscoin that referenced this pull request Apr 22, 2019

Dev 4.x (#8)
* Revert "more dbg"

This reverts commit c114420.

* Revert "more dbg"

This reverts commit 20d7fa1.

* Revert "more dbg"

This reverts commit f2a4621.

* Revert "logging"

This reverts commit 067d813.

* Revert "more dbg"

This reverts commit f2a4621.

* remove dbg

* remove dbg

* add 500 byte check in rpc prepare

* wip on new ban policy

* rpc: Make unloadwallet wait for complete wallet unload

Github-Pull: bitcoin#14941
Rebased-From: c37851d

* Remove errant past from walletcreatefundedpsbt for nLocktime replaceability

Github-Pull: bitcoin#15213
Rebased-From: 85f0ca9

* Update zmq to 4.3.1

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6250

Github-Pull: bitcoin#15188
Rebased-From: 3046e5f

* [0.17] [Doc] Backport release note about PSBT doc

* add pingretries to cmasternode

* start masternode preenabled and switch to enabled on ping check from broadcast

* assetallocationbalance rpc call

* have aamany vs just aa rpc

* display retries

* reset ping retries once successful ping and also ensure mn is in winnerslist

if mn is in winnerlist + 6 signatures a ping is accepted, if mn is in winners list between 0 and 6 signatures it wont be valid but it will not DOS ban peer, if  you are outside of the list entirely you should ban the peer.

The idea is that once you get into the winners list and have enough signatures its ok to start pinging. This avoids the race condition of pingers being banned and being accepted because thre is a delay between having 0 and 6 signatures, outside of the winners list

* change to 2 to test

* remove unused vars

* accept first ping on new broadcast, dont check winners list for that

* dont expire if lastPing is empty

* move enabled after checks and fix fSentinelIsCurrent

* move force check after enabled set

* reset retries only if mn enabled

* remove ping expired/expired states, either your enabled or not

* remove calls to isexpired and issentinelpingexpired

* idb check removed from miner

* rmove status print

* remove sentinel ping activation checks

* comment code out TODO put it back after test

* dont send ping unless in winners list with 6 sigs

* comment out ping

* enabled by default until theres a ping

* add sentinel ping expired state back

* refactor

* move preenabled check above

* relay if sentinel expired

* brackets around logic

* remove dbg

* ping from sentinelping for test

* some dbg

* allow pinging during preenabled state

* Revert "some dbg"

This reverts commit e6b06fe.

* dont increment npingretries if ourmn

* fix sentinelping call, call check every 10 min

* fix common lib

* move corewrite to server

* run mnchecks before calls that use the information

* is mn is not synced set to preenabled

* ensure we lock main first in istransactionvalid

* move pingretries to masternode info

* set pingretries to mnb list

* store ping retries in setmnping

* check retries from broadcast

* use references not copy

* some helpful debug

* dbg

* remove dbg

* allow force check on masternodemanager::Check

pass false by default but pass in true in the rpc calls and istransactionvalid. Don't update npingretries if passed in true, only when its not forced do we update that variable to avoid adding it multiple times

* set sentinel valid by default on broadcast

* set max retries to 60 and maintenance to every min

* only check ping in winners list if winners list is synced

* check winnerslist on sending ping only if list is synced

* serialize ping retries

* manage local ping retries

* check entire list is synced not just mn list

* startup preenabled for 10s

* remove dbg

* send ping only if not preenabled and is synced

* only check masternodes if they are in the winnerslist (or not enabled)

* output for not being in winners list in check

* use adjusted time instead of lastping for preenabled check

* use function already created IsBroadcastedWithin

* future sig time check fix

* give atleast MASTERNODE_SENTINEL_PING_MAX_SECONDS/2 seconds to propogate

* update to MASTERNODE_MIN_MNB_SECONDS

* also adjust broadcast check

* flip logic

* fix switching from pre to enabled state

* const references

* remove mn cache's

* update proto and hrp

* set sigs to 6 again

* update get to 1.8.23

* Updated relayer to 1.0.4

* geth default data dir in syscoin data dir

* pull out boot nodes

* Updated relayer to 1.0.4

* add net specific to data dir

* remove non-witness enforcement in mempool

this was taken out before but we missed this check, non-witness should be allowed

* update min proto

* skip check on empty winners list

* show active time from gettime if no ping has been processed yet

* check for multiple instances of daemon properly

* fix result check on pidof syscoin call result

* check asset consensus checks during mempool validation

* fix tests

* rmv no wallet build

* fix build

* remove graph from lint

* Removed graph and regex from libboost

* fix build

* fix build

* fix build

* typo

* remove main lock on istransactionvalid

the main lock in masternode::check is avoided from istransactionvalid because fForce=true and we only check isoutpoint which locks main in check() is fForce==false

* compile

* expire masternode sanity check if ping retries >= max

* compile

* dbg for getblockchaininfo

* remove potential for deadlock scenarios

* remove retry param

* process mncheck first time on startup

* better process killing

* implement our asset index spec part 1

* Bumped Masternode major version

* finish asset index spec

* add listassetindexassets RPC

* 3.8gb min

* flipped logic suppoed to end firsttime

* add block hash txid mapping to checksyscoininputs

* asset index configuration parameters

* check index on listasset rpc call

* check assets rpc call aswell

* check for assetindex before using it to init db

* default to page 0

* compile

* read txids regardless if doesnt exist

* compile

* write to asset page when changing or initializing

* remove double booking of asset index, refactor into standard functions and reuse

fix missing assetindextxid in the mintsyscoin indexing routine

* fix listassetallocation logic

* re-org txid structures so they don't double up during forks

* order LIFO on results

* refactor reorg of asset index

1) only erase payload
2) when scanning if payload doesn't exist, continue
this means that some pages may not be full (that is the tradeoff but its a performance based tradeoff). The alternatve is to remove TXIDs but that is more intensive because you have to remove for every receiver/sender which requires one to scan back pages to find txid.

* add erase txid on disconnect

we still have to rollback txid's on disconnect, may be slow but for now its required

* remove unneeded code

* remove potential deadlocking scenario with updatelastpaid/check

* remove potential deadlocks from mn rpc calls

updatelastpaid has cs_main locking inside but locks another cs before that, so we have to ensure we lock main first

* minor code cleanup

* dec page if empty

* ensure page doesnt overflow

* skip check() if not in winners list instead of enabled

also ping retries only if not preenabled to avoid cases where preenabled mn's start to count up since we are allowing non enabled checks now to happen in check() we need to ensure that the mn is in the right state before ping retries is incremented.

* fix while loop issue by changing page to int64_t

* only increase ping retries if votes >= 6 and on winners list, otherwise check every mn every 60 seconds

* remove def versions not needed

* fixed typo

* check for >= 0 votes not >= 6 for pings

* fix serialization of daemon version

* remove ban scenario

* Revert "check for >= 0 votes not >= 6 for pings"

This reverts commit a6c2d79.

* update major mn version

* Revert "remove ban scenario"

This reverts commit ff53b31.

* update version

* make new mn qualify after 4 rounds

* update version and genesis hash

* reset mn version

* masternode payment updates

1) on istransactionvalid, where before it would have failed and halted processing block, we also let it go if the paid masternode is NSR
2) don't allow any NSR votes to be accepted (which helps 1 from people trying to vote for themselves even though they are NSR)
3) in check() allow up to 49 ping retries while 6 or more votes are required in the winners list for a masternode and no pings detected in last 10 minutes. Once it gets to 50 or more, it wont check for the winners list and will update every minute if no pings are detected for a masternode until 60, then NSR state ensues

* non enabled mn cannot ping

* remove node1

* update genesis and proto

* increae sleep between kills

* message for signing

* start in preenabled state

* process msgs while geth isn't synced

* accept connections unless mn and not geth synced

* watch for first time geth is synced

* Patched relayer

* Revert "Patched relayer"

This reverts commit aea436f.

* remove min proto

* update versions

* dont do auxpow for genesis

* fix collateral proposal tx check by using new gettx function with blockindex

removes need for txindex, previous version using accessbytxid wasn't working properly because for opreturn outputs it wouln't return true and if any change outputs were spent then accessbytxid on a burn tx would actually return "cant find collateral tx"

* strip out syscoin stuff from scriptPubKey before adding to vin

* fix witness related functionality

syscointxfund wouldnt add the owner output if it found a witness output since the tx fee calc would have been satisfied, instead we start our calculation from 0 everytime so it adds the outputs we need from the owner of the asset (the address passed in).

We also dont need to do any stripping as scantxoutset doesn't give us the syscoin outputs

* remove log output

* update chain params

* refactor custom outputs in favor of tx versioning

remove custom outputs and use tx versioning to understand syscoin txs. This removes an output for every syscoin tx so its more efficient and also more compatible with bitcoin related integrations like electrum which parses transactions to sign, because no custom outputs it wont need a handler to parse syscoin transactions.

Also refactored burning of sys/assets. For assets do not use cassetallocation serialized into opreturn but put the data needed to validate the burn in opreturn manually and recreate cassetallocation when the transaction is deserialized.

* Updated relayer to 1.0.5

* fix bugs

* fix assetsend/assetallocationsend and burn issues

* Add function to close all Db's and reload the databae environment

Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac

* After encrypting the wallet, reload the database environment

Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5

* Move BerkeleyEnvironment deletion from internal method to callsite

Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461

* No longer shutdown after encrypting the wallet

Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a

* wallet: Add WalletLocation utility class

Github-Pull: bitcoin#14350
Rebased-From: 01a4c09

* wallet: Refactor to use WalletLocation

Github-Pull: bitcoin#14350
Rebased-From: 65f3672

* wallet: Fix duplicate fileid

Github-Pull: bitcoin#14320
Rebased-From: 2d796fa

* tests: add test case for loading copied wallet twice

Github-Pull: bitcoin#14320
Rebased-From: 4ea7732

* wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory.

Github-Pull: bitcoin#14552
Rebased-From: 15c93f0

* Refactor: Move m_db pointers into BerkeleyDatabase

This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

* wallet: Create IsDatabaseLoaded function

Github-Pull: bitcoin#14552
Rebased-From: 5912031

* Free BerkeleyEnvironment instances when not in use

Instead of adding BerkeleyEnvironment objects permanently to the g_dbenvs map,
use reference counted shared pointers and remove map entries when the last
BerkeleyEnvironment reference goes out of scope.

This change was requested by Matt Corallo <git@bluematt.me> and makes code that
sets up mock databases cleaner. The mock database environment will now go out
of scope and be reset on destruction so there is no need to call
BerkeleyEnvironment::Reset() during wallet construction to clear out prior
state.

This change does affect bitcoin behavior slightly. On startup, instead of same
wallet environments staying open throughout VerifyWallets() and OpenWallets()
calls, VerifyWallets() will open and close an environment once for each wallet,
and OpenWallets() will create its own environment(s) later.

Github-Pull: bitcoin#11911
Rebased-From: f1f4bb7

* Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment

Github-Pull: bitcoin#11911
Rebased-From: 14bc2a1

* Tests: add unit tests for GetWalletEnv

Github-Pull: bitcoin#11911
Rebased-From: 88b1d95

* wallet: Close dbenv error file db.log

The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

* wallet: Close wallet env lock file

Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

* qa: Test .walletlock file is closed

Github-Pull: bitcoin#15297
Rebased-From: d3bf3b9

* Merge bitcoin#15575: 0.17: Backport 15297

fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

* depends: Enable unicode support on dbd for Windows

* Log progress while verifying blocks at level 4.

When verifying blocks at startup, the progress is printed in 10%
increments to logs.  When -checklevel=4, however, the second half
of the verification (connecting the blocks again) does not log the
progress anymore.  (It is still computed and shown in the UI, but
not printed to logs.)

This change makes the behaviour consistent, by adding the missing
progress logging also for level-4 checks.

* Hide spendable label if priveate key is disabled

* index: Fix for indexers skipping genesis block.

* compile

* remove files

* fix parsing to support current contract format

* use blockindex to index blocks per txid

* tx version based assets

* restructure spv proof call to give list of tx's and index of tx because the internal proof isnt a merkle proof, used bitcoin-proof instead

* reorg strategy around burn sender in mint asset tx

on mint, the sender is set to burn address, remove from burn when minting and ensuring balance > 0

on disconnect, the sender is set to burn address, add to burn when disconnecting

* minor bug fixes

dont send mn related sync commands to older clients.
isstd for burn txs
serialize tx through syscoingetspvproof as non witness

* allow for easy way to call assetssend and allocationsend without object

* fix assetupdate stripping contract and fix assettxtojson

* update contract

* remove burn sig + fix ethereum abi parsing

burn signature not needed as the function signature is always the same.
ethereum abi parsing was broken waiting for final form, now that we know what that is we hook it up and fix the parsing once and for all.

* dont adjust total sipply on mints/burns

* remove powf

* Removed regtest test files for node2 and node3

* fix order of params

* fix rlp data decode to bytes on parseethmethod check

also set max headers to 1 week based on light chain syncing roughly 1 week usually
prune headers on every set call as well to avoid big db

* working spv proofs

* Fix make install

* remove wallet dependency on sys rpc functions

everything except syscointxfund

* miner fixes

* wait till geth is synced before validating

* remove availablecointypes

we don't need to worry about specific masternode amount types since we now have multi wallet support, masternodes should be set up on a seperate wallet if they want to keep their utxos intact and not spend accidentily

* revert unintentional change

* check if geth synced before mint in rpc

* dont write to indexes if miner mode or just checking

since miner does a sanity run around syscoin consensus on txs  we actually want to make sure no database writes happen during this time, just the normal balance maps which are thrown away after checksyscoininputs is finished.

* renamed bins

* move wallet masternode stuff out of cwallet

* move sys consensus code to seperate file

* update header

* remove multiple outputs and wallet check for small sys inputs

* Fixed win64 bin names

* move static decl's out of header, causing issues

put in cpp files it seems it was declaring multiple times causing a test to fail regularily

convert sorted_vector to unsorted_set for standards

* only do concurrent processing for zdag transactions (allocation send)

* save 3*MAX_ETHEREUM_TX_ROOTS before pruning and check txroot if exists

we want to validate txroot if it exists, likely your node will have up to 120k txroots (last 3 weeks approx) so when you turn node off and on it should process all txroots of last 3 weeks if it finds them.

once you are synced up it should enforce mint transactions be a week(40k  eth blocks) or under otherwise its not allowed.

this fixes the case where a txroot may be invalid and then you turn node off and on and it skips validation, it should not proceed if txroot is invalid. When syncing up pull the txroot from db and validate it if it exists, if it doesn't exist AND your already synced then it should return an error.

* rmv files

* use blockindex to find collateral tx in governance proposal validation code

this means we can use existing sys3 process for proposals not breaking syshub

* rework blockindex

-blockindex is done by default, not a config option
   - this is because governance propsoal validator code needs it now, might as well make it default
-put it in its own db
-write to it using it batch processing for performance since its usually multiple writes at a time

* remove user syscointxfund requirement

this is now built into the rpc calls (except syscoinburn where it doesnt know what address to fund it from).

removed all 3000 satoshi hardcoded dust outputs in various calls, it will find the first usable output and spend that for proof of ownership

* remove txfund req from syscoinburn

we pass in funding address into syscoinburn and remove syscointxfund step for user

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

Refactor: Move m_db pointers into BerkeleyDatabase
This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.