Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix] wallet: Fix duplicate fileid detection #14320

Merged

Conversation

Projects
None yet
7 participants
@ken2812221
Copy link
Member

commented Sep 25, 2018

The implementation in current master can not detect if the file ID is duplicate with flushed BerkeleyEnvironment. This PR would store the file ID in a global variable g_fileids and release it when the BerkeleyDatabase close. So it won't have to rely on a Db*.

Fix #14304

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 25, 2018

@ryanofsky
Copy link
Contributor

left a comment

Great catch, I was surprised to see this bug. It seems like the problem here is that BerkeleyEnvironment::Flush actually fully unloads databases, so the check for duplicate fileids when opening a new database doesn't work at all after flushing.

I wonder if you could make a python test case out of the steps in #14304.

src/wallet/db.cpp Outdated
struct BerkeleyFileid {
u_int8_t value[DB_FILE_ID_LEN];
};
std::unordered_map<std::string, BerkeleyFileid> g_fileids;

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

I think it'd be more correct to add a std::unordered_map<std::string, BerkeleyFileid> m_fileids member in BerkeleyEnvironment member instead of a g_fileids global because there can be duplicate filenames if the files are in different directories.

Also, making this a member would be consistent with existing mapFileUseCount mapDb members there which are also maps indexed by filename. In the future it could be nice to consolidate the 3 maps.

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Sep 25, 2018

Author Member

Updated

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 25, 2018

@DrahtBot

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2018

Reviewers, this pull request conflicts with the following ones:
  • #14531 (Replace fs::relative call with custom GetRelativePath by promag)
  • #11911 (Free BerkeleyEnvironment instances when not in use by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

src/wallet/db.cpp Outdated
const char* item_filename = nullptr;
item.second->get_dbname(&item_filename, nullptr);
for (const auto& item : env.m_fileids) {
if (item.first != filename && memcmp(fileid.value, item.second.value, sizeof(fileid)) == 0) {

This comment has been minimized.

Copy link
@promag

promag Sep 25, 2018

Member

move memcmp to BerkeleyFileid::operator==?

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Sep 25, 2018

Author Member

done

src/wallet/db.cpp Outdated
}
}

env.m_fileids.emplace(filename, fileid);

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

Since CheckUniqueFileid is called on all BerkeleyEnvironment instances, this will incorrectly add the filename to unrelated BerkeleyEnvironment instances that don't actually contain filename. I'd suggest dropping this line, adding a BerkeleyFileid& fileid output parameter to CheckUniqueFileid, and passing in this->env->m_fileids[strFilename] where CheckUniqueFileid is called in BerkeleyBatch::BerkeleyBatch.

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Sep 25, 2018

Author Member

done

src/fs.cpp Outdated
@@ -89,7 +89,7 @@ bool FileLock::TryLock()
return false;
}
_OVERLAPPED overlapped = {0};
if (!LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY, 0, 0, 0, &overlapped)) {
if (!LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY, 0, -1, -1, &overlapped)) {

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

In commit "util: Fix broken Windows file lock"

Can you describe how the lock is broken in the commit message?

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Sep 25, 2018

Author Member

That was introduced by #13862. I assume that it could work with length 0 if it is a exclusive lock, but it doesn't. I would have to specify -1, -1 to make it work.

The boost implementation:

https://github.com/boostorg/interprocess/blob/8b3621353017aa527a22124655c9458d0b64b358/include/boost/interprocess/detail/os_file_functions.hpp#L230-L243

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 25, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch 2 times, most recently Sep 25, 2018

@@ -826,6 +826,8 @@ void BerkeleyDatabase::Flush(bool shutdown)
LOCK(cs_db);

This comment has been minimized.

Copy link
@ken2812221

ken2812221 Sep 25, 2018

Author Member

Note that if two or more BerkeleyDatabase share the same BerkeleyEnvironment, the env would be a wild pointer after the first call.

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

Good catch. This appears to be a bug introduced in 0b82bac from #13111 and then tweaked in a769461 in #12493. It seems not directly related to this issue, so probably best addressed separately.

This comment has been minimized.

Copy link
@promag

promag Oct 24, 2018

Member

Good catch indeed. The bug was not introduced in 0b82bac but in a769461?

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Oct 24, 2018

Contributor

Thread #14320 (comment)

Good catch indeed. The bug was not introduced in 0b82bac but in a769461?

0b82bac added the call to erase the environment too early and a769461 just moved the call, IIUC.

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 25, 2018

src/wallet/db.cpp Outdated
@@ -503,8 +503,8 @@ BerkeleyBatch::BerkeleyBatch(BerkeleyDatabase& database, const char* pszMode, bo
// be implemented, so no equality checks are needed at all. (Newer
// versions of BDB have an set_lk_exclusive method for this
// purpose, but the older version we use does not.)
for (const auto& env : g_dbenvs) {
CheckUniqueFileid(env.second, strFilename, *pdb_temp);
for (auto& env : g_dbenvs) {

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

I think you could restore auto& env to const auto& env here.

src/wallet/db.cpp Outdated
const char* item_filename = nullptr;
item.second->get_dbname(&item_filename, nullptr);
for (const auto& item : env.m_fileids) {
if (item.first != filename && fileid == item.second) {

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

I think item.first != filename condition makes the check too loose again because the same filename can exist in different directories (and we'd expect pretty much all wallet directories to have a wallet.dat file).

Perhaps change condition to: if (item.second == fileid && !(&item.second == &fileid))

src/wallet/db.cpp Outdated
@@ -56,6 +51,11 @@ CCriticalSection cs_db;
std::map<std::string, BerkeleyEnvironment> g_dbenvs GUARDED_BY(cs_db); //!< Map from directory name to open db environment.
} // namespace

bool BerkeleyFileid::operator==(const BerkeleyFileid& b) const
{
return memcmp(this, &b, sizeof(BerkeleyFileid)) == 0;

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

It might be safer if memcmp compared just the value field rather than the whole object, since it seems possible the compiler could pad the objects. Maybe:

return memcmp(this->field, b.field, sizeof(this->field)) == 0;
@@ -826,6 +826,8 @@ void BerkeleyDatabase::Flush(bool shutdown)
LOCK(cs_db);

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Sep 25, 2018

Contributor

Good catch. This appears to be a bug introduced in 0b82bac from #13111 and then tweaked in a769461 in #12493. It seems not directly related to this issue, so probably best addressed separately.

@fanquake fanquake added the Wallet label Sep 25, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 26, 2018

@ryanofsky
Copy link
Contributor

left a comment

utACK cb4af1969c03b56bc18b63286627922db86743d2. Thanks for the fix and all the followup!

Show resolved Hide resolved src/wallet/db.cpp

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 26, 2018

@ken2812221 ken2812221 changed the title wallet: Fix duplicate fileid detection [bugfix] wallet: Fix duplicate fileid detection Sep 27, 2018

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Sep 27, 2018

Is this for backport?

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Sep 27, 2018

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2018

Is this for backport?

Could be backported, but I think the benefit would be very minimal. I think you have to literally copy a wallet database file and use the loadwallet RPC in order to trigger the missing error check that this PR restores.

@ryanofsky
Copy link
Contributor

left a comment

utACK f936b8d14bfa8c758984430181db20514d34accd. Only change since previous review is adding code comment.

@DrahtBot

This comment has been minimized.

Copy link
Contributor

commented Sep 28, 2018

Coverage Change (pull 14320) Reference (master)
Lines +0.0019 % 87.0361 %
Functions -0.0049 % 84.1130 %
Branches -0.0008 % 51.5451 %

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Oct 21, 2018

@ken2812221

This comment has been minimized.

Copy link
Member Author

commented Oct 21, 2018

Rebased

@DrahtBot DrahtBot removed the Needs rebase label Oct 21, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch from ef16fc5 Oct 24, 2018

@ken2812221 ken2812221 force-pushed the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch to 4ea7732 Oct 24, 2018

@ken2812221

This comment has been minimized.

Copy link
Member Author

commented Oct 24, 2018

ef16fc5 -> 4ea7732

  1. Rename BerkeleyFileid -> WalletDatabaseFileId (@promag)
  2. Added @ryanofsky's comments
@ryanofsky
Copy link
Contributor

left a comment

utACK 4ea7732. Only changes are fileid class rename and todo comment.

@promag

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

Tested ACK 4ea7732, new test fails without this fix.

@laanwj

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

utACK 6241eb3

@laanwj laanwj merged commit 4ea7732 into bitcoin:master Oct 24, 2018

0 of 2 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

laanwj added a commit that referenced this pull request Oct 24, 2018

Merge #14320: [bugfix] wallet: Fix duplicate fileid detection
4ea7732 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
2d796fa wallet: Fix duplicate fileid (Chun Kuan Lee)

Pull request description:

  The implementation in current master can not detect if the file ID is duplicate with flushed `BerkeleyEnvironment`. This PR would store the file ID in a global variable `g_fileids` and release it when the `BerkeleyDatabase` close. So it won't have to rely on a `Db*`.

  Fix #14304

Tree-SHA512: 0632254b696bb4c671b5e2e5781e9012df54ba3c6ab0f919d9f6d31f374d3b0f8bd968b90b537884ac8c3d2906afdd58c2ce258666263464c7dbd636960b0e8f

@ken2812221 ken2812221 deleted the ken2812221:2018-09-25-wallet-duplicate-fileid-fix branch Oct 24, 2018

MarcoFalke added a commit to MarcoFalke/bitcoin that referenced this pull request Oct 24, 2018

Merge bitcoin#14559: appveyor: Enable multiwallet tests
4dca7d0 appveyor: Enable multiwallet test (Chun Kuan Lee)

Pull request description:

  Based on bitcoin#14320

  This PR enable multiwallet test on appveyor. Also re-enable symlink tests on Windows which is available after Windows Vista.

  I disable these tests in bitcoin#13964 because I suppose that Windows does not support symlink, but I was wrong.

Tree-SHA512: 852cd4dedf36ec9c34aff8926cb34e6a560aea0bb9170c7a2264fc292dbb605622d561568d8df39aeb90d3d2bb700901d218ea7e7c5e21d84827c40d6370b369

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

@promag promag referenced this pull request Mar 11, 2019

Merged

0.17: Backport 15297 #15575

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

promag added a commit to promag/bitcoin that referenced this pull request Mar 11, 2019

LitecoinZ added a commit to litecoinz-core/devrepo that referenced this pull request Mar 17, 2019

Backport bitcoin#15297
- bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
- bitcoin#14350 Add WalletLocation class
- bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
- bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
- bitcoin#11911 Free BerkeleyEnvironment instances when not in use
- bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

laanwj added a commit that referenced this pull request Mar 20, 2019

Merge #15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - #12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - #14350 Add WalletLocation class
   - #14320 [bugfix] wallet: Fix duplicate fileid detection
   - #14552 wallet: detecting duplicate wallet by comparing the db filename.
   - #11911 Free BerkeleyEnvironment instances when not in use
   - #15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 28, 2019

Merge bitcoin#15575: 0.17: Backport 15297
fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

uhliksk added a commit to fxtc/fxtc that referenced this pull request Apr 21, 2019

Rishabh42 added a commit to Rishabh42/syscoin that referenced this pull request Apr 22, 2019

Dev 4.x (#8)
* Revert "more dbg"

This reverts commit c114420.

* Revert "more dbg"

This reverts commit 20d7fa1.

* Revert "more dbg"

This reverts commit f2a4621.

* Revert "logging"

This reverts commit 067d813.

* Revert "more dbg"

This reverts commit f2a4621.

* remove dbg

* remove dbg

* add 500 byte check in rpc prepare

* wip on new ban policy

* rpc: Make unloadwallet wait for complete wallet unload

Github-Pull: bitcoin#14941
Rebased-From: c37851d

* Remove errant past from walletcreatefundedpsbt for nLocktime replaceability

Github-Pull: bitcoin#15213
Rebased-From: 85f0ca9

* Update zmq to 4.3.1

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6250

Github-Pull: bitcoin#15188
Rebased-From: 3046e5f

* [0.17] [Doc] Backport release note about PSBT doc

* add pingretries to cmasternode

* start masternode preenabled and switch to enabled on ping check from broadcast

* assetallocationbalance rpc call

* have aamany vs just aa rpc

* display retries

* reset ping retries once successful ping and also ensure mn is in winnerslist

if mn is in winnerlist + 6 signatures a ping is accepted, if mn is in winners list between 0 and 6 signatures it wont be valid but it will not DOS ban peer, if  you are outside of the list entirely you should ban the peer.

The idea is that once you get into the winners list and have enough signatures its ok to start pinging. This avoids the race condition of pingers being banned and being accepted because thre is a delay between having 0 and 6 signatures, outside of the winners list

* change to 2 to test

* remove unused vars

* accept first ping on new broadcast, dont check winners list for that

* dont expire if lastPing is empty

* move enabled after checks and fix fSentinelIsCurrent

* move force check after enabled set

* reset retries only if mn enabled

* remove ping expired/expired states, either your enabled or not

* remove calls to isexpired and issentinelpingexpired

* idb check removed from miner

* rmove status print

* remove sentinel ping activation checks

* comment code out TODO put it back after test

* dont send ping unless in winners list with 6 sigs

* comment out ping

* enabled by default until theres a ping

* add sentinel ping expired state back

* refactor

* move preenabled check above

* relay if sentinel expired

* brackets around logic

* remove dbg

* ping from sentinelping for test

* some dbg

* allow pinging during preenabled state

* Revert "some dbg"

This reverts commit e6b06fe.

* dont increment npingretries if ourmn

* fix sentinelping call, call check every 10 min

* fix common lib

* move corewrite to server

* run mnchecks before calls that use the information

* is mn is not synced set to preenabled

* ensure we lock main first in istransactionvalid

* move pingretries to masternode info

* set pingretries to mnb list

* store ping retries in setmnping

* check retries from broadcast

* use references not copy

* some helpful debug

* dbg

* remove dbg

* allow force check on masternodemanager::Check

pass false by default but pass in true in the rpc calls and istransactionvalid. Don't update npingretries if passed in true, only when its not forced do we update that variable to avoid adding it multiple times

* set sentinel valid by default on broadcast

* set max retries to 60 and maintenance to every min

* only check ping in winners list if winners list is synced

* check winnerslist on sending ping only if list is synced

* serialize ping retries

* manage local ping retries

* check entire list is synced not just mn list

* startup preenabled for 10s

* remove dbg

* send ping only if not preenabled and is synced

* only check masternodes if they are in the winnerslist (or not enabled)

* output for not being in winners list in check

* use adjusted time instead of lastping for preenabled check

* use function already created IsBroadcastedWithin

* future sig time check fix

* give atleast MASTERNODE_SENTINEL_PING_MAX_SECONDS/2 seconds to propogate

* update to MASTERNODE_MIN_MNB_SECONDS

* also adjust broadcast check

* flip logic

* fix switching from pre to enabled state

* const references

* remove mn cache's

* update proto and hrp

* set sigs to 6 again

* update get to 1.8.23

* Updated relayer to 1.0.4

* geth default data dir in syscoin data dir

* pull out boot nodes

* Updated relayer to 1.0.4

* add net specific to data dir

* remove non-witness enforcement in mempool

this was taken out before but we missed this check, non-witness should be allowed

* update min proto

* skip check on empty winners list

* show active time from gettime if no ping has been processed yet

* check for multiple instances of daemon properly

* fix result check on pidof syscoin call result

* check asset consensus checks during mempool validation

* fix tests

* rmv no wallet build

* fix build

* remove graph from lint

* Removed graph and regex from libboost

* fix build

* fix build

* fix build

* typo

* remove main lock on istransactionvalid

the main lock in masternode::check is avoided from istransactionvalid because fForce=true and we only check isoutpoint which locks main in check() is fForce==false

* compile

* expire masternode sanity check if ping retries >= max

* compile

* dbg for getblockchaininfo

* remove potential for deadlock scenarios

* remove retry param

* process mncheck first time on startup

* better process killing

* implement our asset index spec part 1

* Bumped Masternode major version

* finish asset index spec

* add listassetindexassets RPC

* 3.8gb min

* flipped logic suppoed to end firsttime

* add block hash txid mapping to checksyscoininputs

* asset index configuration parameters

* check index on listasset rpc call

* check assets rpc call aswell

* check for assetindex before using it to init db

* default to page 0

* compile

* read txids regardless if doesnt exist

* compile

* write to asset page when changing or initializing

* remove double booking of asset index, refactor into standard functions and reuse

fix missing assetindextxid in the mintsyscoin indexing routine

* fix listassetallocation logic

* re-org txid structures so they don't double up during forks

* order LIFO on results

* refactor reorg of asset index

1) only erase payload
2) when scanning if payload doesn't exist, continue
this means that some pages may not be full (that is the tradeoff but its a performance based tradeoff). The alternatve is to remove TXIDs but that is more intensive because you have to remove for every receiver/sender which requires one to scan back pages to find txid.

* add erase txid on disconnect

we still have to rollback txid's on disconnect, may be slow but for now its required

* remove unneeded code

* remove potential deadlocking scenario with updatelastpaid/check

* remove potential deadlocks from mn rpc calls

updatelastpaid has cs_main locking inside but locks another cs before that, so we have to ensure we lock main first

* minor code cleanup

* dec page if empty

* ensure page doesnt overflow

* skip check() if not in winners list instead of enabled

also ping retries only if not preenabled to avoid cases where preenabled mn's start to count up since we are allowing non enabled checks now to happen in check() we need to ensure that the mn is in the right state before ping retries is incremented.

* fix while loop issue by changing page to int64_t

* only increase ping retries if votes >= 6 and on winners list, otherwise check every mn every 60 seconds

* remove def versions not needed

* fixed typo

* check for >= 0 votes not >= 6 for pings

* fix serialization of daemon version

* remove ban scenario

* Revert "check for >= 0 votes not >= 6 for pings"

This reverts commit a6c2d79.

* update major mn version

* Revert "remove ban scenario"

This reverts commit ff53b31.

* update version

* make new mn qualify after 4 rounds

* update version and genesis hash

* reset mn version

* masternode payment updates

1) on istransactionvalid, where before it would have failed and halted processing block, we also let it go if the paid masternode is NSR
2) don't allow any NSR votes to be accepted (which helps 1 from people trying to vote for themselves even though they are NSR)
3) in check() allow up to 49 ping retries while 6 or more votes are required in the winners list for a masternode and no pings detected in last 10 minutes. Once it gets to 50 or more, it wont check for the winners list and will update every minute if no pings are detected for a masternode until 60, then NSR state ensues

* non enabled mn cannot ping

* remove node1

* update genesis and proto

* increae sleep between kills

* message for signing

* start in preenabled state

* process msgs while geth isn't synced

* accept connections unless mn and not geth synced

* watch for first time geth is synced

* Patched relayer

* Revert "Patched relayer"

This reverts commit aea436f.

* remove min proto

* update versions

* dont do auxpow for genesis

* fix collateral proposal tx check by using new gettx function with blockindex

removes need for txindex, previous version using accessbytxid wasn't working properly because for opreturn outputs it wouln't return true and if any change outputs were spent then accessbytxid on a burn tx would actually return "cant find collateral tx"

* strip out syscoin stuff from scriptPubKey before adding to vin

* fix witness related functionality

syscointxfund wouldnt add the owner output if it found a witness output since the tx fee calc would have been satisfied, instead we start our calculation from 0 everytime so it adds the outputs we need from the owner of the asset (the address passed in).

We also dont need to do any stripping as scantxoutset doesn't give us the syscoin outputs

* remove log output

* update chain params

* refactor custom outputs in favor of tx versioning

remove custom outputs and use tx versioning to understand syscoin txs. This removes an output for every syscoin tx so its more efficient and also more compatible with bitcoin related integrations like electrum which parses transactions to sign, because no custom outputs it wont need a handler to parse syscoin transactions.

Also refactored burning of sys/assets. For assets do not use cassetallocation serialized into opreturn but put the data needed to validate the burn in opreturn manually and recreate cassetallocation when the transaction is deserialized.

* Updated relayer to 1.0.5

* fix bugs

* fix assetsend/assetallocationsend and burn issues

* Add function to close all Db's and reload the databae environment

Adds a ReloadDbEnv function to BerkeleyEnvironment in order to close all Db
instances, closes the environment, resets it, and then reopens
the BerkeleyEnvironment.

Also adds a ReloadDbEnv function to BerkeleyDatabase that calls
BerkeleyEnvironment's ReloadDbEnv.

Github-Pull: bitcoin#12493
Rebased-From: 5d296ac

* After encrypting the wallet, reload the database environment

Calls ReloadDbEnv after encrypting the wallet so that the database
environment is flushed, closed, and reopened to prevent unencrypted
keys from being saved on disk.

Github-Pull: bitcoin#12493
Rebased-From: d7637c5

* Move BerkeleyEnvironment deletion from internal method to callsite

Instead of having the object destroy itself, having the caller
destroy it.

Github-Pull: bitcoin#12493
Rebased-From: a769461

* No longer shutdown after encrypting the wallet

Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.

Github-Pull: bitcoin#12493
Rebased-From: c1dde3a

* wallet: Add WalletLocation utility class

Github-Pull: bitcoin#14350
Rebased-From: 01a4c09

* wallet: Refactor to use WalletLocation

Github-Pull: bitcoin#14350
Rebased-From: 65f3672

* wallet: Fix duplicate fileid

Github-Pull: bitcoin#14320
Rebased-From: 2d796fa

* tests: add test case for loading copied wallet twice

Github-Pull: bitcoin#14320
Rebased-From: 4ea7732

* wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory.

Github-Pull: bitcoin#14552
Rebased-From: 15c93f0

* Refactor: Move m_db pointers into BerkeleyDatabase

This is a refactoring change that doesn't affect behavior. The motivation
behind the change is give BerkeleyEnvironment objects access to
BerkeleyDatabase objects so it will be possible to simplify the duplicate
wallet check and more reliably avoid opening the same databases twice.

Github-Pull: bitcoin#14552
Rebased-From: c456fbd

* wallet: Create IsDatabaseLoaded function

Github-Pull: bitcoin#14552
Rebased-From: 5912031

* Free BerkeleyEnvironment instances when not in use

Instead of adding BerkeleyEnvironment objects permanently to the g_dbenvs map,
use reference counted shared pointers and remove map entries when the last
BerkeleyEnvironment reference goes out of scope.

This change was requested by Matt Corallo <git@bluematt.me> and makes code that
sets up mock databases cleaner. The mock database environment will now go out
of scope and be reset on destruction so there is no need to call
BerkeleyEnvironment::Reset() during wallet construction to clear out prior
state.

This change does affect bitcoin behavior slightly. On startup, instead of same
wallet environments staying open throughout VerifyWallets() and OpenWallets()
calls, VerifyWallets() will open and close an environment once for each wallet,
and OpenWallets() will create its own environment(s) later.

Github-Pull: bitcoin#11911
Rebased-From: f1f4bb7

* Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment

Github-Pull: bitcoin#11911
Rebased-From: 14bc2a1

* Tests: add unit tests for GetWalletEnv

Github-Pull: bitcoin#11911
Rebased-From: 88b1d95

* wallet: Close dbenv error file db.log

The error file db.log is opened by BerkeleyEnvironment instance and
should be closed after dbenv is closed.

Github-Pull: bitcoin#15297
Rebased-From: 8602a1e

* wallet: Close wallet env lock file

Close .walletlock file when a BerkeleyEnvironment is deleted.

Github-Pull: bitcoin#15297
Rebased-From: 2f8b8f4

* qa: Test .walletlock file is closed

Github-Pull: bitcoin#15297
Rebased-From: d3bf3b9

* Merge bitcoin#15575: 0.17: Backport 15297

fe95f84 qa: Test .walletlock file is closed (João Barbosa)
2e9e904 wallet: Close wallet env lock file (João Barbosa)
22cdb6c wallet: Close dbenv error file db.log (João Barbosa)
f20513b Tests: add unit tests for GetWalletEnv (Pierre Rochard)
85c6263 Trivial: add doxygen-compatible comments relating to BerkeleyEnvironment (Pierre Rochard)
f22d02f Free BerkeleyEnvironment instances when not in use (Russell Yanofsky)
0a9af2d wallet: Create IsDatabaseLoaded function (Chun Kuan Lee)
7751ea3 Refactor: Move m_db pointers into BerkeleyDatabase (Russell Yanofsky)
caf1146 wallet: Add trailing wallet.dat when detecting duplicate wallet if it's a directory. (Chun Kuan Lee)
34da2b7 tests: add test case for loading copied wallet twice (Chun Kuan Lee)
8965b6a wallet: Fix duplicate fileid (Chun Kuan Lee)
16e5759 wallet: Refactor to use WalletLocation (João Barbosa)
21693ff wallet: Add WalletLocation utility class (João Barbosa)
1c98a75 No longer shutdown after encrypting the wallet (Andrew Chow)
435df68 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
048fda2 After encrypting the wallet, reload the database environment (Andrew Chow)
f455979 Add function to close all Db's and reload the databae environment (Andrew Chow)

Pull request description:

  This PR backports the following pull requests:
   - bitcoin#12493 [wallet] Reopen CDBEnv after encryption instead of shutting down
   - bitcoin#14350 Add WalletLocation class
   - bitcoin#14320 [bugfix] wallet: Fix duplicate fileid detection
   - bitcoin#14552 wallet: detecting duplicate wallet by comparing the db filename.
   - bitcoin#11911 Free BerkeleyEnvironment instances when not in use
   - bitcoin#15297 wallet: Releases dangling files on BerkeleyEnvironment::Close

Tree-SHA512: 52d759bc4f140ca96e39b37746cc20e786741b08ddc658a87ea77fbcfbb481f1c7b75aba4fc57ca9bca8ca7154e535da1fdd650fd114873655cd85c490c79f14

* depends: Enable unicode support on dbd for Windows

* Log progress while verifying blocks at level 4.

When verifying blocks at startup, the progress is printed in 10%
increments to logs.  When -checklevel=4, however, the second half
of the verification (connecting the blocks again) does not log the
progress anymore.  (It is still computed and shown in the UI, but
not printed to logs.)

This change makes the behaviour consistent, by adding the missing
progress logging also for level-4 checks.

* Hide spendable label if priveate key is disabled

* index: Fix for indexers skipping genesis block.

* compile

* remove files

* fix parsing to support current contract format

* use blockindex to index blocks per txid

* tx version based assets

* restructure spv proof call to give list of tx's and index of tx because the internal proof isnt a merkle proof, used bitcoin-proof instead

* reorg strategy around burn sender in mint asset tx

on mint, the sender is set to burn address, remove from burn when minting and ensuring balance > 0

on disconnect, the sender is set to burn address, add to burn when disconnecting

* minor bug fixes

dont send mn related sync commands to older clients.
isstd for burn txs
serialize tx through syscoingetspvproof as non witness

* allow for easy way to call assetssend and allocationsend without object

* fix assetupdate stripping contract and fix assettxtojson

* update contract

* remove burn sig + fix ethereum abi parsing

burn signature not needed as the function signature is always the same.
ethereum abi parsing was broken waiting for final form, now that we know what that is we hook it up and fix the parsing once and for all.

* dont adjust total sipply on mints/burns

* remove powf

* Removed regtest test files for node2 and node3

* fix order of params

* fix rlp data decode to bytes on parseethmethod check

also set max headers to 1 week based on light chain syncing roughly 1 week usually
prune headers on every set call as well to avoid big db

* working spv proofs

* Fix make install

* remove wallet dependency on sys rpc functions

everything except syscointxfund

* miner fixes

* wait till geth is synced before validating

* remove availablecointypes

we don't need to worry about specific masternode amount types since we now have multi wallet support, masternodes should be set up on a seperate wallet if they want to keep their utxos intact and not spend accidentily

* revert unintentional change

* check if geth synced before mint in rpc

* dont write to indexes if miner mode or just checking

since miner does a sanity run around syscoin consensus on txs  we actually want to make sure no database writes happen during this time, just the normal balance maps which are thrown away after checksyscoininputs is finished.

* renamed bins

* move wallet masternode stuff out of cwallet

* move sys consensus code to seperate file

* update header

* remove multiple outputs and wallet check for small sys inputs

* Fixed win64 bin names

* move static decl's out of header, causing issues

put in cpp files it seems it was declaring multiple times causing a test to fail regularily

convert sorted_vector to unsorted_set for standards

* only do concurrent processing for zdag transactions (allocation send)

* save 3*MAX_ETHEREUM_TX_ROOTS before pruning and check txroot if exists

we want to validate txroot if it exists, likely your node will have up to 120k txroots (last 3 weeks approx) so when you turn node off and on it should process all txroots of last 3 weeks if it finds them.

once you are synced up it should enforce mint transactions be a week(40k  eth blocks) or under otherwise its not allowed.

this fixes the case where a txroot may be invalid and then you turn node off and on and it skips validation, it should not proceed if txroot is invalid. When syncing up pull the txroot from db and validate it if it exists, if it doesn't exist AND your already synced then it should return an error.

* rmv files

* use blockindex to find collateral tx in governance proposal validation code

this means we can use existing sys3 process for proposals not breaking syshub

* rework blockindex

-blockindex is done by default, not a config option
   - this is because governance propsoal validator code needs it now, might as well make it default
-put it in its own db
-write to it using it batch processing for performance since its usually multiple writes at a time

* remove user syscointxfund requirement

this is now built into the rpc calls (except syscoinburn where it doesnt know what address to fund it from).

removed all 3000 satoshi hardcoded dust outputs in various calls, it will find the first usable output and spend that for proof of ownership

* remove txfund req from syscoinburn

we pass in funding address into syscoinburn and remove syscointxfund step for user

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

wallet: Fix duplicate fileid
Github-Pull: bitcoin#14320
Rebased-From: 2d796fa

uhliksk added a commit to FxTC-Bata-development/Bataoshi that referenced this pull request May 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.