New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obfuscate chainstate #6650

Merged
merged 1 commit into from Oct 6, 2015

Conversation

Projects
None yet
@jamesob
Member

jamesob commented Sep 7, 2015

Fixes #6613 (discussion here: #4069)

Obfuscate the leveldb-stored chainstate with a simple XOR to avoid spurious detection by anti-virus software.

Chainstate obfuscation will happen atomically, i.e. either all or none of the database will be obfuscated. A runtime error will be thrown if a user attempts to invoke bitcoind -obfuscatechaindata with existing, unobfuscated chainstate data.

On reindexing, all new chainstates will be obfuscated. until then, we specify a degenerate obfuscation key (\000\000...) that has no effect when XOR'd with existing data.

TODO after Concept ACK

  • write tests covering CDataStream.Xor
  • write tests covering obfuscation ops with CLevelDBWrapper
@sipa

View changes

Show outdated Hide outdated src/txdb.cpp
@@ -40,7 +40,7 @@ void static BatchWriteHashBestChain(CLevelDBBatch &batch, const uint256 &hash) {
batch.Write(DB_BEST_BLOCK, hash);
}
CCoinsViewDB::CCoinsViewDB(size_t nCacheSize, bool fMemory, bool fWipe) : db(GetDataDir() / "chainstate", nCacheSize, fMemory, fWipe) {
CCoinsViewDB::CCoinsViewDB(size_t nCacheSize, bool fMemory, bool fWipe) : db(GetDataDir() / "chainstate", nCacheSize, fMemory, fWipe, GetArg("-obfuscatechainstate", 0)) {

This comment has been minimized.

@sipa

sipa Sep 8, 2015

Member

Use GetBoolArg, so it enables the -no prefix etc.

@sipa

sipa Sep 8, 2015

Member

Use GetBoolArg, so it enables the -no prefix etc.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 8, 2015

Member

Concept ACK, this looks pretty good already.

Member

sipa commented Sep 8, 2015

Concept ACK, this looks pretty good already.

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Sep 8, 2015

Member

Concept and approach ACK. Obviously this will need a fair amount of testing. (In particular we should get an AV afflicted system and confirm this fixes it).

I think it should be on by default. The people who need this won't know they need it, anyone who doesn't need it will quickly figure it out. (Do other people have opinions?)

I'm not even sure if we should support turning it off? It's easy enough to support (as your code shows) that maybe we just should; but I'd like to hear an argument for it.

Thank you for working on this.

Member

gmaxwell commented Sep 8, 2015

Concept and approach ACK. Obviously this will need a fair amount of testing. (In particular we should get an AV afflicted system and confirm this fixes it).

I think it should be on by default. The people who need this won't know they need it, anyone who doesn't need it will quickly figure it out. (Do other people have opinions?)

I'm not even sure if we should support turning it off? It's easy enough to support (as your code shows) that maybe we just should; but I'd like to hear an argument for it.

Thank you for working on this.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 8, 2015

Member

@gmaxwell The only argument against supporting unobfuscated is 1) it would need upgrading code (but I think that's pretty simple, though supporting stop/restart in the middle is not trivial) 2) it would break backward compatibility.

Member

sipa commented Sep 8, 2015

@gmaxwell The only argument against supporting unobfuscated is 1) it would need upgrading code (but I think that's pretty simple, though supporting stop/restart in the middle is not trivial) 2) it would break backward compatibility.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 8, 2015

Member

@gmaxwell I agree that having the obfuscation enabled is a sane (and safe) default, but is it acceptable for that runtime_error to be thrown on upgrade for everyone with an existing chainstate? Seems harsh. Is it desirable to find a more gentle way of doing the migration?

Member

jamesob commented Sep 8, 2015

@gmaxwell I agree that having the obfuscation enabled is a sane (and safe) default, but is it acceptable for that runtime_error to be thrown on upgrade for everyone with an existing chainstate? Seems harsh. Is it desirable to find a more gentle way of doing the migration?

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Sep 8, 2015

Member

This is why supporting mixed obfuscated/unobfuscated is nice: no slow upgrade process. :)

Member

luke-jr commented Sep 8, 2015

This is why supporting mixed obfuscated/unobfuscated is nice: no slow upgrade process. :)

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 8, 2015

Member

@luke-jr I do think the mixed migration is a better option if we're going to enforce this (or even make it the default behavior). Will implement if @gmaxwell, @laanwj agree.

Member

jamesob commented Sep 8, 2015

@luke-jr I do think the mixed migration is a better option if we're going to enforce this (or even make it the default behavior). Will implement if @gmaxwell, @laanwj agree.

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 8, 2015

Contributor

As noted in #6613 (comment) a mix of cleartext and obfu-text does not actually fix the problem seem in the field for active users.

Users reporting this problem will still see the problem after upgrading to a release with this feature.

Contributor

jgarzik commented Sep 8, 2015

As noted in #6613 (comment) a mix of cleartext and obfu-text does not actually fix the problem seem in the field for active users.

Users reporting this problem will still see the problem after upgrading to a release with this feature.

@pstratem

View changes

Show outdated Hide outdated src/leveldbwrapper.cpp
std::string CLevelDBWrapper::CreateObfuscateKey() const {
unsigned char buff[8];
GetRandBytes(buff, 8);
return std::string((const char *)buff);

This comment has been minimized.

@pstratem

pstratem Sep 8, 2015

Contributor

This should be std::string(&buff[0], &buff[8]);

@pstratem

pstratem Sep 8, 2015

Contributor

This should be std::string(&buff[0], &buff[8]);

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 8, 2015

Member

I've integrated @gmaxwell's suggestion and I think it's looking much better.

Member

jamesob commented Sep 8, 2015

I've integrated @gmaxwell's suggestion and I think it's looking much better.

@dcousens

View changes

Show outdated Hide outdated src/leveldbwrapper.h
*/
bool IsEmpty()
{
leveldb::Iterator* it = NewIterator();

This comment has been minimized.

@dcousens

dcousens Sep 8, 2015

Contributor

Does this have to be allocated?

@dcousens

dcousens Sep 8, 2015

Contributor

Does this have to be allocated?

This comment has been minimized.

@jamesob

jamesob Sep 8, 2015

Member

Is there an alternative? apologies, I'm a C++ beginner :)

@jamesob

jamesob Sep 8, 2015

Member

Is there an alternative? apologies, I'm a C++ beginner :)

This comment has been minimized.

@jamesob

jamesob Sep 9, 2015

Member

Yeah, looks like this only comes allocated.

@jamesob

jamesob Sep 9, 2015

Member

Yeah, looks like this only comes allocated.

This comment has been minimized.

@dcousens

dcousens Sep 9, 2015

Contributor

Thanks for clearing that up @jamesob

@dcousens

dcousens Sep 9, 2015

Contributor

Thanks for clearing that up @jamesob

This comment has been minimized.

@laanwj

laanwj Sep 29, 2015

Member

Please use a boost::scoped_ptr<leveldb::Iterator> here, like in txdb.cpp, to automatically free the iterator when it goes out of scope.

@laanwj

laanwj Sep 29, 2015

Member

Please use a boost::scoped_ptr<leveldb::Iterator> here, like in txdb.cpp, to automatically free the iterator when it goes out of scope.

@pstratem

View changes

Show outdated Hide outdated src/leveldbwrapper.h
@@ -44,6 +44,8 @@ class CLevelDBBatch
CDataStream ssValue(SER_DISK, CLIENT_VERSION);
ssValue.reserve(ssValue.GetSerializeSize(value));
ssValue << value;
if (!obfuscate_key.empty())

This comment has been minimized.

@pstratem

pstratem Sep 8, 2015

Contributor

You can drop the conditional.

@pstratem

pstratem Sep 8, 2015

Contributor

You can drop the conditional.

@pstratem

View changes

Show outdated Hide outdated src/leveldbwrapper.h
@@ -34,7 +34,7 @@ class CLevelDBBatch
public:
template <typename K, typename V>
void Write(const K& key, const V& value)
void Write(const K& key, const V& value, const std::string& obfuscate_key = std::string())

This comment has been minimized.

@pstratem

pstratem Sep 8, 2015

Contributor

And the default.

@pstratem

pstratem Sep 8, 2015

Contributor

And the default.

This comment has been minimized.

@jamesob

jamesob Sep 8, 2015

Member

I initially did that, but this is called from contexts where obfuscate_key isn't in scope and hasn't been defined, namely CBlockTreeDB::WriteTxIndex. Now that I'm looking at it, there're some missing parameterizations of obfuscate_key (e.g. here).

It isn't immediately obvious to me how to convey db.obfuscate_key to that context... might need some refactoring.

@jamesob

jamesob Sep 8, 2015

Member

I initially did that, but this is called from contexts where obfuscate_key isn't in scope and hasn't been defined, namely CBlockTreeDB::WriteTxIndex. Now that I'm looking at it, there're some missing parameterizations of obfuscate_key (e.g. here).

It isn't immediately obvious to me how to convey db.obfuscate_key to that context... might need some refactoring.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 8, 2015

Member

CLevelDBBatch needs a good refactoring at some point... unless there's something I'm not seeing, it should probably be absorbed into CLevelDBWrapper.That may be out of scope for this PR, though.

Member

jamesob commented Sep 8, 2015

CLevelDBBatch needs a good refactoring at some point... unless there's something I'm not seeing, it should probably be absorbed into CLevelDBWrapper.That may be out of scope for this PR, though.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 8, 2015

Member

Now that I think about it, the Batch refactoring may be more contained than I thought. Anyone have qualms with me doing it in this PR?

Member

jamesob commented Sep 8, 2015

Now that I think about it, the Batch refactoring may be more contained than I thought. Anyone have qualms with me doing it in this PR?

@dexX7

View changes

Show outdated Hide outdated src/leveldbwrapper.cpp
}
Write(OBFUSCATE_KEY_KEY, obfuscate_key);
LogPrintf("Wrote new obfuscate key '%s'\n", obfuscate_key);

This comment has been minimized.

@dexX7

dexX7 Sep 8, 2015

Contributor

Hmm.. obfuscate_key is not necessarily printable. Maybe it would be better to log a hexified version of the key?

@dexX7

dexX7 Sep 8, 2015

Contributor

Hmm.. obfuscate_key is not necessarily printable. Maybe it would be better to log a hexified version of the key?

@dexX7

View changes

Show outdated Hide outdated src/leveldbwrapper.h
CLevelDBBatch* batch = GetBatch();
batch->Write(key, value);
bool written = WriteBatch(batch, fSync);
delete batch;

This comment has been minimized.

@dexX7

dexX7 Sep 8, 2015

Contributor

As far as I can see it's not guaranteed that batch is deleted, namely if WriteBatch -> HandleError -> throw.

How about using something like boost::scoped_ptr to avoid the manual cleanup altogether?

@dexX7

dexX7 Sep 8, 2015

Contributor

As far as I can see it's not guaranteed that batch is deleted, namely if WriteBatch -> HandleError -> throw.

How about using something like boost::scoped_ptr to avoid the manual cleanup altogether?

This comment has been minimized.

@jamesob

jamesob Sep 8, 2015

Member

Sounds good to me, though honestly I'm hoping to obviate the heap allocations entirely by absorbing CLevelDBBatch into CLevelDBWrapper... this is pending an okay from @sipa @gmaxwell.

@jamesob

jamesob Sep 8, 2015

Member

Sounds good to me, though honestly I'm hoping to obviate the heap allocations entirely by absorbing CLevelDBBatch into CLevelDBWrapper... this is pending an okay from @sipa @gmaxwell.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 9, 2015

Member

Okay, I think this is in a good place. Obfuscation now happens automatically whenever the chainstate is initialized (either via -reindex or otherwise). The degenerate key that we use ("\000\000...") isn't persisted, but it's specified during class construction in the case that we can't do real obfuscate due to existing data.

IMO this is ready to go, modulo the addition of a few unittests.

Member

jamesob commented Sep 9, 2015

Okay, I think this is in a good place. Obfuscation now happens automatically whenever the chainstate is initialized (either via -reindex or otherwise). The degenerate key that we use ("\000\000...") isn't persisted, but it's specified during class construction in the case that we can't do real obfuscate due to existing data.

IMO this is ready to go, modulo the addition of a few unittests.

@jamesob jamesob closed this Sep 9, 2015

@jamesob jamesob reopened this Sep 9, 2015

@dexX7

View changes

Show outdated Hide outdated src/leveldbwrapper.h
template <typename K, typename V>
bool Read(const K& key, V& value) const throw(leveldb_error)
bool Read(const K& key, V& value, bool unobfuscate = true) const throw(leveldb_error)

This comment has been minimized.

@dexX7

dexX7 Sep 9, 2015

Contributor

Is there any other case where unobfuscate == false, except in the constructor? If not, you might get away without it (and the condition in L140), because obfuscate_key is still blank during the read of the actual key, if I'm not mistaken.

@dexX7

dexX7 Sep 9, 2015

Contributor

Is there any other case where unobfuscate == false, except in the constructor? If not, you might get away without it (and the condition in L140), because obfuscate_key is still blank during the read of the actual key, if I'm not mistaken.

This comment has been minimized.

@jamesob

jamesob Sep 9, 2015

Member

Ah yep, you're totally right. Initially I rationalized this by telling myself this param would be convenient for tests, but looks like there's no real need for it now. Will push a fix.

@jamesob

jamesob Sep 9, 2015

Member

Ah yep, you're totally right. Initially I rationalized this by telling myself this param would be convenient for tests, but looks like there's no real need for it now. Will push a fix.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 10, 2015

Member

OK, unittests here are done. If anyone would like to see additional testcases, let me know, but I think I've covered the two important cases:

  1. No existing chainstate/ data (obfuscation)
  2. Existing data (no-op obfuscation)

It'd be great if we could get some testers who are affected by anti-virus software (@KyrosKrane ?) trying out this changeset.

Member

jamesob commented Sep 10, 2015

OK, unittests here are done. If anyone would like to see additional testcases, let me know, but I think I've covered the two important cases:

  1. No existing chainstate/ data (obfuscation)
  2. Existing data (no-op obfuscation)

It'd be great if we could get some testers who are affected by anti-virus software (@KyrosKrane ?) trying out this changeset.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 12, 2015

I'd be more than happy to test, but I'm new to testing with Bitcoin off Github. Do I need to download and compile from source?

KyrosKrane commented Sep 12, 2015

I'd be more than happy to test, but I'm new to testing with Bitcoin off Github. Do I need to download and compile from source?

@fanquake

This comment has been minimized.

Show comment
Hide comment
@fanquake

fanquake Sep 12, 2015

Member

@KyrosKrane if your comfortable with, and are able to compile from source then the github-merge script in /contrib/devtools is an easy way to test pull requests. Once you've cloned the repo, cd in to the directory and run

contrib/devtools/github-merge.sh 6650

That will pull in the code changes from this pull request, and you can then compile as normal.

Otherwise, @jonasschnelli has gitian buillt binaries for certain pull requests available from https://builds.jonasschnelli.ch/pulls/. He can probably spin up a build for this pull once he gets a chance.

Member

fanquake commented Sep 12, 2015

@KyrosKrane if your comfortable with, and are able to compile from source then the github-merge script in /contrib/devtools is an easy way to test pull requests. Once you've cloned the repo, cd in to the directory and run

contrib/devtools/github-merge.sh 6650

That will pull in the code changes from this pull request, and you can then compile as normal.

Otherwise, @jonasschnelli has gitian buillt binaries for certain pull requests available from https://builds.jonasschnelli.ch/pulls/. He can probably spin up a build for this pull once he gets a chance.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 12, 2015

Thanks for the build; that will make this much faster for me. :)

Here's what I've done so far.

  1. Backed up my existing bitcoin config directory.
  2. Backed up my existing bitcoin exe files.
  3. Downloaded the test build from @jonasschnelli 's site
  4. Deleted everything in my config dir other than the basic config file (just has my connection info and settings)
  5. Replaced the exes in my Program Files directory with the exes from the test build.
  6. Removed the antivirus exclusion from the bitcoin config directory.
  7. Started the test exe.

I'm now waiting for it to download the blockchain (I have a second node on my local network, so it shouldn't be too slow) and rebuild the chainstate and index files. Once that completes, I'll do a manual scan of the config dir and see if Norton yells at me.

KyrosKrane commented Sep 12, 2015

Thanks for the build; that will make this much faster for me. :)

Here's what I've done so far.

  1. Backed up my existing bitcoin config directory.
  2. Backed up my existing bitcoin exe files.
  3. Downloaded the test build from @jonasschnelli 's site
  4. Deleted everything in my config dir other than the basic config file (just has my connection info and settings)
  5. Replaced the exes in my Program Files directory with the exes from the test build.
  6. Removed the antivirus exclusion from the bitcoin config directory.
  7. Started the test exe.

I'm now waiting for it to download the blockchain (I have a second node on my local network, so it shouldn't be too slow) and rebuild the chainstate and index files. Once that completes, I'll do a manual scan of the config dir and see if Norton yells at me.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 12, 2015

Member

@KyrosKrane that sounds great, thanks for your help.

Member

jamesob commented Sep 12, 2015

@KyrosKrane that sounds great, thanks for your help.

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 12, 2015

Member

@KyrosKrane If you have the time, could you also test with the "infected" data directory, verify that by just running the current bitcoin core version does not fix the AV alert?
Afterward, you could check if a -reindex fixes the AV problem on the "infected" data directory.

Member

MarcoFalke commented Sep 12, 2015

@KyrosKrane If you have the time, could you also test with the "infected" data directory, verify that by just running the current bitcoin core version does not fix the AV alert?
Afterward, you could check if a -reindex fixes the AV problem on the "infected" data directory.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 12, 2015

You read my mind. :) That's exactly what I'm going to do when this current run ends. It will probably be tomorrow, though; the re-download is taking longer than I thought, even though my other node is locally connected.

KyrosKrane commented Sep 12, 2015

You read my mind. :) That's exactly what I'm going to do when this current run ends. It will probably be tomorrow, though; the re-download is taking longer than I thought, even though my other node is locally connected.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 13, 2015

OK, update for today. The re-download completed overnight. No antivirus alerts. I manually scanned the bitcoin data dir. No alerts. This is as expected.

Next, I stopped bitcoin-qt.exe and renamed the data directory. I copied my original (unobfuscated) blocks and chainstate folders, and the bitcoin.conf file from my backup into the now-empty data directory. I tested it by scanning that freshly-copied directory - no antivirus alert. This isn't good. It should be alerting at this point.

I tried renaming the files from *.ldb to *.sst, but that didn't help. I even tried renaming them to *.exe, which got Norton to actually scan the files in depth, but still no alerts.

I'm going to continue testing the debug build to see if it behaves as expected.

KyrosKrane commented Sep 13, 2015

OK, update for today. The re-download completed overnight. No antivirus alerts. I manually scanned the bitcoin data dir. No alerts. This is as expected.

Next, I stopped bitcoin-qt.exe and renamed the data directory. I copied my original (unobfuscated) blocks and chainstate folders, and the bitcoin.conf file from my backup into the now-empty data directory. I tested it by scanning that freshly-copied directory - no antivirus alert. This isn't good. It should be alerting at this point.

I tried renaming the files from *.ldb to *.sst, but that didn't help. I even tried renaming them to *.exe, which got Norton to actually scan the files in depth, but still no alerts.

I'm going to continue testing the debug build to see if it behaves as expected.

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 13, 2015

Member

Do you know if your AV "fixed" the "infected" data directory back when it originally happened?

Member

MarcoFalke commented Sep 13, 2015

Do you know if your AV "fixed" the "infected" data directory back when it originally happened?

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 13, 2015

Well, it certainly broke bitcoin-qt and caused it to crash. It wouldn't restart afterwards until I did a -reindex, which promptly broke a second time. Then I wised up and added the exclusion to my antivirus, and a second -reindex worked. I'll try doing a reindex in a bit; I want to start bitcoin-qt normally first and see if it will read the (unobfuscated) files correctly.

KyrosKrane commented Sep 13, 2015

Well, it certainly broke bitcoin-qt and caused it to crash. It wouldn't restart afterwards until I did a -reindex, which promptly broke a second time. Then I wised up and added the exclusion to my antivirus, and a second -reindex worked. I'll try doing a reindex in a bit; I want to start bitcoin-qt normally first and see if it will read the (unobfuscated) files correctly.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 13, 2015

bitcoin-qt happily opened with the unobfuscated files. I tested with an old wallet file I had, and it quickly rescanned the blockchain and updated the balance. (I now have three satoshis more than this morning, yay me!) Next, I exited bitcoin-qt, opened a command prompt, and ran bitcoin-qt.exe -reindex. This errored out. I repeated, in case this was a one time thing, and it gave me the same error. The message is "Error opening block database". I attached a screenshot.

block database error

KyrosKrane commented Sep 13, 2015

bitcoin-qt happily opened with the unobfuscated files. I tested with an old wallet file I had, and it quickly rescanned the blockchain and updated the balance. (I now have three satoshis more than this morning, yay me!) Next, I exited bitcoin-qt, opened a command prompt, and ran bitcoin-qt.exe -reindex. This errored out. I repeated, in case this was a one time thing, and it gave me the same error. The message is "Error opening block database". I attached a screenshot.

block database error

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 13, 2015

Member

@KyrosKrane can you check your log file for anything that might indicate
what went wrong, e.g. a stacktrace?

On Sunday, September 13, 2015, KyrosKrane notifications@github.com wrote:

bitcoin-qt happily opened with the unobfuscated files. I tested with an
old wallet file I had, and it quickly rescanned the blockchain and updated
the balance. (I now have three satoshis more than this morning, yay me!)
Next, I exited bitcoin-qt, opened a command prompt, and ran bitcoin-qt.exe
-reindex. This errored out. I repeated, in case this was a one time thing,
and it gave me the same error. The message is "Error opening block
database". I attached a screenshot.

[image: block database error]
https://cloud.githubusercontent.com/assets/458727/9837936/b6329116-5a60-11e5-9e59-4e0369f36dc2.png


Reply to this email directly or view it on GitHub
#6650 (comment).

Member

jamesob commented Sep 13, 2015

@KyrosKrane can you check your log file for anything that might indicate
what went wrong, e.g. a stacktrace?

On Sunday, September 13, 2015, KyrosKrane notifications@github.com wrote:

bitcoin-qt happily opened with the unobfuscated files. I tested with an
old wallet file I had, and it quickly rescanned the blockchain and updated
the balance. (I now have three satoshis more than this morning, yay me!)
Next, I exited bitcoin-qt, opened a command prompt, and ran bitcoin-qt.exe
-reindex. This errored out. I repeated, in case this was a one time thing,
and it gave me the same error. The message is "Error opening block
database". I attached a screenshot.

[image: block database error]
https://cloud.githubusercontent.com/assets/458727/9837936/b6329116-5a60-11e5-9e59-4e0369f36dc2.png


Reply to this email directly or view it on GitHub
#6650 (comment).

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 13, 2015

I'm slowing down in my old age ... I should have thought of that. Debug.log file below.

2015-09-13 17:36:27 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:36:27 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:36:27 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:36:27 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:36:27 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:36:27 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:36:27 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:36:27 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:36:27 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:36:27 Using 4 threads for script verification
2015-09-13 17:36:27 Using wallet wallet.dat
2015-09-13 17:36:27 scheduler thread start
2015-09-13 17:36:27 init message: Verifying wallet...
2015-09-13 17:36:27 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:36:27 Bound to [::]:8333
2015-09-13 17:36:27 Bound to 0.0.0.0:8333
2015-09-13 17:36:27 Cache configuration:
2015-09-13 17:36:27 * Using 2.0MiB for block index database
2015-09-13 17:36:27 * Using 32.5MiB for chain state database
2015-09-13 17:36:27 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:36:27 init message: Loading block index...
2015-09-13 17:36:27 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:36:29 scheduler thread interrupt
2015-09-13 17:36:29 Shutdown: In progress...
2015-09-13 17:36:29 StopNode()
2015-09-13 17:36:29 Shutdown: done
2015-09-13 17:37:14 



















2015-09-13 17:37:14 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:37:14 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:37:14 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:37:14 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:37:14 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:37:14 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:14 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:14 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:37:14 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:37:14 Using 4 threads for script verification
2015-09-13 17:37:14 Using wallet wallet.dat
2015-09-13 17:37:14 scheduler thread start
2015-09-13 17:37:14 init message: Verifying wallet...
2015-09-13 17:37:14 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:37:14 Bound to [::]:8333
2015-09-13 17:37:14 Bound to 0.0.0.0:8333
2015-09-13 17:37:14 Cache configuration:
2015-09-13 17:37:14 * Using 2.0MiB for block index database
2015-09-13 17:37:14 * Using 32.5MiB for chain state database
2015-09-13 17:37:14 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:37:14 init message: Loading block index...
2015-09-13 17:37:14 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:37:14 Opening LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:37:14 Opened LevelDB successfully
2015-09-13 17:37:14 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\chainstate
2015-09-13 17:37:19 scheduler thread interrupt
2015-09-13 17:37:19 Shutdown: In progress...
2015-09-13 17:37:19 StopNode()
2015-09-13 17:37:19 Shutdown: done
2015-09-13 17:37:38 



















2015-09-13 17:37:38 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:37:38 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:37:38 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:37:38 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:37:38 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:37:38 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:38 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:38 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:37:38 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:37:38 Using 4 threads for script verification
2015-09-13 17:37:38 Using wallet wallet.dat
2015-09-13 17:37:38 scheduler thread start
2015-09-13 17:37:38 init message: Verifying wallet...
2015-09-13 17:37:38 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:37:38 Bound to [::]:8333
2015-09-13 17:37:38 Bound to 0.0.0.0:8333
2015-09-13 17:37:38 Cache configuration:
2015-09-13 17:37:38 * Using 2.0MiB for block index database
2015-09-13 17:37:38 * Using 32.5MiB for chain state database
2015-09-13 17:37:38 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:37:38 init message: Loading block index...
2015-09-13 17:37:38 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:42:00 scheduler thread interrupt
2015-09-13 17:42:00 Shutdown: In progress...
2015-09-13 17:42:00 StopNode()
2015-09-13 17:42:00 Shutdown: done

The chainstate directory was deleted, as was the index directory under the blocks directory. Under my main bitcoin data dir, there was a new directory named "database" with a single file in it named log.0000000001. The contents appear binary.

KyrosKrane commented Sep 13, 2015

I'm slowing down in my old age ... I should have thought of that. Debug.log file below.

2015-09-13 17:36:27 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:36:27 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:36:27 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:36:27 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:36:27 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:36:27 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:36:27 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:36:27 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:36:27 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:36:27 Using 4 threads for script verification
2015-09-13 17:36:27 Using wallet wallet.dat
2015-09-13 17:36:27 scheduler thread start
2015-09-13 17:36:27 init message: Verifying wallet...
2015-09-13 17:36:27 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:36:27 Bound to [::]:8333
2015-09-13 17:36:27 Bound to 0.0.0.0:8333
2015-09-13 17:36:27 Cache configuration:
2015-09-13 17:36:27 * Using 2.0MiB for block index database
2015-09-13 17:36:27 * Using 32.5MiB for chain state database
2015-09-13 17:36:27 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:36:27 init message: Loading block index...
2015-09-13 17:36:27 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:36:29 scheduler thread interrupt
2015-09-13 17:36:29 Shutdown: In progress...
2015-09-13 17:36:29 StopNode()
2015-09-13 17:36:29 Shutdown: done
2015-09-13 17:37:14 



















2015-09-13 17:37:14 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:37:14 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:37:14 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:37:14 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:37:14 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:37:14 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:14 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:14 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:37:14 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:37:14 Using 4 threads for script verification
2015-09-13 17:37:14 Using wallet wallet.dat
2015-09-13 17:37:14 scheduler thread start
2015-09-13 17:37:14 init message: Verifying wallet...
2015-09-13 17:37:14 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:37:14 Bound to [::]:8333
2015-09-13 17:37:14 Bound to 0.0.0.0:8333
2015-09-13 17:37:14 Cache configuration:
2015-09-13 17:37:14 * Using 2.0MiB for block index database
2015-09-13 17:37:14 * Using 32.5MiB for chain state database
2015-09-13 17:37:14 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:37:14 init message: Loading block index...
2015-09-13 17:37:14 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:37:14 Opening LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:37:14 Opened LevelDB successfully
2015-09-13 17:37:14 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\chainstate
2015-09-13 17:37:19 scheduler thread interrupt
2015-09-13 17:37:19 Shutdown: In progress...
2015-09-13 17:37:19 StopNode()
2015-09-13 17:37:19 Shutdown: done
2015-09-13 17:37:38 



















2015-09-13 17:37:38 Bitcoin version v0.11.99.0-f1803c1 (2015-09-12 12:01:31 +0200)
2015-09-13 17:37:38 AppInit2: parameter interaction: -connect set -> setting -dnsseed=0
2015-09-13 17:37:38 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2015-09-13 17:37:38 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-09-13 17:37:38 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-13 17:37:38 Default data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:38 Using data directory D:\Users\USERNAME\AppData\Roaming\Bitcoin
2015-09-13 17:37:38 Using config file D:\Users\USERNAME\AppData\Roaming\Bitcoin\bitcoin.conf
2015-09-13 17:37:38 Using at most 125 connections (2048 file descriptors available)
2015-09-13 17:37:38 Using 4 threads for script verification
2015-09-13 17:37:38 Using wallet wallet.dat
2015-09-13 17:37:38 scheduler thread start
2015-09-13 17:37:38 init message: Verifying wallet...
2015-09-13 17:37:38 CDBEnv::Open: LogDir=D:\Users\USERNAME\AppData\Roaming\Bitcoin\database ErrorFile=D:\Users\USERNAME\AppData\Roaming\Bitcoin\db.log
2015-09-13 17:37:38 Bound to [::]:8333
2015-09-13 17:37:38 Bound to 0.0.0.0:8333
2015-09-13 17:37:38 Cache configuration:
2015-09-13 17:37:38 * Using 2.0MiB for block index database
2015-09-13 17:37:38 * Using 32.5MiB for chain state database
2015-09-13 17:37:38 * Using 65.5MiB for in-memory UTXO set
2015-09-13 17:37:38 init message: Loading block index...
2015-09-13 17:37:38 Wiping LevelDB in D:\Users\USERNAME\AppData\Roaming\Bitcoin\blocks\index
2015-09-13 17:42:00 scheduler thread interrupt
2015-09-13 17:42:00 Shutdown: In progress...
2015-09-13 17:42:00 StopNode()
2015-09-13 17:42:00 Shutdown: done

The chainstate directory was deleted, as was the index directory under the blocks directory. Under my main bitcoin data dir, there was a new directory named "database" with a single file in it named log.0000000001. The contents appear binary.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 15, 2015

Member

@KyrosKrane sorry it's taken me a bit to respond; I finally have some time to sit down and take a look. I'm going to add a testcase that simulates a -reindex at the CLevelDBWrapper layer to see if that will reproduce this failure for me locally.

Thanks again for putting in the work to test this.

Member

jamesob commented Sep 15, 2015

@KyrosKrane sorry it's taken me a bit to respond; I finally have some time to sit down and take a look. I'm going to add a testcase that simulates a -reindex at the CLevelDBWrapper layer to see if that will reproduce this failure for me locally.

Thanks again for putting in the work to test this.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Sep 17, 2015

Member

@KyrosKrane dang, can't reproduce with a unittest. Any chance that running with the -debug flag yields any more information? e.g. bitcoind -reindex -debug? If not, I'll have to try to reproduce on my end with a full dataset. As a side-note, I wonder why the behavior you're seeing isn't reflected in qa/rpc-tests/reindex.py...

Member

jamesob commented Sep 17, 2015

@KyrosKrane dang, can't reproduce with a unittest. Any chance that running with the -debug flag yields any more information? e.g. bitcoind -reindex -debug? If not, I'll have to try to reproduce on my end with a full dataset. As a side-note, I wonder why the behavior you're seeing isn't reflected in qa/rpc-tests/reindex.py...

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 18, 2015

OK, now this is weird. I ran bitcoin-qt.exe -reindex -debug, and it worked like a champ. It took almost as long as re-downloading the entire blockchain again; I had to leave it running overnight. But it didn't give any errors and it seems to have completed successfully. The debug.log file is a whopping 335MB uncompressed, so I'm not going to post that unless someone really needs it.

I'm going to restore my original, unobfuscated data directory, then try running -reindex -debug on it to see if it causes the error. If that works, I'll restore once again, then do just -reindex to see whether I get the error once more.

KyrosKrane commented Sep 18, 2015

OK, now this is weird. I ran bitcoin-qt.exe -reindex -debug, and it worked like a champ. It took almost as long as re-downloading the entire blockchain again; I had to leave it running overnight. But it didn't give any errors and it seems to have completed successfully. The debug.log file is a whopping 335MB uncompressed, so I'm not going to post that unless someone really needs it.

I'm going to restore my original, unobfuscated data directory, then try running -reindex -debug on it to see if it causes the error. If that works, I'll restore once again, then do just -reindex to see whether I get the error once more.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Sep 18, 2015

OK, I've tested this twice now. With my original, unobfuscated data directory, if I do bitcoin-qt.exe -reindex -debug, it will work successfully. If I only do bitcoin-qt.exe -reindex, it fails with the error I posted above.

KyrosKrane commented Sep 18, 2015

OK, I've tested this twice now. With my original, unobfuscated data directory, if I do bitcoin-qt.exe -reindex -debug, it will work successfully. If I only do bitcoin-qt.exe -reindex, it fails with the error I posted above.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Sep 29, 2015

Member

Looks good to me apart from the mentioned small things. Will start testing this.

Member

laanwj commented Sep 29, 2015

Looks good to me apart from the mentioned small things. Will start testing this.

@@ -168,6 +199,44 @@ class CLevelDBWrapper
{
return pdb->NewIterator(iteroptions);
}
/**

This comment has been minimized.

@laanwj

laanwj Sep 29, 2015

Member

I'd put the implementation of these functions in leveldbwrapper.cpp instead of the header file (unless there's a reason I'm overlooking)

@laanwj

laanwj Sep 29, 2015

Member

I'd put the implementation of these functions in leveldbwrapper.cpp instead of the header file (unless there's a reason I'm overlooking)

This comment has been minimized.

@jamesob

jamesob Sep 29, 2015

Member

I implemented these in the header to maintain consistency with much of the rest of CLevelDBWrapper, which is also defined here. I'll move what's below over to the .cpp file, but we should probably think about moving the rest of it out of the header at some point.

@jamesob

jamesob Sep 29, 2015

Member

I implemented these in the header to maintain consistency with much of the rest of CLevelDBWrapper, which is also defined here. I'll move what's below over to the .cpp file, but we should probably think about moving the rest of it out of the header at some point.

@laanwj

View changes

Show outdated Hide outdated src/streams.h
*
* @param[in] key The key used to XOR the data in this stream.
*/
void Xor(const std::string& key) {

This comment has been minimized.

@laanwj

laanwj Sep 29, 2015

Member

To save a % operation for every byte (which is essentially a division!) and add a sanity check for empty key, I suggest:

    void Xor(const std::string& key)
    {
        assert(key.size() > 0);
        for (size_type i = 0, j = 0; i != size(); i++) {
            vch[i] ^= key[j++];
            if (j == key.size())
                j = 0;
        }
    }
@laanwj

laanwj Sep 29, 2015

Member

To save a % operation for every byte (which is essentially a division!) and add a sanity check for empty key, I suggest:

    void Xor(const std::string& key)
    {
        assert(key.size() > 0);
        for (size_type i = 0, j = 0; i != size(); i++) {
            vch[i] ^= key[j++];
            if (j == key.size())
                j = 0;
        }
    }

This comment has been minimized.

@laanwj

laanwj Sep 29, 2015

Member

As this is task-specific ideally we wouldn't have this operation on CDataStream itself, but as a function that manipulates a CDataStream.

@laanwj

laanwj Sep 29, 2015

Member

As this is task-specific ideally we wouldn't have this operation on CDataStream itself, but as a function that manipulates a CDataStream.

This comment has been minimized.

@jamesob

jamesob Sep 30, 2015

Member

Really good suggestion, will integrate. Two questions:

  • won't a key that is "\000..." have .size() == 0? should we just return early if that's the case instead of blowing up with an assert?
  • I put this in CDataStream because, basically, it has to mutate vch. That's a protected member variable, so a method Xor became. Is there an alternate way to do this that you'd prefer?
@jamesob

jamesob Sep 30, 2015

Member

Really good suggestion, will integrate. Two questions:

  • won't a key that is "\000..." have .size() == 0? should we just return early if that's the case instead of blowing up with an assert?
  • I put this in CDataStream because, basically, it has to mutate vch. That's a protected member variable, so a method Xor became. Is there an alternate way to do this that you'd prefer?

This comment has been minimized.

@laanwj

laanwj Sep 30, 2015

Member
@laanwj

laanwj via email Sep 30, 2015

Member

This comment has been minimized.

@jamesob

jamesob Sep 30, 2015

Member

I think you misunderstand... we use a string consisting entirely of null-terminators (\000) as a default obfuscation key. Given https://coderpad.io/2GF6EKCD , the default obfuscation key would cause your code above to raise an assertion error. I went with an early return to avoid this.

@jamesob

jamesob Sep 30, 2015

Member

I think you misunderstand... we use a string consisting entirely of null-terminators (\000) as a default obfuscation key. Given https://coderpad.io/2GF6EKCD , the default obfuscation key would cause your code above to raise an assertion error. I went with an early return to avoid this.

This comment has been minimized.

@laanwj

laanwj Sep 30, 2015

Member

std::string doesn't use 'NULL terminators', they are 8-bit-clean.

Also NULL-terminated strings don't matter here at all: you should do obfuscation with a sequence of bytes.

The length of [0,0,0,0,0,0,0,0] is 8 bytes just as well as [255,255,255,255,255,255,255,255] is. If not, that's a bug.

e.g. if your obfuscation key is [1,0,1,2,3,4,5,6], you don't want to loop just ones.

@laanwj

laanwj Sep 30, 2015

Member

std::string doesn't use 'NULL terminators', they are 8-bit-clean.

Also NULL-terminated strings don't matter here at all: you should do obfuscation with a sequence of bytes.

The length of [0,0,0,0,0,0,0,0] is 8 bytes just as well as [255,255,255,255,255,255,255,255] is. If not, that's a bug.

e.g. if your obfuscation key is [1,0,1,2,3,4,5,6], you don't want to loop just ones.

This comment has been minimized.

@laanwj

laanwj Sep 30, 2015

Member

The problem is that you use eg. std::string("\000\000\000").
If you do that, C++ will use strlen() to determine the length, which looks for 0 characters.
Pass in a length explicitly.

So do either:

std::string(3, '\000');  // Create a string with three 0 bytes

Or

static const char data[] = {0,0,0};
std::string(data, data+sizeof(data));

...

Don't use the NUL-terminated string constructor for raw data that can contain 0.

@laanwj

laanwj Sep 30, 2015

Member

The problem is that you use eg. std::string("\000\000\000").
If you do that, C++ will use strlen() to determine the length, which looks for 0 characters.
Pass in a length explicitly.

So do either:

std::string(3, '\000');  // Create a string with three 0 bytes

Or

static const char data[] = {0,0,0};
std::string(data, data+sizeof(data));

...

Don't use the NUL-terminated string constructor for raw data that can contain 0.

This comment has been minimized.

@jamesob

jamesob Sep 30, 2015

Member

Ah, I see now. Thanks for typing up the examples on coderpad. This is more of my C++ inexperience coming out :).

@jamesob

jamesob Sep 30, 2015

Member

Ah, I see now. Thanks for typing up the examples on coderpad. This is more of my C++ inexperience coming out :).

This comment has been minimized.

@jamesob

jamesob Sep 30, 2015

Member

Do you think it still makes sense to do an early exit if the value of the key amounts to a char array of 0s and thus will result in a noop for the xor?

@jamesob

jamesob Sep 30, 2015

Member

Do you think it still makes sense to do an early exit if the value of the key amounts to a char array of 0s and thus will result in a noop for the xor?

This comment has been minimized.

@laanwj

laanwj Sep 30, 2015

Member

Do you think it still makes sense to do an early exit if the value of the key amounts to a char array of 0s and thus will result in a noop for the xor?

Well I really don't mind in this case.
In general, if something is always a coding bug, and could result in dangerous situations if left unchecked, it needs to be asserted.
But having Xor("") as a no-op seems reasonable to me too.

@laanwj

laanwj Sep 30, 2015

Member

Do you think it still makes sense to do an early exit if the value of the key amounts to a char array of 0s and thus will result in a noop for the xor?

Well I really don't mind in this case.
In general, if something is always a coding bug, and could result in dangerous situations if left unchecked, it needs to be asserted.
But having Xor("") as a no-op seems reasonable to me too.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Sep 30, 2015

Member

BTW; seems to work great. I have encountered no problems while testing this, either with an old chainstate or after a reindex (also tested with pruning).

Member

laanwj commented Sep 30, 2015

BTW; seems to work great. I have encountered no problems while testing this, either with an old chainstate or after a reindex (also tested with pruning).

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 1, 2015

Member

ACK after squash.

I do think we need to mention this concisely in release_notes.md:

  • a database built on 0.12+ cannot be used with older versions, but downgrading an existing install is not a problem
  • third-party tools that parse the UTXO database (I don't know of any, I have https://github.com/laanwj/blockdb-troubleshoot but that only reads the block db) have to be updated
Member

laanwj commented Oct 1, 2015

ACK after squash.

I do think we need to mention this concisely in release_notes.md:

  • a database built on 0.12+ cannot be used with older versions, but downgrading an existing install is not a problem
  • third-party tools that parse the UTXO database (I don't know of any, I have https://github.com/laanwj/blockdb-troubleshoot but that only reads the block db) have to be updated
@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Oct 1, 2015

Contributor

We should also describe, how to continue having unobfuscated database...

Contributor

paveljanik commented Oct 1, 2015

We should also describe, how to continue having unobfuscated database...

@dexX7

This comment has been minimized.

Show comment
Hide comment
@dexX7

dexX7 Oct 1, 2015

Contributor

Given that there were several issues with using std::string so far, I'm wondering, whether something else may be used instead, such as std::vector<unsigned char>, to prevent potential pitfalls altogether?

Contributor

dexX7 commented Oct 1, 2015

Given that there were several issues with using std::string so far, I'm wondering, whether something else may be used instead, such as std::vector<unsigned char>, to prevent potential pitfalls altogether?

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Oct 1, 2015

Member

@paveljanik any new chainstate databases (including those built via -reindex) will be obfuscated; it's not optional. See discussion above.

@dexX7 I think that's a great suggestion, and I've been inclined to try something like that. @laanwj are you in favor of making obfuscate_key a std::vector<unsigned char> in lieu of a std::string? I think it'll make things much less awkward.

Member

jamesob commented Oct 1, 2015

@paveljanik any new chainstate databases (including those built via -reindex) will be obfuscated; it's not optional. See discussion above.

@dexX7 I think that's a great suggestion, and I've been inclined to try something like that. @laanwj are you in favor of making obfuscate_key a std::vector<unsigned char> in lieu of a std::string? I think it'll make things much less awkward.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Oct 1, 2015

Member

@jamesob I'm in favor of turning it into a std::vector<unsigned char>. No reason why it would be a string.

Member

sipa commented Oct 1, 2015

@jamesob I'm in favor of turning it into a std::vector<unsigned char>. No reason why it would be a string.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Oct 1, 2015

Member

obfuscate_key is now a vector<unsigned char>, which feels much better. If no one else has any outstanding issues, I'll squash.

Member

jamesob commented Oct 1, 2015

obfuscate_key is now a vector<unsigned char>, which feels much better. If no one else has any outstanding issues, I'll squash.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 1, 2015

Member

making it a std::vector<unsigned char> (or equivalently, std::vector<uint8_t>) is a great idea, it avoids the initializer issue completely

Member

laanwj commented Oct 1, 2015

making it a std::vector<unsigned char> (or equivalently, std::vector<uint8_t>) is a great idea, it avoids the initializer issue completely

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Oct 1, 2015

Member

Squashed.

Member

jamesob commented Oct 1, 2015

Squashed.

@dexX7

This comment has been minimized.

Show comment
Hide comment
@dexX7

dexX7 Oct 1, 2015

Contributor

I tested the obfuscation in regtest mode, and started with unobfuscated 0.10 DBs.

It works mostly as expected, though I faced a few potential issues:

  • the block index database isn't obfuscated (probably not needed?)
  • client hung/froze after the first reindexing (this was not reproducible)
  • after that, the DBs were corrupted, potentially due to the freeze + forced shutdown (also not reproducible)
  • the first time I disabled the txindex with an obfuscated chainstate, the DB was appearingly in a bad state (not not reproducible)

I took some notes here: https://gist.github.com/dexX7/d757485228b3ef7e37e9

Contributor

dexX7 commented Oct 1, 2015

I tested the obfuscation in regtest mode, and started with unobfuscated 0.10 DBs.

It works mostly as expected, though I faced a few potential issues:

  • the block index database isn't obfuscated (probably not needed?)
  • client hung/froze after the first reindexing (this was not reproducible)
  • after that, the DBs were corrupted, potentially due to the freeze + forced shutdown (also not reproducible)
  • the first time I disabled the txindex with an obfuscated chainstate, the DB was appearingly in a bad state (not not reproducible)

I took some notes here: https://gist.github.com/dexX7/d757485228b3ef7e37e9

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Oct 1, 2015

Member

@dexX7 thanks for the detailed notes; very helpful.

the block index database isn't obfuscated (probably not needed?)

Yeah this is by design, and per discussion in the issue thread.

not not reproducible

Is that a typo, or do you actually mean they were reproducible issues?

Member

jamesob commented Oct 1, 2015

@dexX7 thanks for the detailed notes; very helpful.

the block index database isn't obfuscated (probably not needed?)

Yeah this is by design, and per discussion in the issue thread.

not not reproducible

Is that a typo, or do you actually mean they were reproducible issues?

@dexX7

This comment has been minimized.

Show comment
Hide comment
@dexX7

dexX7 Oct 1, 2015

Contributor

Sorry, this was probably a bit confusing! :)

I noticed each issue only once, and unfortunally (or fortunally? hehe) I was not able to reproduce them. All subsequent attempts went fine, and I successfully disabled and enabled the -txindex multiple times. I also reindexed more than once unobfuscated 0.10 based databases with the patched client.

Contributor

dexX7 commented Oct 1, 2015

Sorry, this was probably a bit confusing! :)

I noticed each issue only once, and unfortunally (or fortunally? hehe) I was not able to reproduce them. All subsequent attempts went fine, and I successfully disabled and enabled the -txindex multiple times. I also reindexed more than once unobfuscated 0.10 based databases with the patched client.

@jamesob

This comment has been minimized.

Show comment
Hide comment
@jamesob

jamesob Oct 1, 2015

Member

May be similar to what @KyrosKrane was reporting earlier, though I have to admit I'm mostly stumped. I thought it may have something to do with Reading obfuscate_keys that haven't been written out to leveldb (in the case of the block index db), but after looking at the code I really doubt that's the case. If anyone has guesses for likely trouble spots, I'm glad to push more changes.

Member

jamesob commented Oct 1, 2015

May be similar to what @KyrosKrane was reporting earlier, though I have to admit I'm mostly stumped. I thought it may have something to do with Reading obfuscate_keys that haven't been written out to leveldb (in the case of the block index db), but after looking at the code I really doubt that's the case. If anyone has guesses for likely trouble spots, I'm glad to push more changes.

@KyrosKrane

View changes

Show outdated Hide outdated src/leveldbwrapper.h
~CLevelDBWrapper();
/**
* @param[in] unobfuscate This should always be true unless we're reading

This comment has been minimized.

@KyrosKrane

KyrosKrane Oct 2, 2015

Is this comment still correct? I don't see this parameter referenced anywhere.

@KyrosKrane

KyrosKrane Oct 2, 2015

Is this comment still correct? I don't see this parameter referenced anywhere.

This comment has been minimized.

@jamesob

jamesob Oct 5, 2015

Member

Good catch; will remove.

@jamesob

jamesob Oct 5, 2015

Member

Good catch; will remove.

This comment has been minimized.

@jamesob

jamesob Oct 6, 2015

Member

Fixed; thanks!

@jamesob

jamesob Oct 6, 2015

Member

Fixed; thanks!

@CodeShark

This comment has been minimized.

Show comment
Hide comment
@CodeShark

CodeShark Oct 4, 2015

Contributor

utACK

Contributor

CodeShark commented Oct 4, 2015

utACK

Add chainstate obfuscation to avoid spurious antivirus detection
Adds an `obfuscate` parameter to `CLevelDBWrapper` and makes use of it
for all new chainstate stores built via `CCoinsViewDB`. Also adds an
`Xor` method to `CDataStream`.

Thanks to @sipa @laanwj @pstratem @dexX7 @KyrosKrane @gmaxwell.
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 6, 2015

Member

I went over the code a few times and cannot find any changes that would cause it to hang or get in an infinite loop.
Also haven't been able to reproduce any such behavior.
So I'm just going ahead and merge this.

Member

laanwj commented Oct 6, 2015

I went over the code a few times and cannot find any changes that would cause it to hang or get in an infinite loop.
Also haven't been able to reproduce any such behavior.
So I'm just going ahead and merge this.

@laanwj laanwj merged commit 42cb388 into bitcoin:master Oct 6, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Oct 6, 2015

Merge pull request #6650
42cb388 Add chainstate obfuscation to avoid spurious antivirus detection (James O'Beirne)

@jamesob jamesob deleted the jamesob:obfuscate_chainstate branch Oct 6, 2015

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Oct 7, 2015

Member

@KyrosKrane [https://github.com/bitcoin/bitcoin/pull/6650#issuecomment-141416946]: OK, I've tested this twice now. With my original, unobfuscated data directory, if I do bitcoin-qt.exe -reindex -debug, it will work successfully. If I only do bitcoin-qt.exe -reindex, it fails with the error I posted above.

Can you still reproduce with bitcoin/master? Maybe this should go into a new issue then.

Member

MarcoFalke commented Oct 7, 2015

@KyrosKrane [https://github.com/bitcoin/bitcoin/pull/6650#issuecomment-141416946]: OK, I've tested this twice now. With my original, unobfuscated data directory, if I do bitcoin-qt.exe -reindex -debug, it will work successfully. If I only do bitcoin-qt.exe -reindex, it fails with the error I posted above.

Can you still reproduce with bitcoin/master? Maybe this should go into a new issue then.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Oct 7, 2015

@MarcoFalke I used the test build from here:

https://bitcoin.jonasschnelli.ch/nightlybuilds/2015-10-07/
File name bitcoin-0.11.99-win64.zip

The behavior has now changed. -reindex still crashes as before, but now so does -reindex -debug. This is new.

Just to make sure this isn't because of some error in my process, I want to document how I got to this point. I started with my normal production environment, which I got by using the Windows x64 setup file. I ran this, completed the installation normally, started bitcoin-qt using the Start menu shortcut, and downloaded the blockchain normally, until it was all updated. Then I completely exited bitcoin-qt.exe. I downloaded the test build from the links in this thread and unzipped them. Then I just replaced the exe files in my program files directory - bitcoin-qt.exe, bitcoind.exe, and bitcoin-cli.exe. I left everything else unchanged. Then I opened a command window, switched to the bitcoin directory under program files, and ran the executable from the command line as above.

If it would be helpful, I can upload my data directory somewhere, minus the wallet file. I'm zipping it up now, and I'll see if I can find a file upload site that can take a file that big.

KyrosKrane commented Oct 7, 2015

@MarcoFalke I used the test build from here:

https://bitcoin.jonasschnelli.ch/nightlybuilds/2015-10-07/
File name bitcoin-0.11.99-win64.zip

The behavior has now changed. -reindex still crashes as before, but now so does -reindex -debug. This is new.

Just to make sure this isn't because of some error in my process, I want to document how I got to this point. I started with my normal production environment, which I got by using the Windows x64 setup file. I ran this, completed the installation normally, started bitcoin-qt using the Start menu shortcut, and downloaded the blockchain normally, until it was all updated. Then I completely exited bitcoin-qt.exe. I downloaded the test build from the links in this thread and unzipped them. Then I just replaced the exe files in my program files directory - bitcoin-qt.exe, bitcoind.exe, and bitcoin-cli.exe. I left everything else unchanged. Then I opened a command window, switched to the bitcoin directory under program files, and ran the executable from the command line as above.

If it would be helpful, I can upload my data directory somewhere, minus the wallet file. I'm zipping it up now, and I'll see if I can find a file upload site that can take a file that big.

@KyrosKrane

This comment has been minimized.

Show comment
Hide comment
@KyrosKrane

KyrosKrane Oct 10, 2015

I'm throwing up the file into my Dropbox now, if anyone wants it. This is my bitcoin data directory, with minimal changes (wallet removed, conf file modified to change my RPC username and password). It's about 33GB, so it may take a bit of time to finish uploading. You can download it here.

https://www.dropbox.com/s/k4zfs7f6qmevvbu/Bitcoin%20backup%202015-10.7z?dl=0

KyrosKrane commented Oct 10, 2015

I'm throwing up the file into my Dropbox now, if anyone wants it. This is my bitcoin data directory, with minimal changes (wallet removed, conf file modified to change my RPC username and password). It's about 33GB, so it may take a bit of time to finish uploading. You can download it here.

https://www.dropbox.com/s/k4zfs7f6qmevvbu/Bitcoin%20backup%202015-10.7z?dl=0

@dexX7

This comment has been minimized.

Show comment
Hide comment
@dexX7

dexX7 Oct 15, 2015

Contributor

I really appreciate your effort, @KyrosKrane! Unfortunally, and to be honest, I'm not sure where to go from here. I saw DB corruption once too, while testing this PR, but I'm not able to reproduce it.

Since you mentioned -reindex causes the crashes, I'm wondering, whether this is also the case with a version without obfuscation? As far as I can see the nightly build from 2015-10-06 should be the latest nightly build before this PR was merged.

If this turns out to be the case, then it's an issue with the master, and not with the obfuscation, which would be good to know.

Contributor

dexX7 commented Oct 15, 2015

I really appreciate your effort, @KyrosKrane! Unfortunally, and to be honest, I'm not sure where to go from here. I saw DB corruption once too, while testing this PR, but I'm not able to reproduce it.

Since you mentioned -reindex causes the crashes, I'm wondering, whether this is also the case with a version without obfuscation? As far as I can see the nightly build from 2015-10-06 should be the latest nightly build before this PR was merged.

If this turns out to be the case, then it's an issue with the master, and not with the obfuscation, which would be good to know.

laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 31, 2015

Add chainstate obfuscation to avoid spurious antivirus detection
Adds an `obfuscate` parameter to `CLevelDBWrapper` and makes use of it
for all new chainstate stores built via `CCoinsViewDB`. Also adds an
`Xor` method to `CDataStream`.

Thanks to @sipa @laanwj @pstratem @dexX7 @KyrosKrane @gmaxwell.

Conflicts:
	src/Makefile.test.include

Github-Pull: #6650
Rebased-From: 42cb388

kazcw added a commit to kazcw/bitcoin that referenced this pull request Apr 22, 2016

Enable leveldb obfuscation for new databases
This was implemented some time ago in #6650 to fix #6613 but does not appear
ever to have been enabled under any circumstances outside of test/

zkbot added a commit to zcash/zcash that referenced this pull request Jan 15, 2018

Auto merge of #2598 - str4d:2074-dbwrapper, r=<try>
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.

litecoinz-project added a commit to litecoinz-project/litecoinz that referenced this pull request Mar 15, 2018

zkbot added a commit to zcash/zcash that referenced this pull request Apr 3, 2018

Auto merge of #2598 - str4d:2074-dbwrapper, r=str4d
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.

zkbot added a commit to zcash/zcash that referenced this pull request Apr 3, 2018

Auto merge of #2598 - str4d:2074-dbwrapper, r=str4d
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment