Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unobfuscate chainstate data in CCoinsViewDB::GetStats #6777

Merged
merged 4 commits into from Oct 13, 2015

Conversation

@jamesob
Copy link
Member

jamesob commented Oct 8, 2015

Per the thread on the mailing list, we missed (at least) one use of CLevelDBWrapper when adding chainstate obfuscation.

Preferably, this PR (or a followup) will also add automated tests that prevent future bugs of this kind. Subsequently we should also introduce an abstraction that prevents this sort of leak when performing iteration with CLevelDBWrapper.

cc @domob1812 @dexX7

@dexX7
Copy link
Contributor

dexX7 commented Oct 8, 2015

Tested, "gettxoutsetinfo" now works as expected.

Based on a quick scan for value() it looks like the only other place where raw iterator data is used, is related to the block index DB: src/txdb.cpp#L205 (which isn't obfuscated)

@jamesob
Copy link
Member Author

jamesob commented Oct 8, 2015

@dexX7 thanks for testing. I'm going to go ahead and apply the same treatment to the block index DB as well so that we won't have an issue if we decide to obfuscate it in the future.

@sipa
Copy link
Member

sipa commented Oct 8, 2015

@jamesob
Copy link
Member Author

jamesob commented Oct 8, 2015

Okay @sipa, I'm thinking I'll just subclass leveldb's Iterator for the purposes of CLevelDBWrapper, then return that type from NewIterator.

@sipa
Copy link
Member

sipa commented Oct 8, 2015

@jamesob
Copy link
Member Author

jamesob commented Oct 8, 2015

@sipa ha! and like magic... wish writing code was always this easy ;). Will cherry-pick and layer on modifications if possible.

@laanwj
Copy link
Member

laanwj commented Oct 8, 2015

Good catch @domob1812 , thanks for fix @jamesob @sipa

I'm getting an issue starting bitcoind on my node w/ obfuscation, though:
assertion "hashPrevBlock == view.GetBestBlock()" failed: file "main.cpp", line 1694, function "ConnectBlock" Abort trap (core dumped)

Not sure whether it's related to this, will investigate further. From debug.log:

2015-10-08 14:05:08 Using obfuscation key for /home/orion/.bitcoin/chainstate: 0000000000000000
...
2015-10-08 14:05:19 LoadBlockIndexDB: transaction index disabled
2015-10-08 14:05:20 Initializing databases...

Looks like my obfuscation key was lost, and it's now trying to interpret the data in raw.
debug.log unfortunately doesn't go back far enough to check what the key was before.

@laanwj
Copy link
Member

laanwj commented Oct 8, 2015

Found the issue. Indeed has nothing to do with this pull. On my node I was using a previous version of your patch, using a different OBFUSCATE_KEY_KEY. So no, the key was not lost, just misplaced. This shouldn't affect anyone starting with master.

 // Prefixed with null character to avoid collisions with other keys
-const std::string CLevelDBWrapper::OBFUSCATE_KEY_KEY = "\000obfuscate_key";
+//
+// We must use a string constructor which specifies length so that we copy
+// past the null-terminator.
+const std::string CLevelDBWrapper::OBFUSCATE_KEY_KEY("\000obfuscate_key", 14);
@laanwj
Copy link
Member

laanwj commented Oct 8, 2015

Tested ACK. gettxoutsetinfo works again after this.

@jamesob
Copy link
Member Author

jamesob commented Oct 8, 2015

@laanwj good to hear. Thanks for testing.

@jamesob
Copy link
Member Author

jamesob commented Oct 8, 2015

I've added some test coverage on the CLevelDB utilities. Provided travis is happy, I'm done adding changes.

@jamesob jamesob force-pushed the jamesob:obfuscate_fix branch 2 times, most recently Oct 8, 2015
sipa and others added 2 commits Oct 8, 2015
@jamesob jamesob force-pushed the jamesob:obfuscate_fix branch Oct 8, 2015
@dexX7
dexX7 reviewed Oct 9, 2015
View changes
qa/rpc-tests/blockchain.py Outdated
u'total_amount': decimal.Decimal('8725.00000000'),
u'transactions': 200,
u'height': 200,
u'bestblock': u'6189e9cc58bedca8cb8e917cefe831839d296c2d12ae2e460066aa38d4a06f3e',

This comment has been minimized.

Copy link
@dexX7

dexX7 Oct 9, 2015

Contributor

Is this always deterministic?

This comment has been minimized.

Copy link
@dexX7

dexX7 Oct 9, 2015

Contributor

Just to clarify: unfortunally it's not (due to the potentially different time to create the blocks + random address generation).

Even if it were, I'm not sure, if such a strong coupling is desired. You might setup the chain or a few blocks manually and use util.set_node_times + pre-generated key-pairs, but this seems pretty laborious.

A middle ground could be to check, whether the JSON keys are available (as in your previous version), and then do some sanity checks, such as res['height'] >= 0, len(res['bestblock']) == 64, [...].

@jamesob
Copy link
Member Author

jamesob commented Oct 9, 2015

@sipa any comments here?

@sipa

This comment has been minimized.

Copy link

sipa commented on src/leveldbwrapper.h in 0fdf8c8 Oct 9, 2015

No need to copy the obfuscation key, you can just store a pointer, as it's immutable data from the database object itself, so it won't disappear before the iterator is gone.

This comment has been minimized.

Copy link
Owner Author

jamesob replied Oct 10, 2015

Done, thanks.

@sipa
Copy link
Member

sipa commented Oct 9, 2015

Code ACK, I haven't reviewed the tests.

@jamesob jamesob force-pushed the jamesob:obfuscate_fix branch to dcd8e27 Oct 9, 2015
@jamesob
Copy link
Member Author

jamesob commented Oct 10, 2015

Ping on this. I think it's ready for merge.

@laanwj
Copy link
Member

laanwj commented Oct 13, 2015

@jamesob Yes going to merge this now, I've been sidetracked by the upnp vulnerabilty the last few days.

@laanwj laanwj merged commit dcd8e27 into bitcoin:master Oct 13, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
laanwj added a commit that referenced this pull request Oct 13, 2015
dcd8e27 Refer to obfuscate_key via pointer in peripheral CLevelDB classes (James O'Beirne)
1488506 Add tests for gettxoutsetinfo, CLevelDBBatch, CLevelDBIterator (James O'Beirne)
0fdf8c8 Handle obfuscation in CLevelDBIterator (James O'Beirne)
3499ce1 Encapsulate CLevelDB iterators cleanly (Pieter Wuille)
laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 31, 2015
Conflicts:
	src/leveldb.cpp
	src/leveldb.h
	src/txdb.cpp

Github-Pull: bitcoin#6777
Rebased-From: 3499ce1
laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 31, 2015
Github-Pull: bitcoin#6777
Rebased-From: 0fdf8c8
laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 31, 2015
Thanks @dexX7.

Conflicts:
	qa/pull-tester/rpc-tests.py

Github-Pull: bitcoin#6777
Rebased-From: 1488506
laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 31, 2015
zkbot added a commit to zcash/zcash that referenced this pull request Jan 15, 2018
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.
litecoinz-project pushed a commit to litecoinz-project/litecoinz that referenced this pull request Mar 15, 2018
Cherry-picked from bitcoin/bitcoin#6777
zkbot added a commit to zcash/zcash that referenced this pull request Apr 3, 2018
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.
zkbot added a commit to zcash/zcash that referenced this pull request Apr 3, 2018
Bitcoin 0.12+ dbwrapper improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6650
  - Only refactor - excludes obfuscation
- bitcoin/bitcoin#6777
  - Excluding obfuscation-related changes
- bitcoin/bitcoin#6865
- bitcoin/bitcoin#6823
- bitcoin/bitcoin#6873
- bitcoin/bitcoin#7927
  - Excluding first commit (already included) and second commit (obfuscation-related)
- bitcoin/bitcoin#8467

Part of #2074.
random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request May 27, 2020
82f9088 Refactor: Remove using namespace <xxx> from /dbwrapper_tests (random-zebra)
6db0b37 rpc: make `gettxoutsettinfo` run lock-free (random-zebra)
f009cf4 Do not shadow members in dbwrapper (random-zebra)
b7e540c dbwrapper: Move `HandleError` to `dbwrapper_private` (random-zebra)
43004d0 leveldbwrapper file rename to dbwrapper.* (random-zebra)
c882dd9 leveldbwrapper symbol rename: Remove "Level" from class, etc. names (random-zebra)
f6496da leveldbwrapper: Remove unused .Prev(), .SeekToLast() methods (random-zebra)
cacf3c2 Fix chainstate serialized_size computation (random-zebra)
a2a3d33 Add tests for CLevelDBBatch, CLevelDBIterator (random-zebra)
94150ac Encapsulate CLevelDB iterators cleanly (random-zebra)
21df7cc [DB] Refactor leveldbwrapper (random-zebra)
2251db3 change hardcoded character constants to a set of descriptive named co… (random-zebra)

Pull request description:

  This backports a series of updates and cleanups to the LevelDB wrapper from:

  - bitcoin#5707
  - bitcoin#6650 [`*`]
  - bitcoin#6777 [`*`]
  - bitcoin#6865
  - bitcoin#6873
  - bitcoin#7927 [`*`]
  - bitcoin#8467
  - bitcoin#6290
  - bitcoin#9281

  PIVX-specific edits were required to keep the sporks and zerocoin databases in line.

  [`*`] NOTE: excluding the obfuscation of databases by xoring data, as we might not want this feature (e.g. as zcash/zcash#2598). Otherwise it can be discussed, and added, with a separate PR.

ACKs for top commit:
  furszy:
    Re ACK 82f9088 .
  Fuzzbawls:
    ACK 82f9088

Tree-SHA512: 1e4a75621d2ec2eb68e01523d15321d1d2176b81aac0525617852899ab38c9b4980daecb9056d054e7961fc758a22143edf914c40d1819144a394f2869a8ad57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.