Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix OOM when deserializing UTXO entries with invalid length #7933

Merged
merged 5 commits into from
Apr 26, 2016
Merged

Fix OOM when deserializing UTXO entries with invalid length #7933

merged 5 commits into from
Apr 26, 2016

Conversation

sipa
Copy link
Member

@sipa sipa commented Apr 24, 2016 •
edited
Loading

Thanks to @pstratem for finding this.

The normal vector deserializer reads data in chunks of at most 5 MB, preventing OOM when insane vector lengths are encoded. This protection is not present in CScriptCompressor's specialized deserializer, however, resulting in a potential OOM when very large length descriptors exist, as the target CScript is resized before attempting to read that much data.

However, CScripts have a maximum length above which they're always invalid. We can treat scriptPubKeys with such lengths as unspendable, preventing them from going into the UTXO set even, and skipping them when deserializing.

Note that none of this is exposed to the network, as the P2P code uses normal (pre)vectors, which do have this OOM protection directly in serialize.h.

@sipa sipa force-pushed the fixcompressoroom branch 2 times, most recently from 87f71d9 to 236e132 Compare April 24, 2016 18:17
@laanwj
Copy link
Member

laanwj commented Apr 25, 2016

I think this needs a test (that fails without this, and succeeds with it). Or are we talking about 'insane lengths' of such magnitude that would result in a very long running test case?

@sipa
Copy link
Member Author

sipa commented Apr 25, 2016 via email

@laanwj
Copy link
Member

laanwj commented Apr 25, 2016

The test (without fix) would use 2 GB+ RAM, and segfault.

Maybe we can cap it off before that happens?
In any case a test that OOMs and crashes without this, but runs quickly with it, would be great too, it effectively prevents regression.

@sipa sipa mentioned this pull request Apr 25, 2016
Closed
@sipa
Copy link
Member Author

sipa commented Apr 25, 2016

Added a test, and included #7936 (the test fails without).

@laanwj
Copy link
Member

laanwj commented Apr 25, 2016

utACK 1e44169

@maflcko
Copy link
Member

maflcko commented Apr 25, 2016

Can confirm the test fails:

$ src/test/test_bitcoin -t coins_tests
Running 3 test cases...
unknown location(0): fatal error in "ccoins_serialization": memory access violation at address: 0x...: no mapping at fault address
test/coins_tests.cpp(411): last checkpoint

*** 1 failure detected in test suite "Bitcoin Test Suite"

@dcousens
Copy link
Contributor

utACK 1e44169

@laanwj laanwj merged commit 1e44169 into bitcoin:master Apr 26, 2016
laanwj added a commit that referenced this pull request Apr 26, 2016
@laanwj
Merge #7933: Fix OOM when deserializing UTXO entries with invalid length
e26b620 
1e44169 Add tests for CCoins deserialization (Pieter Wuille)
5d0434d Fix OOM bug: UTXO entries with invalid script length (Pieter Wuille)
4bf631e CDataStream::ignore Throw exception instead of assert on negative nSize. (Patrick Strateman)
4f87af6 Treat overly long scriptPubKeys as unspendable (Pieter Wuille)
f8e6fb1 Introduce constant for maximum CScript length (Pieter Wuille)
@bitcartel bitcartel mentioned this pull request Oct 21, 2016
Merged
zkbot pushed a commit to zcash/zcash that referenced this pull request Oct 22, 2016
Auto merge of #1591 - bitcartel:upstream_7933_fix_out_of_memory_deser…
7ccbcca 
…ializing_utxo, r=daira

Upstream: fix out of memory problem when deserializing utxo

bitcoin/bitcoin#7933
@ptschip ptschip mentioned this pull request Oct 3, 2017
Merged
codablock pushed a commit to codablock/dash that referenced this pull request Oct 19, 2017
@laanwj @codablock
Merge bitcoin#7933: Fix OOM when deserializing UTXO entries with inva…
952383e 
…lid length

1e44169 Add tests for CCoins deserialization (Pieter Wuille)
5d0434d Fix OOM bug: UTXO entries with invalid script length (Pieter Wuille)
4bf631e CDataStream::ignore Throw exception instead of assert on negative nSize. (Patrick Strateman)
4f87af6 Treat overly long scriptPubKeys as unspendable (Pieter Wuille)
f8e6fb1 Introduce constant for maximum CScript length (Pieter Wuille)
@dagurval dagurval mentioned this pull request Mar 5, 2018
Merged
@random-zebra random-zebra mentioned this pull request Oct 8, 2019
Merged
random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request Oct 9, 2019
@random-zebra
Merge #1039: Fix OOM when deserializing UTXO entries with invalid length
41d19c1 
9f0868a [Tests] Add tests for CCoins deserialization (random-zebra)
5006d45 CDataStream::ignore Throw exception instead of assert on negative nSize (random-zebra)
b8bc0d5 [Bug] Fix OOM when deserializing UTXO entries with invalid length (random-zebra)
0657a13 [Script] Treat overly long scriptPubKeys as unspendable (random-zebra)
4bfc161 [Script] Introduce constant for maximum CScript length (random-zebra)

Pull request description:

  ref: bitcoin#7933

ACKs for top commit:
  furszy:
    Good, tests passing 👍 , ACK 9f0868a
  Warrows:
    ACK 9f0868a

Tree-SHA512: 9f978d55cc2564ff905642fe624df43f502a297fbc966480164556328091933a9d6eb861bf287f1d07edb1d0e363d0a63ce76df7f01f01b9e73f69ba87a5576f
akshaynexus added a commit to dogecash/dogecash-old that referenced this pull request Oct 26, 2019
@akshaynexus
Merge #1039: Fix OOM when deserializing UTXO entries with invalid length
fc00be9 
9f0868a [Tests] Add tests for CCoins deserialization (random-zebra)
5006d45 CDataStream::ignore Throw exception instead of assert on negative nSize (random-zebra)
b8bc0d5 [Bug] Fix OOM when deserializing UTXO entries with invalid length (random-zebra)
0657a13 [Script] Treat overly long scriptPubKeys as unspendable (random-zebra)
4bfc161 [Script] Introduce constant for maximum CScript length (random-zebra)

Pull request description:

  ref: bitcoin/bitcoin#7933

ACKs for top commit:
  furszy:
    Good, tests passing 👍 , ACK 9f0868a
  Warrows:
    ACK 9f0868a

Tree-SHA512: 9f978d55cc2564ff905642fe624df43f502a297fbc966480164556328091933a9d6eb861bf287f1d07edb1d0e363d0a63ce76df7f01f01b9e73f69ba87a5576f
akshaynexus added a commit to dogecash/dogecash-old that referenced this pull request Oct 26, 2019
@akshaynexus
Merge #1039: Fix OOM when deserializing UTXO entries with invalid length
9676003 
9f0868a [Tests] Add tests for CCoins deserialization (random-zebra)
5006d45 CDataStream::ignore Throw exception instead of assert on negative nSize (random-zebra)
b8bc0d5 [Bug] Fix OOM when deserializing UTXO entries with invalid length (random-zebra)
0657a13 [Script] Treat overly long scriptPubKeys as unspendable (random-zebra)
4bfc161 [Script] Introduce constant for maximum CScript length (random-zebra)

Pull request description:

  ref: bitcoin/bitcoin#7933

ACKs for top commit:
  furszy:
    Good, tests passing 👍 , ACK 9f0868a
  Warrows:
    ACK 9f0868a

Tree-SHA512: 9f978d55cc2564ff905642fe624df43f502a297fbc966480164556328091933a9d6eb861bf287f1d07edb1d0e363d0a63ce76df7f01f01b9e73f69ba87a5576f
akshaynexus added a commit to dogecash/dogecash-old that referenced this pull request Oct 26, 2019
@akshaynexus
Merge #1039: Fix OOM when deserializing UTXO entries with invalid length
90b43ef 
9f0868a [Tests] Add tests for CCoins deserialization (random-zebra)
5006d45 CDataStream::ignore Throw exception instead of assert on negative nSize (random-zebra)
b8bc0d5 [Bug] Fix OOM when deserializing UTXO entries with invalid length (random-zebra)
0657a13 [Script] Treat overly long scriptPubKeys as unspendable (random-zebra)
4bfc161 [Script] Introduce constant for maximum CScript length (random-zebra)

Pull request description:

  ref: bitcoin/bitcoin#7933

ACKs for top commit:
  furszy:
    Good, tests passing 👍 , ACK 9f0868a
  Warrows:
    ACK 9f0868a

Tree-SHA512: 9f978d55cc2564ff905642fe624df43f502a297fbc966480164556328091933a9d6eb861bf287f1d07edb1d0e363d0a63ce76df7f01f01b9e73f69ba87a5576f
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
Bug UTXO Db and Indexes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sipa @laanwj @maflcko @dcousens @pstratem