Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

UnknownUncleParent while syncing #6761

Closed
phahulin opened this issue Oct 14, 2017 · 14 comments
Closed

UnknownUncleParent while syncing #6761

phahulin opened this issue Oct 14, 2017 · 14 comments
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. P5-sometimesoon 🌲 Issue is worth doing soon.
Milestone

Comments

@phahulin
Copy link
Contributor

Hello

I'm running:

  • Parity version: Parity/v1.7.0-beta-5f2cabd-20170727/x86_64-linux-gnu/rustc1.18.0
  • Operating system: Ubuntu 16.04.1 LTS
  • And installed: via binaries

We have a private PoA network with validation via a contract. Yesterday we had a network issue and some nodes ran out of sync for a period of time. Later that day they all synced and network seemed to be working fine. However, today clients can't sync with it, the following error appears in logs:

Stage 3 block verification failed for #880003 (72cd…3db3)
Error: Block(UnknownUncleParent(6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69))

More logs here

Could someone help figure out what might be the origin of this error? Using web3 I can find block 880003, its uncles, their parent blocks... so why are they unknown? what does it mean?

  1. Block 880003:
> web3.eth.getBlock(880003)
{ author: '0xdd0bb0e2a1594240fed0c2f2c17c1e9ab4f87126',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x72cd3a741f6d69f98fc14d32bb45d5ff022f0b35feea4be27b04a2c2c8a63db3',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0xdd0bb0e2a1594240fed0c2f2c17c1e9ab4f87126',
  number: 880003,
  parentHash: '0xe6628345aa8b67a828586946ef2ca69073de7a9cd2bf8e444440693576c84c94',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bda0',
     '0xb84110607e64493c5fcc9c6d9e04d43e8b04ebc8c40187c4f3c79c0d1093b118adbe673c845c8ba09f121af6b0b02b38a9075c17ff39309294f281d5980cdb0a6f5300' ],
  sha3Uncles: '0x746f7881ee006ba9a1cb67cabd04d24462a4c079c6d63e3a41ec8bd76307f36f',
  signature: '10607e64493c5fcc9c6d9e04d43e8b04ebc8c40187c4f3c79c0d1093b118adbe673c845c8ba09f121af6b0b02b38a9075c17ff39309294f281d5980cdb0a6f5300',
  size: 1729,
  stateRoot: '0x6a7cbe1ecd84fe8bf3beea54e74785009cc50bda4d46e01bd2e8543a04ad2875',
  step: '301579680',
  timestamp: 1507898399,
  totalDifficulty:
   { [String: '2.99449503737526610663160044663778321083585757e+44']
     s: 1,
     e: 44,
     c: [ 299, 44950373752661, 6631600446637, 78321083585757 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles:
   [ '0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9',
     '0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd' ] }
  1. First uncle:
> web3.eth.getBlock('0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9')
{ author: '0x0e59f9957722ecae11cb65ba46bcac53f6761fe1',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x0e59f9957722ecae11cb65ba46bcac53f6761fe1',
  number: 880000,
  parentHash: '0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9c',
     '0xb8416094980d922b8337dbb4fd947e081750ed09f849189d5af100aeb9510f80538777e483ed74984a7138456e2551354de711265f04d5f3e342e153d52842a36e3400' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '6094980d922b8337dbb4fd947e081750ed09f849189d5af100aeb9510f80538777e483ed74984a7138456e2551354de711265f04d5f3e342e153d52842a36e3400',
  size: 579,
  stateRoot: '0x65af5c00b8be251a46fc94ff30f31110d5c97a8d735c018f5364283fd96ad739',
  step: '301579676',
  timestamp: 1507898380,
  totalDifficulty:
   { [String: '2.99448482890425847847769654539956025778951396e+44']
     s: 1,
     e: 44,
     c: [ 299, 44848289042584, 78477696545399, 56025778951396 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }
  1. First uncle's parent:
> web3.eth.getBlock('0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69')
{ author: '0x0b55fdba9e288c23cf3eb04c6f237d3bd0f751d6',
  difficulty: { [String: '3.40282366920938463463374607431768211454e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211454 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x0b55fdba9e288c23cf3eb04c6f237d3bd0f751d6',
  number: 879999,
  parentHash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9a',
     '0xb84124622d83b323bfb43503c4fcb8228c44e51b8d31da26dff11fe4f292bcc0a02e005233a3592deb3a7d8b6d40c2b97d2d3228a29dfd87768c27e6633a7078f75401' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '24622d83b323bfb43503c4fcb8228c44e51b8d31da26dff11fe4f292bcc0a02e005233a3592deb3a7d8b6d40c2b97d2d3228a29dfd87768c27e6633a7078f75401',
  size: 579,
  stateRoot: '0x060367fdb4fbb80ea5c794d177983bcea9a7cdae24c47d5717c11b09097a0b22',
  step: '301579674',
  timestamp: 1507898370,
  totalDifficulty:
   { [String: '2.99448142608058926909306191165348594010739943e+44']
     s: 1,
     e: 44,
     c: [ 299, 44814260805892, 69093061911653, 48594010739943 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }
  1. Second uncle:
> web3.eth.getBlock('0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd')
{ author: '0x1f6bde6d907e25d907820b6180589537d077a6f9',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x1f6bde6d907e25d907820b6180589537d077a6f9',
  number: 879999,
  parentHash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9b',
     '0xb841722581c66b98e6cf39f66207a6bffcd95918014fae9dd3234797475f066ee7be5382fe0a73fe28ddbefc04e55c09d0d91fbe4ad5571e66fbd4399efa32a886fa00' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '722581c66b98e6cf39f66207a6bffcd95918014fae9dd3234797475f066ee7be5382fe0a73fe28ddbefc04e55c09d0d91fbe4ad5571e66fbd4399efa32a886fa00',
  size: 579,
  stateRoot: '0x89c9c790d1858fa18e61bda8031d747e1174a255d92ec8ed2a5eaca9e2630dda',
  step: '301579675',
  timestamp: 1507898374,
  totalDifficulty:
   { [String: '2.99448142608058926909306191165348594010739942e+44']
     s: 1,
     e: 44,
     c: [ 299, 44814260805892, 69093061911653, 48594010739942 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }
  1. Second uncle's parent:
> web3.eth.getBlock('0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0')
{ author: '0x58bad7cf29428c3d17cb859f5c580d3c25f134cd',
  difficulty: { [String: '3.40282366920938463463374607431768211454e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211454 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x58bad7cf29428c3d17cb859f5c580d3c25f134cd',
  number: 879998,
  parentHash: '0x651fc3b8ab4653c01c44165ebd71f36fdb2860352f16c855cbd864bf0a0a4180',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd99',
     '0xb841dece66ba152187b0446830e8a900e31fe12c502ec6d560cb3d5c7abacc3bfdb94a326aee69a0c18cd1d82566e86c5b5743ee624ef79bdcf27043620e1f13f73d00' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: 'dece66ba152187b0446830e8a900e31fe12c502ec6d560cb3d5c7abacc3bfdb94a326aee69a0c18cd1d82566e86c5b5743ee624ef79bdcf27043620e1f13f73d00',
  size: 579,
  stateRoot: '0x37ad4474d2575dfbc3ab7b3e90e6a0198537da25fa2a7fd73019ca2f5e48c603',
  step: '301579673',
  timestamp: 1507898365,
  totalDifficulty:
   { [String: '2.99447802325692005970842727790741162242528489e+44']
     s: 1,
     e: 44,
     c: [ 299, 44780232569200, 59708427277907, 41162242528489 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }
@5chdn 5chdn added F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. P5-sometimesoon 🌲 Issue is worth doing soon. labels Oct 16, 2017
@5chdn 5chdn added this to the 1.9 milestone Oct 16, 2017
@igorbarinov
Copy link
Member

The consensus is broken. Picture of our disaster 😄 http://testnet.oracles.org:3000/
@5chdn Afri. Is there anything we can do besides recreating network from scratch?

@igorbarinov
Copy link
Member

Nodes resynced by a magic way. The consensus is not broken. Thanks to @rstormsf to resolve issues on his node (no disk space).

@5chdn 5chdn closed this as completed Oct 20, 2017
@5chdn
Copy link
Contributor

5chdn commented Oct 20, 2017

Cool!

@phahulin
Copy link
Contributor Author

phahulin commented Nov 3, 2017

Please reopen this issue as the same error reproduces on v1.8.1.
Current block number is 168806, however one can't resync from scratch past 165001 block:

2017-11-03 13:25:17  Verifier #2 WARN client  Stage 3 block verification failed for #165002 (431b…6507)
Error: Block(UnknownUncleParent(10ffb5e23b62916c39880b31c72fa950120a97bd010057da65b82fd4149dd880))

I've attached a log with sync=trace,engine=trace sync_log.txt.gz

Could this be a possible scenario: one of validators starts a second node and sends transactions to it. If current block is his to seal but transaction didn't have time to propagate to the rest of the network, will this cause the network to split? Anyway please review the logs, this is just my guess.
Thanks.

@phahulin
Copy link
Contributor Author

phahulin commented Nov 3, 2017

UPD: Now the error does not reproduce on 165001 block, but rather on 175001 block.

One more UPD on the following morning: Now it's 180001 block:

2017-11-04 11:35:36  Stage 3 block verification failed for #180001 (062f…fd70)
Error: Block(UnknownUncleParent(2faa7731683aefc251998f6d531e78278168fd3fbcb181c665b09f6f33093adb))

@igorbarinov
Copy link
Member

@5chdn Afri, could we reopen the issue?
We have the problem on another 1.8 based network

@5chdn 5chdn reopened this Nov 4, 2017
@5chdn
Copy link
Contributor

5chdn commented Nov 4, 2017

... 165001 block ... 175001 block

... 180001 block

I see a pattern here. Are you changing your chain config?

@phahulin
Copy link
Contributor Author

phahulin commented Nov 4, 2017

No, each time I just completely delete parity data folder on my local machine and start parity --config node.toml to resync from scratch. I use the same spec.json, node.toml and have the same reserved_peers file.

If you'd like to try yourself, here's my node.toml, my peers.txt, our network's spec.json

This time it's 185002 block.

@igorbarinov
Copy link
Member

The same problem happened on Kovan? #6947

@igorbarinov
Copy link
Member

have you got this problem on Kovan guys?
@david-sg @General-Beck @retotrinkler @uzyn

@igorbarinov
Copy link
Member

We are not using Uncles in AuRa, right?
Can we just comment out it as a temp solution?
https://github.com/paritytech/parity/blob/bc167a211bb5d01d6a7baa60ee7551733ddd1cc5/ethcore/src/verification/verification.rs#L129

@5chdn
Copy link
Contributor

5chdn commented Nov 10, 2017

#7006 should address this

@igorbarinov
Copy link
Member

@5chdn Afri, any estimates when it'll be merged?

@5chdn
Copy link
Contributor

5chdn commented Dec 15, 2017

Should be fixed in 1.7.10 and 1.8.4

@5chdn 5chdn closed this as completed Dec 15, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
F2-bug 🐞 The client fails to follow expected behavior. M4-core ⛓ Core client code / Rust. P5-sometimesoon 🌲 Issue is worth doing soon.
Projects
None yet
Development

No branches or pull requests

3 participants