Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The chain contains non-consensus blocks #1644

Closed
goodsoft opened this issue Mar 23, 2019 · 3 comments
Closed

The chain contains non-consensus blocks #1644

goodsoft opened this issue Mar 23, 2019 · 3 comments
Assignees
@goodsoft
Labels
bug 🐛 Something isn't working

Comments

@goodsoft
Copy link
Contributor

goodsoft commented Mar 23, 2019 •
edited

This causes foreign key violations when importing internal transactions for them.

Compare, for example:
https://etherscan.io/txs?block=6479035 — 160 txs
https://blockscout.com/eth/mainnet/blocks/6479035/transactions — 9 txs

This particular block was fetched on October 8th, 2018.
As it was imported 10 seconds after mining, the bug was most probably present in the realtime fetcher.

UPD: the block in our database is a non-consensus one, which can be confirmed by its hash.
Blocks on the both sides from it are consensus ones.
@goodsoft goodsoft added the bug 🐛 Something isn't working label Mar 23, 2019
@goodsoft
Copy link
Contributor Author

Possible remedy: wait until all internal transactions get successfully imported, and re-fetch the remaining blocks.

As each and every transaction has at least one internal transaction, missing transactions will prevent their blocks from being marked as indexed anyway, so we can safely assume that unmarked blocks contain missing transactions and need to be re-fetched.

@goodsoft goodsoft changed the title Some blocks have their transactions imported incompletely. The chain contains non-consensus blocks Mar 23, 2019
@goodsoft
Copy link
Contributor Author

The number of affected blocks:

eth2s=> select count(*) from blocks b1 left join blocks b2 on b2.parent_hash = b1.hash and b2.consensus where b2.hash is null and b1.consensus;
 count 
-------
   841

The bug manifested itself occasionally from 2018-10-07 20:01:21.385577 till 2019-02-07 16:59:03.802962 (roughly half of the number above), and, after an almost month-long break it started again at 2019-03-01 09:59:37.205347 and a bunch of such blocks appear every day.

goodsoft added a commit that referenced this issue Mar 25, 2019
@goodsoft
Refetch blocks that don't belong to the main chain
a7cea3a 
Due to a bug in the realtime fetcher some of the blocks weren't marked as
non-consensus and now they create a discontinuity in the main chain (#1644).

This PR adds a temporary fetcher, which forces a refetch of blocks that
don't belong to the main chain.
goodsoft added a commit that referenced this issue Mar 25, 2019
@goodsoft
Refetch blocks that don't belong to the main chain
c1e3417 
Due to a bug in the realtime fetcher some of the blocks weren't marked as
non-consensus and now they create a discontinuity in the main chain (#1644).

This PR adds a temporary fetcher, which forces a refetch of blocks that
don't belong to the main chain.
goodsoft added a commit that referenced this issue Mar 25, 2019
@goodsoft
Refetch blocks that don't belong to the main chain
a5cc8e6 
Due to a bug in the realtime fetcher some of the blocks weren't marked as
non-consensus and now they create a discontinuity in the main chain (#1644).

This PR adds a temporary fetcher, which forces a refetch of blocks that
don't belong to the main chain.
@goodsoft goodsoft mentioned this issue Mar 25, 2019
Closed
@goodsoft
Copy link
Contributor Author

goodsoft commented Mar 25, 2019 •
edited

It seems that I found a reason for this bug.

Consider this chain of consensus blocks successfully imported and stored in the database:

A(0) --- B(A) --- C(B)

(where Y(X) means a block with hash = Y and parent_hash = X)

Now two new blocks arrive from a node: D(A) and E(D). The latter gets successfully imported causing consensus loss for block C, while the former one failed with timeout.
Now our chain looks like this:

A(0) --- B(A) --- E(D)

As all these blocks are marked as consensus and there're no holes in numbering, we don't try to refetch block D.

Proposed solution: when importing block force consensus loss not only for existing block on the same number with different hash, but also for a block with number - 1 with hash not matching the parent_hash of the imported block.
This will effectively create a hole in numbering that catchup fetcher will try to fill.

If the forked chain is larger than two blocks, it's still fine, as the consensus loss will propagate when the missing block will be refetched.

goodsoft added a commit that referenced this issue Mar 26, 2019
@goodsoft
Add regression test for #1644
b52e2c3
goodsoft added a commit that referenced this issue Mar 26, 2019
@goodsoft
Force consensus loss for parent block if its hash mismatches parent_hash
56c69bc 
Fixes #1644

When reorg occurs and the older of new blocks fails to be imported, the old
consensus block remains in the database. Catchup fetcher ignores it, and we
get a discontinuity in the main chain.

This commit adds an additional consensus loss step during block import:
the parent block with hash not matching parent_hash of current block is
marked non-consensus, thus creating a hole in main chain, forcing catchup
fetcher to retry fetching it.
goodsoft added a commit that referenced this issue Mar 26, 2019
@goodsoft
Add a migration to mark all invalid blocks as non-consensus (#1644)
44b2ddd
@goodsoft goodsoft mentioned this issue Mar 26, 2019
Merged
@ghost ghost assigned goodsoft Mar 26, 2019
@ghost ghost added the in progress label Mar 26, 2019
goodsoft added a commit that referenced this issue Mar 26, 2019
@goodsoft
Force consensus loss for parent block if its hash mismatches parent_hash
56b4f74 
Fixes #1644

When reorg occurs and the older of new blocks fails to be imported, the old
consensus block remains in the database. Catchup fetcher ignores it, and we
get a discontinuity in the main chain.

This commit adds an additional consensus loss step during block import:
the parent block with hash not matching parent_hash of current block is
marked non-consensus, thus creating a hole in main chain, forcing catchup
fetcher to retry fetching it.
goodsoft added a commit that referenced this issue Mar 26, 2019
@goodsoft
Add a migration to mark all invalid blocks as non-consensus (#1644)
5b0618c
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Add regression test for #1644
2e732c3
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Force consensus loss for parent block if its hash mismatches parent_hash
256369e 
Fixes #1644

When reorg occurs and the older of new blocks fails to be imported, the old
consensus block remains in the database. Catchup fetcher ignores it, and we
get a discontinuity in the main chain.

This commit adds an additional consensus loss step during block import:
the parent block with hash not matching parent_hash of current block is
marked non-consensus, thus creating a hole in main chain, forcing catchup
fetcher to retry fetching it.
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Add a migration to mark all invalid blocks as non-consensus (#1644)
9d1fdd5
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Add regression test for #1644
5f7047e
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Force consensus loss for parent block if its hash mismatches parent_hash
e1d13a3 
Fixes #1644

When reorg occurs and the older of new blocks fails to be imported, the old
consensus block remains in the database. Catchup fetcher ignores it, and we
get a discontinuity in the main chain.

This commit adds an additional consensus loss step during block import:
the parent block with hash not matching parent_hash of current block is
marked non-consensus, thus creating a hole in main chain, forcing catchup
fetcher to retry fetching it.
goodsoft added a commit that referenced this issue Mar 27, 2019
@goodsoft
Add a migration to mark all invalid blocks as non-consensus (#1644)
fe25bf7
@ghost ghost removed the in progress label Mar 27, 2019
gabitoesmiapodo added a commit that referenced this issue Mar 28, 2019
@gabitoesmiapodo
Merge branch 'master' into feature/#1476-add-styles-for-POSDAO-network
3607f53 
* master:
  Update changelog
  Address review comments
  Add a migration to mark all invalid blocks as non-consensus (#1644)
  Force consensus loss for parent block if its hash mismatches parent_hash
  Add regression test for #1644
  feat: add listcontracts endpoint
  fix view test
  use correct type for evm_version
  define evm versions in one place
  fix dialyzer
  set only hours in env var
  fetch transaction period from env variables
  add changelog entry
  add petersburg evm version
  add CHANGELOG entry
  fix credo
  fix tests
  use cache for estimated transaction count
  add transaction count cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@goodsoft goodsoft

Labels
bug 🐛 Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@goodsoft