Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store blocks from non-canonical forks; supply them to indexer on startup #487

Closed
MicaiahReid opened this issue Feb 1, 2024 · 2 comments · Fixed by #505
Closed

store blocks from non-canonical forks; supply them to indexer on startup #487

MicaiahReid opened this issue Feb 1, 2024 · 2 comments · Fixed by #505
Assignees
@MicaiahReid
Labels
released
Milestone
Production Reliab...

Comments

@MicaiahReid
Copy link
Collaborator

If Chainhook restarts during a reorg, it needs to have some recent data to determine if incoming blocks are part of the canonical chain or not. This requires three changes:

  • when starting up the event observer, the indexer's initial forks should be seeded with data (all unconfirmed blocks) from rocksdb
  • we should store blocks from non-canonical forks in rocksdb, so the indexer has a full picture of the pre-shutdown state
  • when a block is confirmed, it should prune the now invalid forks that were stored in rocksdb (to prevent a bloated db)
@MicaiahReid MicaiahReid mentioned this issue Feb 1, 2024
Closed
@MicaiahReid MicaiahReid self-assigned this Feb 12, 2024
@MicaiahReid MicaiahReid mentioned this issue Feb 20, 2024
Merged
@smcclellan smcclellan added this to the Production Reliability milestone Mar 8, 2024
MicaiahReid added a commit that referenced this issue Mar 14, 2024
@MicaiahReid
fix: seed forking handler with unconfirmed blocks to improve startup …
b77aca2 
…stability (#505)

### Description

If Chainhook restarts in the middle of a reorg taking place, it doesn't
have any context for choosing the canonical fork. Then when block
collisions take place, Chainhook fails to process the new blocks,
causing gaps in the blocks Chainhook has available for evaluation.

This PR seeds the stacks block indexer with unconfirmed blocks on
startup, so that Chainhook has the necessary context to handle a reorg.

Most of the PR is to add two tests:
 - I've added some new functionality to our very thorough indexer tests
- In addition to providing the blocks to be mined and the order to mine
them, we also now allow providing some "unconfirmed" blocks to seed the
block pool with.
- I've added some test cases that reproduce what caused outages on the
Platform's Chainhook node
 - I've added a new service test that:
   - Verifies that unconfirmed blocks are stored on restart
- Verifies that those blocks are used to seed the block pool, and that a
reorg is handled correctly.

I committed these tests _before_ adding the fix, so you can confirm the
fix by checking out the commits with the new tests
(96d8249 and
9aad55e), seeing that the tests break,
then pulling the last commit to see that the fix works.

Fixes #487
MicaiahReid added a commit that referenced this issue Mar 27, 2024
@MicaiahReid
fix: seed forking handler with unconfirmed blocks to improve startup …
4d5e031 
…stability (#505)

### Description

If Chainhook restarts in the middle of a reorg taking place, it doesn't
have any context for choosing the canonical fork. Then when block
collisions take place, Chainhook fails to process the new blocks,
causing gaps in the blocks Chainhook has available for evaluation.

This PR seeds the stacks block indexer with unconfirmed blocks on
startup, so that Chainhook has the necessary context to handle a reorg.

Most of the PR is to add two tests:
 - I've added some new functionality to our very thorough indexer tests
- In addition to providing the blocks to be mined and the order to mine
them, we also now allow providing some "unconfirmed" blocks to seed the
block pool with.
- I've added some test cases that reproduce what caused outages on the
Platform's Chainhook node
 - I've added a new service test that:
   - Verifies that unconfirmed blocks are stored on restart
- Verifies that those blocks are used to seed the block pool, and that a
reorg is handled correctly.

I committed these tests _before_ adding the fix, so you can confirm the
fix by checking out the commits with the new tests
(96d8249 and
9aad55e), seeing that the tests break,
then pulling the last commit to see that the fix works.

Fixes #487
MicaiahReid added a commit that referenced this issue Mar 27, 2024
@MicaiahReid
fix: seed forking handler with unconfirmed blocks to improve startup …
485394e 
…stability (#505)

### Description

If Chainhook restarts in the middle of a reorg taking place, it doesn't
have any context for choosing the canonical fork. Then when block
collisions take place, Chainhook fails to process the new blocks,
causing gaps in the blocks Chainhook has available for evaluation.

This PR seeds the stacks block indexer with unconfirmed blocks on
startup, so that Chainhook has the necessary context to handle a reorg.

Most of the PR is to add two tests:
 - I've added some new functionality to our very thorough indexer tests
- In addition to providing the blocks to be mined and the order to mine
them, we also now allow providing some "unconfirmed" blocks to seed the
block pool with.
- I've added some test cases that reproduce what caused outages on the
Platform's Chainhook node
 - I've added a new service test that:
   - Verifies that unconfirmed blocks are stored on restart
- Verifies that those blocks are used to seed the block pool, and that a
reorg is handled correctly.

I committed these tests _before_ adding the fix, so you can confirm the
fix by checking out the commits with the new tests
(96d8249 and
9aad55e), seeing that the tests break,
then pulling the last commit to see that the fix works.

Fixes #487
github-actions bot pushed a commit that referenced this issue Mar 27, 2024
@semantic-release-bot
chore(release): 1.4.0 [skip ci]
8024e0c 
## [1.4.0](v1.3.1...v1.4.0) (2024-03-27)

### Features

* detect http / rpc errors as early as possible ([ad78669](ad78669))
* use stacks.rocksdb for predicate scan ([#514](#514)) ([a4f1663](a4f1663)), closes [#513](#513) [#485](#485)

### Bug Fixes

* enable debug logs in release mode ([#537](#537)) ([fb49e28](fb49e28))
* improve error handling, and more! ([#524](#524)) ([86b5c78](86b5c78)), closes [#498](#498) [#521](#521) [#404](#404) [/github.com//issues/517#issuecomment-1992135101](https://github.com/hirosystems//github.com/hirosystems/chainhook/issues/517/issues/issuecomment-1992135101) [#517](#517) [#506](#506) [#510](#510) [#519](#519)
* log errors on block download failure; implement max retries ([#503](#503)) ([0fc38cb](0fc38cb))
* **metrics:** update latest ingested block on reorg ([#515](#515)) ([8f728f7](8f728f7))
* order and filter blocks used to seed forking block pool ([#534](#534)) ([a11bc1c](a11bc1c))
* seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([485394e](485394e)), closes [#487](#487)
* skip db consolidation if no new dataset was downloaded ([#513](#513)) ([983a165](983a165))
* update scan status for non-triggering predicates ([#511](#511)) ([9073f42](9073f42)), closes [#498](#498)
@github-actions GitHub Actions
Copy link

🎉 This issue has been resolved in version 1.4.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

github-actions bot pushed a commit that referenced this issue Apr 6, 2024
@semantic-release-bot
chore(release): 1.5.0 [skip ci]
af7fec4 
## [1.5.0](v1.4.1...v1.5.0) (2024-04-06)

### Features

* add brc-20 schemas for ordhook ([#551](#551)) ([1e25a8f](1e25a8f))
* detect http / rpc errors as early as possible ([e515116](e515116))
* rm blocks ([f35498d](f35498d))
* use stacks.rocksdb for predicate scan ([#514](#514)) ([0baae10](0baae10)), closes [#513](#513) [#485](#485)

### Bug Fixes

* build error ([03b3938](03b3938))
* log errors on block download failure; implement max retries ([#503](#503)) ([3bfb0e1](3bfb0e1))
* **metrics:** update latest ingested block on reorg ([#515](#515)) ([823713a](823713a))
* order and filter blocks used to seed forking block pool ([#534](#534)) ([a2865b7](a2865b7))
* revisit 7+ blocks reorg handling ([#553](#553)) ([184fd07](184fd07))
* seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([b77aca2](b77aca2)), closes [#487](#487)
* skip db consolidation if no new dataset was downloaded ([#513](#513)) ([b1469a6](b1469a6))
* update scan status for non-triggering predicates ([#511](#511)) ([32cdfee](32cdfee)), closes [#498](#498)
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2024

🎉 This issue has been resolved in version 1.5.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MicaiahReid MicaiahReid

Labels
released
Projects
DevTools
Archived in project
Milestone
Production Reliability
Development

Successfully merging a pull request may close this issue.

fix: seed forking handler with unconfirmed blocks to improve startup stability hirosystems/chainhook
2 participants
@smcclellan @MicaiahReid