Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

187-5 fork handler integrated into tezos indexer #903

Merged
merged 19 commits into from
Sep 29, 2020
Merged

187-5 fork handler integrated into tezos indexer #903

merged 19 commits into from
Sep 29, 2020

Conversation

ivanopagano
Copy link
Contributor

@ivanopagano ivanopagano commented Sep 1, 2020 •
edited

This closes #42

This PR is mainly focused on integrating the fork handling in the main indexing process.

This required a couple of adaptations w.r.t. the high-level handler component defined so far:

  • adapting the fork handler to have a generic fork-detector which could be plugged-in to provide more custom behaviour, i.e. the indexer requires a consistent detector to operate safely under non-equilibrium of the tezos node it connects to
  • define a more optimised version of the handler that would re-use shared resources to read data from the node and the db, considering that asking data for a specific block level requires at least 2 roundtrips using the tezos rpc api
  • extract the indexed data accessor module from the node operator, which was used there only to get the highest block level reached locally. Now the indexer computes that outside of the node operator for the sake of the fork-handler too. The node operator doesn't need that dependency anymore

Take note that the fork detection for tezos uses an "optimistic" approach that recursively tries to contact the node for information, but might fail if the node is not responding, thus bringing down the whole lorre process.
We might want to later improve on that with an retry policy internal to the fork component.

This PR additionally contains a few minor but needed unrelated changes regarding configuration of logging layout, corrections to the metadata related to renamed fields, refactoring of packages and some classes for a more consistent organisation of the code, updates to the smoke tests regression suite.

No changes to the db-schema or api surface.

NOTE This is the last of a long-running streak of PRs to define the overall fork handling strategy.
Before merging it we need to merge the previous PRs #896 #898 #899 and rebase on the new master branch, after which the draft tag could be removed.

@ivanopagano ivanopagano force-pushed the feature/187-4-tezos-consistent-fork-detector branch from 83e56d1 to 96f73fd Compare September 4, 2020 14:55
@ivanopagano ivanopagano force-pushed the feature/187-5-fork-handler-tezos branch from 1a4c00e to 3faaf58 Compare September 4, 2020 14:55
@ivanopagano ivanopagano force-pushed the feature/187-4-tezos-consistent-fork-detector branch from 681938b to ff8d4ee Compare September 25, 2020 07:28
@ivanopagano ivanopagano force-pushed the feature/187-5-fork-handler-tezos branch 2 times, most recently from e1228fc to b28ea77 Compare September 25, 2020 15:21
piotrkosecki
piotrkosecki approved these changes Sep 29, 2020
View reviewed changes
Copy link
Contributor

@piotrkosecki piotrkosecki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ivanopagano ivanopagano force-pushed the feature/187-4-tezos-consistent-fork-detector branch 2 times, most recently from 72affdd to ae1917b Compare September 29, 2020 16:14
@ivanopagano
Refactor to block level as 64-bit int value, fix all tests
209472c 
 * correct all database types for level as bigint
 * correct all code modeling levels to use the new alias
 * give an explicit class to BlockReference for better matching
 * simplify the node operator tests and add explanatory comments
 * add test case for loading blocks not indexed in db
@ivanopagano
Resolve some leftovers
cd4c228
@ivanopagano
Polish the PR and fix docs and details.
2257b05
@ivanopagano
Remove duplication of code in the database invalidation functions
2f282e7 
 * add some log about random generation results during tests
 * fix an incorrect test
@ivanopagano
Add local utilities to reduce duplication and verbosity of test
d785b41
@ivanopagano
Address PR review requests
0d00463
@ivanopagano
Refactor to block level as 64-bit int value, fix all tests
20e757c 
 * correct all database types for level as bigint
 * correct all code modeling levels to use the new alias
 * give an explicit class to BlockReference for better matching
 * simplify the node operator tests and add explanatory comments
 * add test case for loading blocks not indexed in db
@ivanopagano
Resolve some leftovers
8d5a4b5
@ivanopagano
Refactor to block level as 64-bit int value, fix all tests
786f944 
 * correct all database types for level as bigint
 * correct all code modeling levels to use the new alias
 * give an explicit class to BlockReference for better matching
 * simplify the node operator tests and add explanatory comments
 * add test case for loading blocks not indexed in db
@ivanopagano
Resolve some leftovers
8bcbd9d
@ivanopagano
Add fork and invalidation info to tables
73ebc6b 
 * define a default id for tezos main fork
 * change the data to be flagged as invalidated on every entity that
   is related to a block in some form
 * remove chimney usage for conversions where the compiler got stuck by
   some possible bug
 * bugfix for bakers flag on accounts
 * update existing tests to get all green
 * complete tests for database operations returning only valid data
@ivanopagano
Stabilize the flaky tests for fork detectors variants
a227141
@ivanopagano
Move the ForkHandler source from test to main sources
dfae18c
@ivanopagano
Define a tezos fork handler implementation
6cedd93 
 * create an component to handle fork searches
 * define a custom processor that uses a specific detector to handle
   forks
 * move components into a more consistent package structure
 * rename test objects for the test scenarios about forks
@ivanopagano
Run the fork-handler during the indexing loop
7d7ec52 
 * move the max-indexed-level call outside of the node-operator in the
   main indexer, to re-use the value in the main loop
 * update tests
@ivanopagano
Run smoke tests, bring expectations and configs up to date
4937c51
@ivanopagano
Fix a flaky test on generated data that violates db type definitions
a56be90
@ivanopagano ivanopagano force-pushed the feature/187-5-fork-handler-tezos branch from dd314ba to 0eb6b1e Compare September 29, 2020 18:40
@ivanopagano ivanopagano changed the base branch from feature/187-4-tezos-consistent-fork-detector to master September 29, 2020 18:41
@sonarcloud
Copy link

sonarcloud bot commented Sep 29, 2020

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
5.0% 5.0% Duplication

This was referenced Sep 29, 2020
Open
Open
@ivanopagano ivanopagano marked this pull request as ready for review September 29, 2020 19:12
@vishakh vishakh merged commit 9a846ff into master Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vishakh vishakh vishakh approved these changes

@piotrkosecki piotrkosecki piotrkosecki approved these changes

@bhaskarkishore bhaskarkishore Awaiting requested review from bhaskarkishore

Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Lorre should be resilient to forks
3 participants
@ivanopagano @vishakh @piotrkosecki