Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexer reimplementation #83

Merged
merged 58 commits into from
Aug 12, 2024
Merged

Indexer reimplementation #83

merged 58 commits into from
Aug 12, 2024

Conversation

pk910
Copy link
Member

@pk910 pk910 commented Jul 29, 2024

Indexer re-implementation & code cleanup.

The overall goal of this PR is to re-implement the whole indexer logic.
The existing indexer logic is quite unstable in case of forks, is very complex and got unmaintainable over time.

  • Decouple indexer from client pool logic
    • Copy / Amend consensus client pool package from assertoor
    • Implement lightweight execution client pool (based on assertoor, but without block caching)
    • Update cl_clients page to use consensus pool as datasource
    • Update el_clients page to use execution pool as datasource
  • Re-Implement Indexer
    • Load specs from consensus pool instead of local file (Feature: Load specs from beacon nodes #49)
    • Proper support for chain forks
      • Decouple block & epoch stats cache, add separate epoch stats loader
      • Improve validator set caching (re-use validator struct from previous validator sets if all values match)
      • Compute proposer/attester/sync duties from state (avoid loading duties from RPC as the api does not support loading duties for non canonical forks)
      • Implement fork handling & canonical fork selection
    • Persist unfinalized blocks in database
    • Persist unfinalized epoch stats in database
    • Implement block cache processing routines
      • Block body pruning
      • Finalized processing
      • Orphaned block processing
    • Implement epoch history synchronization

@pk910
Copy link
Member Author

pk910 commented Jul 29, 2024

fixes #49 as specs are coming from consensus pool now

@pk910 pk910 changed the title Indexer reimplementation WIP: Indexer reimplementation Jul 29, 2024
Bumps the dependencies group with 4 updates: [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) and [github.com/lucasb-eyer/go-colorful](https://github.com/lucasb-eyer/go-colorful).


Updates `github.com/aws/aws-sdk-go-v2/config` from 1.27.26 to 1.27.27
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@config/v1.27.26...config/v1.27.27)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.17.26 to 1.17.27
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@credentials/v1.17.26...credentials/v1.17.27)

Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.17.7 to 1.17.8
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/v1.17.8/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@v1.17.7...v1.17.8)

Updates `github.com/lucasb-eyer/go-colorful` from 1.0.2 to 1.2.0
- [Release notes](https://github.com/lucasb-eyer/go-colorful/releases)
- [Changelog](https://github.com/lucasb-eyer/go-colorful/blob/master/CHANGELOG.md)
- [Commits](lucasb-eyer/go-colorful@v1.0.2...v1.2.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: github.com/lucasb-eyer/go-colorful
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
@pk910 pk910 changed the title WIP: Indexer reimplementation Indexer reimplementation Aug 5, 2024
@pk910 pk910 marked this pull request as ready for review August 5, 2024 19:09
@pk910 pk910 added the build-docker-image Automatically build docker image for PR branch label Aug 5, 2024
@pk910 pk910 merged commit 7749653 into electra-support Aug 12, 2024
11 checks passed
@pk910 pk910 deleted the indexer-reimplementation branch August 12, 2024 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build-docker-image Automatically build docker image for PR branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant