Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

checkpoint: list api increase max limit #1154

Conversation

taratorio
Copy link
Contributor

@taratorio taratorio commented Mar 29, 2024

Description

The max limit of checkpoint/list api is set to 20 which is very small. E.g. https://heimdall-api.polygon.technology/checkpoints/list?page=1&limit=10000 returns only 20 instead of 10,000.

Checkpoint objects are very small ~100 bytes each. We can fit ~ 10,000 in a 1 MB response size. Feels like this should not be something that can affect the process badly.

Context: Erigon would like to efficiently fetch checkpoints from heimdall for an improved initial sync mechanism. We would like to batch fetch efficiently hence the change.

Changes

  • Bugfix (non-breaking change that solves an issue)
  • Hotfix (change that solves an urgent issue, and requires immediate attention)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (change that is not backwards-compatible and/or changes current functionality)
  • Changes only for a subset of nodes

Breaking changes

Please complete this section if any breaking changes have been made, otherwise delete it

Nodes audience

In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)

Checklist

  • I have added at least 2 reviewer or the whole pos-v1 team
  • I have added sufficient documentation in code
  • I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply

Cross repository changes

  • This PR requires changes to bor
    • In case link the PR here:
  • This PR requires changes to matic-cli
    • In case link the PR here:

Testing

  • I have added unit tests
  • I have added tests to CI
  • I have tested this code manually on local environment
  • I have tested this code manually on remote devnet using express-cli
  • I have tested this code manually on mumbai or amoy
  • I have created new e2e tests into express-cli

Manual tests

➜  ~ curl 'http://0.0.0.0:1317/checkpoints/list?page=6&limit=10' | jq | grep "proposer" | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2298    0  2298    0     0   178k      0 --:--:-- --:--:-- --:--:--  187k
10
➜  ~ curl 'http://0.0.0.0:1317/checkpoints/list?page=6&limit=10000' | jq | grep "proposer" | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2206k    0 2206k    0     0   537k      0 --:--:--  0:00:04 --:--:--  537k
10000
➜  ~ curl 'http://0.0.0.0:1317/checkpoints/list?page=6&limit=1000000' | jq | grep "proposer" | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2206k    0 2206k    0     0   503k      0 --:--:--  0:00:04 --:--:--  528k
10000
➜  ~ 

Copy link
Member

@Raneet10 Raneet10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

codecov bot commented Apr 1, 2024

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 76.59%. Comparing base (d6c9c72) to head (911626f).

Files Patch % Lines
checkpoint/keeper.go 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1154   +/-   ##
========================================
  Coverage    76.59%   76.59%           
========================================
  Files           53       53           
  Lines         5922     5922           
========================================
  Hits          4536     4536           
  Misses        1128     1128           
  Partials       258      258           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marcello33
Copy link
Contributor

Hi @taratorio, thanks for your contribution.
Can you raise this PR against develop?
Thank you

@ssandeep ssandeep changed the base branch from master to develop April 4, 2024 11:24
@ssandeep ssandeep changed the base branch from develop to master April 4, 2024 11:37
@taratorio taratorio changed the base branch from master to develop April 4, 2024 12:37
@taratorio taratorio force-pushed the checkpoint-list-api-higher-max-limit branch from 9165173 to 911626f Compare April 4, 2024 12:52
@taratorio
Copy link
Contributor Author

Hi @taratorio, thanks for your contribution. Can you raise this PR against develop? Thank you

hey @marcello33, ive just changed the base branch and have rebased my code - thanks for letting me know

@manav2401 manav2401 merged commit 4730997 into maticnetwork:develop Apr 8, 2024
10 of 11 checks passed
taratorio added a commit to ledgerwatch/erigon that referenced this pull request Apr 11, 2024
Brings down time to fetch all checkpoints from 4 hours to 42 seconds in
combination with maticnetwork/heimdall#1154

```
DBUG[04-09|21:12:39.754] [bor.heimdall] fetching all checkpoints 
DBUG[04-09|21:13:10.355] [bor.heimdall] fetching all checkpoints progress page=5 len=40000
DBUG[04-09|21:13:22.061] [bor.heimdall] fetching all checkpoints done len=60474 duration=42.307967083s
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants