-
Notifications
You must be signed in to change notification settings - Fork 176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
checkpoint: list api increase max limit #1154
checkpoint: list api increase max limit #1154
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1154 +/- ##
========================================
Coverage 76.59% 76.59%
========================================
Files 53 53
Lines 5922 5922
========================================
Hits 4536 4536
Misses 1128 1128
Partials 258 258 ☔ View full report in Codecov by Sentry. |
Hi @taratorio, thanks for your contribution. |
9165173
to
911626f
Compare
hey @marcello33, ive just changed the base branch and have rebased my code - thanks for letting me know |
Brings down time to fetch all checkpoints from 4 hours to 42 seconds in combination with maticnetwork/heimdall#1154 ``` DBUG[04-09|21:12:39.754] [bor.heimdall] fetching all checkpoints DBUG[04-09|21:13:10.355] [bor.heimdall] fetching all checkpoints progress page=5 len=40000 DBUG[04-09|21:13:22.061] [bor.heimdall] fetching all checkpoints done len=60474 duration=42.307967083s ```
Description
The max limit of checkpoint/list api is set to 20 which is very small. E.g. https://heimdall-api.polygon.technology/checkpoints/list?page=1&limit=10000 returns only 20 instead of 10,000.
Checkpoint objects are very small ~100 bytes each. We can fit ~ 10,000 in a 1 MB response size. Feels like this should not be something that can affect the process badly.
Context: Erigon would like to efficiently fetch checkpoints from heimdall for an improved initial sync mechanism. We would like to batch fetch efficiently hence the change.
Changes
Breaking changes
Please complete this section if any breaking changes have been made, otherwise delete it
Nodes audience
In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)
Checklist
Cross repository changes
Testing
Manual tests