Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster ledger state #287

Merged
merged 10 commits into from Jan 14, 2024
Merged

Faster ledger state #287

merged 10 commits into from Jan 14, 2024

Conversation

berewt
Copy link
Contributor

@berewt berewt commented Jan 8, 2024

Several memory/speed improvement:

  • Store volatile ledger states in memory (faster rollback)
  • Use Builder instead of Bytestring for serialisation
  • Save LedgerState less often, and always save it on close.
  • Add a --batch-size flag

Pre-submit checklist:

  • Branch
    • Tests are provided (if possible)
    • Commit sequence broadly makes sense and have useful messages
    • Important changes are reflected in changelog.d of the affected packages
    • Relevant tickets are mentioned in commit messages
  • PR
    • (For external contributions) Corresponding issue exists and is linked in the description
    • Targeting main unless this is a cherry-pick backport
    • Self-reviewed the diff
    • Useful pull request description
    • If relevant, reference the ADR in the PR and reference the PR in the ADR
    • Reviewer requested

@berewt berewt marked this pull request as ready for review January 9, 2024 08:24
@berewt berewt force-pushed the faster-ledger-state branch 3 times, most recently from 4bdd6d0 to b2d30fb Compare January 11, 2024 19:06
Copy link
Contributor

@brendanrbrown brendanrbrown left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good. My comments are minor and mostly about documentation, particularly in cases where it's unclear that bad values of certain parameters are possible.

The CI currently fails because of some fourmolu thing.

But the tests failed for me locally because of golden files: I went ahead and fixed those up as in #286, in case you were still having issues with whitespace.

* File indexer maintain a list of file in memory
* Ledger state is maintained in memory
* We use builder to serialise event in the file indexer
* Add a batch-size flag
@berewt
Copy link
Contributor Author

berewt commented Jan 12, 2024

This all looks good. My comments are minor and mostly about documentation, particularly in cases where it's unclear that bad values of certain parameters are possible.

The CI currently fails because of some fourmolu thing.

But the tests failed for me locally because of golden files: I went ahead and fixed those up as in #286, in case you were still having issues with whitespace.

I still have these issues (I think I was able to do it in the past though, which is weird).

@brendanrbrown
Copy link
Contributor

I still have these issues

Let me know if you want me to push the whitespace fix again.

@berewt berewt merged commit 66be793 into main Jan 14, 2024
4 checks passed
@berewt berewt deleted the faster-ledger-state branch January 14, 2024 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants