Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush to disk faster beyond some number of finalized blocks #190

Open
Scooletz opened this issue Nov 24, 2023 · 2 comments
Open

Flush to disk faster beyond some number of finalized blocks #190

Scooletz opened this issue Nov 24, 2023 · 2 comments

Comments

@Scooletz
Copy link
Contributor

When processing a lot of blocks that are already finalized and beyond a given threshold M some blocks could be squashed together or omitted in other way so that they do not incur that much IO.

Potential solutions:

  1. change the flusher in Blockchain so that if it notices that there's more than N blocks (N - history depth) it squashes N blocks applying them in the same transaction. This would reduce the number of writes greatly without changing the way blocks are flushed.
  2. make the flusher use approach like Paprika.Importer where the memory mapped file is not written at all only to be flushed at the end
  3. others...
@Scooletz
Copy link
Contributor Author

I think that with 1st, we can go safe and fast. Especially if the marking as finalized happens in large chunks. Squash every kN + 1 (+1 to move to another slot) and write only then

@Scooletz
Copy link
Contributor Author

Scooletz commented Apr 4, 2024

This should not be the issue right now because of two things from general pov of operations.

  1. Batching file writes #296 that reduces a lot of syscalls, meaning, the writes should be much faster
  2. var level = reader.TryPeek(out _) ? CommitOptions.DangerNoFlush : CommitOptions.FlushDataOnly;
    that skips the FSync as it's most time consuming where the queue has a lot of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant