Batch accounts when decrypting transaction notes #4727
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dguenther
force-pushed
the
decrypt-parallel
branch
from
d081849
to
7de7f71
Compare
danield9tqh
reviewed
Apr 18, 2024
hughy
approved these changes
Apr 18, 2024
danield9tqh
reviewed
Apr 19, 2024
dguenther
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Scanning 40+ accounts on one node right now is almost impossible due to only executing one decryptNotes batch at a time, and splitting up the decryptNotes batches per transaction and per account. In the past, this was more of a concern that the worker pool job queue would get blocked, but now that we process tasks round robin from every task queue, we should be able to keep the queue full.
With nodeWorkersMax and nodeWorkers set to 12, this increases decryption time from about 1 block per second to about 10, depending on the number of notes per block. A secondary benefit is the decryption no longer happens inside the DB lock -- the account status seems to update more frequently on this branch, but it could be due to the scan speed.
Edit: Tests pass now, this is ready for review + manual testing.
Testing Plan
Documentation
Does this change require any updates to the Iron Fish Docs (ex. the RPC API
Reference)? If yes, link a
related documentation pull request for the website.
Breaking Change
Is this a breaking change? If yes, add notes below on why this is breaking and
what additional work is required, if any.