Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch add filters to database #1725

Merged
merged 4 commits into from
Aug 4, 2020

Conversation

benthecarman
Copy link
Collaborator

@benthecarman benthecarman commented Jul 30, 2020

Closes #1724

Show now batch the the adding new filters to the db.

@benthecarman benthecarman added chain work on the chain project optimization labels Jul 30, 2020
Copy link
Contributor

@Christewart Christewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this solves the problem. createAll is effectively a batching operation, it is just a batch size of 1. In #1724 it indicates there is 1000 queued tasks (batches) at the database level. You are effectively creating more tasks (batches) with this PR.

@benthecarman
Copy link
Collaborator Author

I don't think this solves the problem. createAll is effectively a batching operation, it is just a batch size of 1. In #1724 it indicates there is 1000 queued tasks (batches) at the database level. You are effectively creating more tasks (batches) with this PR.

Oh, I see, misunderstood the problem.

@benthecarman
Copy link
Collaborator Author

Okay, changed it from adding each filter individually while syncing to add them in batches, like our callbacks

@benthecarman benthecarman changed the title Batch add headers and filters Batch add filters to database Jul 30, 2020
@Christewart Christewart added this to the v0.4.0 milestone Jul 31, 2020
@benthecarman
Copy link
Collaborator Author

Converted the variables to Option[Int]s

@Christewart Christewart merged commit f4d4f8e into bitcoin-s:master Aug 4, 2020
@benthecarman benthecarman deleted the batch-filters branch August 4, 2020 17:33
Christewart pushed a commit that referenced this pull request May 1, 2021
* Process filter headers in batch

* Use correct batch

* Cache filter heights

* Convert to option
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chain work on the chain project optimization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Database queue fills and throws an exception when syncing filters checkpoints
3 participants