Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

neutrino: never write filters to disk unless signaled, instead utilize LRU cache #65

Closed
3 tasks
Roasbeef opened this issue Jul 8, 2018 · 2 comments
Closed
3 tasks

Comments

@Roasbeef
Copy link
Member

Roasbeef commented Jul 8, 2018

Currently we always write every filter we need to fetch to disk. This results in high-ish disk and I/O utilization when booting back up after a long dormant period, or aggressive usage of the GetUTXO call. For "normal" wallets, they'll almost never need to re-match a filter once it's been fetched unless they're doing something like a key import. As a result, we should only actually keep a limited (and configurable) set of filters in memory.

Steps To Completion

  • Modify the GetCfilter method to only write to disk if specified by the caller. If we don't write the filter to disk, then we should write it to the cache.

  • Create a new generic (as in []byte) LRU cache. This can double for usage to cache blocks, as well as filters. It may be the case that a more advanced eviction strategy will be more optimal, but we'll at least start with this just to pick the low hanging fruit.

  • Modify the GetCfilter method to first check the cache, then disk if a filter is requested.

@rawtxapp
Copy link
Contributor

rawtxapp commented Jul 8, 2018

This looks like a relatively good first issue, I'd like to take a stab at it if it isn't assigned to anyone, what do you think @Roasbeef ? If yes,

I looked through the code and instead of putting the caching inside GetCfilter, what do you think of putting it directly inside the FilterStore ? It seems like a logical place and that way GetCfilter logic would stay the same (other than just passing an extra persistToDisk bool to PutFilter).

For the LRU, what do you think of using an existing lib like golang-lru ? It also has Adaptive Replacement Cache and TwoQueueCache implementations that we could experiment with.

@Roasbeef
Copy link
Member Author

Roasbeef commented Jul 9, 2018

Ideally we don't bring in additional lib just to implement an LRU cache. Also this should be done outside of the FilterStore all together. This is a caching layer that sits on top of the filter store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants