Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support larger memory #277

Open
sharpboy2008 opened this issue May 8, 2020 · 2 comments
Open

support larger memory #277

sharpboy2008 opened this issue May 8, 2020 · 2 comments

Comments

@sharpboy2008
Copy link

it seems it consumes very little memory. nowadays ,memory is so cheap. most have 32GB.can it make memory configurable or automated increase mem use whenever system has more mem?
by doing this ,it may increase search/filter performance significantly.

@variar
Copy link
Contributor

variar commented May 9, 2020

Glogg is cpu bound on initial indexing due to end of line searching code (see discussion in #227). After initial reading if file is not too large it should stay in OS file cache. That way next reading operations for search/filtering are from memory, not from disk. Keeping file in memory won't increase performance very much.

In klogg I use additional memory to keep cache of search results so repeated searching using same pattern don't actually do the search but uses last results. Using multiple threads for search operations also increases memory use but not very much.

However, more simple index data structure can be used for smaller files (may be configurable). That can increase search performance by using more memory. I'll try this idea.

@variar
Copy link
Contributor

variar commented May 16, 2020

I've done some benchmarking in klogg using plain vector for index as the most simple structure .
For small files (about 1Gb) in single threaded mode there is about 5% search performance improvement. However, in multi-threaded mode there is almost no difference. Right now I don't see where else trading memory for performance could help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants