New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LRU Crawler #64
LRU Crawler #64
Commits on Apr 17, 2014
-
barebones LRU crawler proof of concept
so many things undone... TODO is inline in items.c. this seems to work, and the locking should be correct. it is a background thread so shouldn't cause significant latency. However it does quickly roll through the entire LRU (and as of this PoC it just constantly runs), so there will be cpu impact.
-
-
return next item from crawler_crawl_q
... instead of return flags and such.
-
-
don't waste an item flag for crawler
detect via an impossible item: nbytes/nkey are 0. There're other clever ways we could do this as well (nbytes == maxint, etc).
-
-
nothing internally magically fires it off yet, but now there is an external command: lru_crawler crawl [classid] ... will signal the thread to wake up and immediately reap through a particular class. need some thought/feedback for internal kickoffs (plugins?)
-
optionally take a list of slabs to run against.
lru_crawler crawl 1,2,3,10,20 will kick crawlers off for all of those slabs in parallel.
-
command to change the sleep between runs
lru_crawler sleep 1000 - sleeps for one millisecond between runs.
-
starttime options for lru crawler
-o lru_crawler,lru_crawler_sleep=500 - enable it and set sleep for 500 microseconds.
-
reduce indentation, shorten the function a bit. also now sleeps after every inspected item vs after each run through all active slabs.
-
-
-
use do_item_remove in lru crawler
the crawler was saving a few operations by directly freeing an item, but lets use the normal code path instead. The assert looks to ensure item_free() was actually called as we expected. Though I guess lots of people don't use the debug binary...
-
-
lru_crawler_tocrawl to limit inspected per run
Since we start from the tail, and some slabs could be many millions of items, it doesn't always make sense to walk the entire thing looking for data to use. This could be a lot smarter: say specify a percentage, and it'll only crawl a percentage of used_chunks for that slab for instance.