Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LRU Crawler #64

Merged
merged 19 commits into from Apr 17, 2014
Merged

LRU Crawler #64

merged 19 commits into from Apr 17, 2014

Commits on Apr 17, 2014

  1. barebones LRU crawler proof of concept

    so many things undone... TODO is inline in items.c.
    
    this seems to work, and the locking should be correct. it is a background
    thread so shouldn't cause significant latency. However it does quickly roll
    through the entire LRU (and as of this PoC it just constantly runs), so there
    will be cpu impact.
    dormando committed Apr 17, 2014
    Copy the full SHA
    0d1f505 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    d425b35 View commit details
    Browse the repository at this point in the history
  3. return next item from crawler_crawl_q

    ... instead of return flags and such.
    dormando committed Apr 17, 2014
    Copy the full SHA
    2418511 View commit details
    Browse the repository at this point in the history
  4. adjust some comments.

    dormando committed Apr 17, 2014
    Copy the full SHA
    45177c0 View commit details
    Browse the repository at this point in the history
  5. don't waste an item flag for crawler

    detect via an impossible item: nbytes/nkey are 0. There're other clever ways
    we could do this as well (nbytes == maxint, etc).
    dormando committed Apr 17, 2014
    Copy the full SHA
    5e05fca View commit details
    Browse the repository at this point in the history
  6. crawler reclaims stat

    dormando committed Apr 17, 2014
    Copy the full SHA
    649f7f0 View commit details
    Browse the repository at this point in the history
  7. control system

    nothing internally magically fires it off yet, but now there is an external
    command:
    
    lru_crawler crawl [classid]
    ... will signal the thread to wake up and immediately reap through a
    particular class.
    
    need some thought/feedback for internal kickoffs (plugins?)
    dormando committed Apr 17, 2014
    Copy the full SHA
    6be2b6c View commit details
    Browse the repository at this point in the history
  8. notes

    dormando committed Apr 17, 2014
    Copy the full SHA
    2846242 View commit details
    Browse the repository at this point in the history
  9. optionally take a list of slabs to run against.

    lru_crawler crawl 1,2,3,10,20 will kick crawlers off for all of those slabs in
    parallel.
    dormando committed Apr 17, 2014
    Copy the full SHA
    e8711e1 View commit details
    Browse the repository at this point in the history
  10. command to change the sleep between runs

    lru_crawler sleep 1000 - sleeps for one millisecond between runs.
    dormando committed Apr 17, 2014
    Copy the full SHA
    31d533f View commit details
    Browse the repository at this point in the history
  11. starttime options for lru crawler

    -o lru_crawler,lru_crawler_sleep=500 - enable it and set sleep for 500
    microseconds.
    dormando committed Apr 17, 2014
    Copy the full SHA
    bb63a3e View commit details
    Browse the repository at this point in the history
  12. refactor crawler thread a bit

    reduce indentation, shorten the function a bit. also now sleeps after every
    inspected item vs after each run through all active slabs.
    dormando committed Apr 17, 2014
    Copy the full SHA
    c58291e View commit details
    Browse the repository at this point in the history
  13. LRU Crawler command documentation.

    lacks autorunning stuff still.
    dormando committed Apr 17, 2014
    Copy the full SHA
    561823d View commit details
    Browse the repository at this point in the history
  14. Copy the full SHA
    cb9cafa View commit details
    Browse the repository at this point in the history
  15. -h docs for lru_crawler

    dormando committed Apr 17, 2014
    Copy the full SHA
    67ac109 View commit details
    Browse the repository at this point in the history
  16. use do_item_remove in lru crawler

    the crawler was saving a few operations by directly freeing an item, but lets
    use the normal code path instead. The assert looks to ensure item_free() was
    actually called as we expected. Though I guess lots of people don't use the
    debug binary...
    dormando committed Apr 17, 2014
    Copy the full SHA
    93d5637 View commit details
    Browse the repository at this point in the history
  17. lru_crawler crawl all

    ... go crawl all available classes.
    dormando committed Apr 17, 2014
    Copy the full SHA
    f89e861 View commit details
    Browse the repository at this point in the history
  18. Copy the full SHA
    b3bbd3b View commit details
    Browse the repository at this point in the history
  19. lru_crawler_tocrawl to limit inspected per run

    Since we start from the tail, and some slabs could be many millions of items,
    it doesn't always make sense to walk the entire thing looking for data to use.
    
    This could be a lot smarter: say specify a percentage, and it'll only crawl a
    percentage of used_chunks for that slab for instance.
    dormando committed Apr 17, 2014
    Copy the full SHA
    e31a591 View commit details
    Browse the repository at this point in the history