Skip to content
This repository


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Apr 28, 2014

  1. dormando

    disable tail_repair_time by default

    We've cleaned up a lot of the refcount leaks lately, and this feature does
    have a race for soemthing grabbing active data from the tail.
    dormando authored

Apr 17, 2014

  1. dormando

    fix LRU contention for first minute of uptime

    Noticed in benchmarks that memcached was slow for the first minute. ~350k
    gets/sec max, then it would jump up to millions/sec.
        if (it->time < current_time - ITEM_UPDATE_INTERVAL) {
    current_time starts at 2, which then underflows toward 4 billion, which is
    larger than it->time, so all fetches for the first minute require shuffling
    the LRU. Which happens under a global lock.
    This probably goofed up all short-term benchmarks, and potentially slowed down
    high traffic instances which were recently restarted.
    Unfortunately this means the tests no longer exercise item_update(), so some
    more work is necessary.
    dormando authored
  2. dormando

    lru_crawler_tocrawl to limit inspected per run

    Since we start from the tail, and some slabs could be many millions of items,
    it doesn't always make sense to walk the entire thing looking for data to use.
    This could be a lot smarter: say specify a percentage, and it'll only crawl a
    percentage of used_chunks for that slab for instance.
    dormando authored
  3. dormando

    command to change the sleep between runs

    lru_crawler sleep 1000 - sleeps for one millisecond between runs.
    dormando authored
  4. dormando

    control system

    nothing internally magically fires it off yet, but now there is an external
    lru_crawler crawl [classid]
    ... will signal the thread to wake up and immediately reap through a
    particular class.
    need some thought/feedback for internal kickoffs (plugins?)
    dormando authored
  5. dormando

    don't waste an item flag for crawler

    detect via an impossible item: nbytes/nkey are 0. There're other clever ways
    we could do this as well (nbytes == maxint, etc).
    dormando authored
  6. dormando

    basic tests + lrucrawler -> lru_crawler

    dormando authored
  7. dormando

    barebones LRU crawler proof of concept

    so many things undone... TODO is inline in items.c.
    this seems to work, and the locking should be correct. it is a background
    thread so shouldn't cause significant latency. However it does quickly roll
    through the entire LRU (and as of this PoC it just constantly runs), so there
    will be cpu impact.
    dormando authored
  8. dormando

    Make hash table algorithm selectable

    jenkins hash is old. Lets try murmur3 to start! Default is the old one, so
    people aren't surprised.
    dormando authored

Dec 09, 2013

  1. dormando

    Some tail_repair_time adjustments

    fix t/binary.t test.
    Default of 3 hours is super overkill. 1 hour is still way too much but within
    realm of reason.
    Bail if someone sets it really low (10s or negative)
    dormando authored

Sep 03, 2012

  1. dormando

    remove global stats lock from item allocation

    This doesn't reduce mutex contention much, if at all, for the global stats
    lock, but it does remove a handful of instructions from the alloc hot path,
    which is always worth doing.
    Previous commits possibly added a handful of instructions for the loop and for
    the bucket readlock trylock, but this is still faster than .14 for writes
    dormando authored
  2. dormando

    item locks now lock hash table buckets

    expansion requires switching to a global lock temporarily, so all buckets have
    a covered read lock.
    slab rebalancer is paused during hash table expansion.
    internal item "trylocks" are always issued, and tracked as the hash power
    variable can change out from under it.
    dormando authored
  3. dormando

    alloc loop now attempts an item_lock

    Fixes a few issues with a restructuring... I think -M was broken before,
    should be fixed now. It had a refcount leak.
    Now walks up to five items from the bottom in case of the bottomost items
    being item_locked, or refcount locked. Helps avoid excessive OOM errors for
    some oddball cases. Those happen more often if you're hammering on a handful
    of pages in a very large class size (100k+)
    The hash item lock ensures that if we're holding that lock, no other thread
    can be incrementing the refcount lock at that time. It will mean more in
    future patches.
    slab rebalancer gets a similar update.
    dormando authored

Jul 29, 2012

  1. dormando

    If we're preallocating memory, prealloc slab pages

    I'll probably get in trouble for removing DONT_PREALLOC_SLABS
    ... however tons of people like using the -L option, which does nothing under
    linux. It should soon do *something* under linux, and when it does they'll
    report the same errors of not being able to store things into certain slab
    So just give them a useful error and bail instead.
    dormando authored

Jul 27, 2012

  1. dormando

    automove levels are an int instead of bool now

    also fix a bug causing slab rebalance thread to spin instead of waiting on the
    condition... duhr.
    dormando authored

Jan 10, 2012

  1. dormando

    more portable refcount atomics

    Most credit to Dustin and Trond for showing me the way, though I have no way
    of testing this myself.
    These should probably just be defines...
    dormando authored

Dec 20, 2011

  1. dormando

    slab reassignment

    Adds a "slabs reassign src dst" manual command, and a thread to safely process
    slab moves in the background.
    - slab freelist is now a linked list, reusing the item structure
    - is -o slab_reassign is enabled, an extra background thread is started
    - thread attempts to safely free up items when it's been told to move a page
      from one slab to another.
    -o slab_automove is stubbed.
    There are some limitations. Most notable is that you cannot repeatedly move
    pages around without first having items use up the memory. Slabs with newly
    assigned memory work off of a pointer, handing out chunks individually. We
    would need to change that to quickly split chunks for all newly assigned pages
    into that slabs freelist.
    Further testing is required to ensure such is possible without impacting
    dormando authored

Nov 10, 2011

  1. dormando

    use item partitioned lock for as much as possible

    push cache_lock deeper into the abyss
    dormando authored
  2. dormando

    move hash calls outside of cache_lock

    been hard to measure while using the intel hash (since it's very fast), but
    should help with the software hash.
    dormando authored
  3. dormando

    Use spinlocks for main cache lock

    Partly by Ripduman Sohan
    Appears to significantly help prevent performance dropoff from additional
    threads, but only when the locks are frequently contested and are short.
    dormando authored

Sep 29, 2011

  1. dormando

    Add new stats expired_unfetched, evicted_unfetched

    The two stats represent items which expired and memory was reused, and valid
    items which were evicted, but never touched by get/incr/append/etc
    operations in the meantime.
    Useful for seeing how many wasted items are being set and then rolling out
    through the bottom of the LRU's.
    dormando authored

Sep 28, 2011

  1. dormando

    Allow setting initial size of the hash table

    Instances which run many millions of items can now have its hash table
    presized. This can avoid some minor memory churn during the warmup
    dormando authored
  2. dormando

    expose stats for the internal hash table

    Now users can tell how much memory is being used for the hash table structure.
    It also exposes the current hash power level, which is useful for presizing
    the structure.
    dormando authored
  3. dormando

    experimental maxconns_fast option

    Also fixes -c option to allow reducing the maximum connection limit.
    This gives a new option "-o maxconns_fast", which changes how memcached
    handles hitting the maximum connection limit. By default, it disables the
    accept listener and new connections will wait in the listen queue. With
    maxconns_fast enabled, new connections over the limited have an error written
    to them and are immediately closed by the listener thread.
    This is currently experimental, as we aren't sure how clients will handle the
    change. It may become the default in the future.
    dormando authored

Sep 27, 2011

  1. dormando

    Backport binary TOUCH/GAT/GATQ commands

    Taken from the 1.6 branch, partly written by Trond. I hope the CAS handling is
    dormando authored

Jul 12, 2011

  1. dormando

    fix incr/decr race conditions for binary prot

    there were two race conditions in the incr/decr binary protocol handler. One
    was the original "fetches item outside of add_delta", and the second was in
    the initializer.
    I went for the quick fix by changing the semantics of the store request to be
    an ADD instead of a SET, so if someone beat them in that very narrow race the
    request simply bounces. Not perfect but this is an improvement and good enough
    for now.
    dormando authored

Jul 11, 2011

  1. dormando

    fix incr/decr race conditions for ASCII prot

    binprot requires more work, since it touches CAS and also has a race for
    initializing a missed incr.
    dormando authored

Oct 07, 2009

  1. dormando

    Make item size max start time configurable.

    Adds new option -I, accepts bytes, or NNk or NNm.
    dormando authored dustin committed

Apr 12, 2009

  1. dormando

    fix resumption of accept loop under multithread

    critical fix:
    under multithreaded mode, in version 1.2.7, memcached would not resume
    accepting connections after hitting the max connections limit.
    this is due to a pre-existing bug in the accept_new_conns code, which
    was hit when the "fix" was rolled in to ensure listening threads never
    did real work.  Previously, at least one closing connection would
    (randomly) be on the accepting thread, so the test for (are we the
    listening thread?) would still work.
    dormando authored dustin committed

Mar 30, 2009

  1. dormando

    two new troubleshooting stats

    accepting_conns for completeness, and listen_disabled_num to see how many
    times you've hit maxconns and disabled incoming connections. probably a good
    stat to monitor and flip out on.
    dormando authored dustin committed
  2. dormando

    add a cmd_flush stat

    shouldn't add much lock contention for just this.
    I want to add this one stat (mayb a few more?) since it's happened more than
    once that folks think memcached is broken when a cron or something is calling
    'flush_all' once a minute.
    dormando authored dustin committed

Jan 03, 2009

  1. dormando

    Remove managed instance mode (Dormando) - We should come back and do …

    …this right, probably built on top of the binary protocol and SE stuff.
    dormando authored dustin committed
  2. dormando

    Prevent starvation by busy threads. This gives a configurable limit t…

    …o how many commands to process before the next connection is checked.
    dormando authored dustin committed

Apr 27, 2008

  1. dormando

    BIN_RES_MAGIC should be 0x81 as described in the protocol document.

    Also useful for wireshark/etc for analyzing the protocol.
    git-svn-id: b0b603af-a30f-0410-a34e-baf09ae79d0b
    dormando authored

Mar 03, 2008

  1. dormando

    Don't re-calculate the slab class id for slabs_free() either.

    This + previous patch slightly reduce user CPU time, especially during heavy evictions.
    git-svn-id: b0b603af-a30f-0410-a34e-baf09ae79d0b
    dormando authored
Something went wrong with that request. Please try again.