Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Commits on May 14, 2014
  1. @jamesgpearce

    Update README

    jamesgpearce authored
Commits on Nov 5, 2008
  1. auto detect the number of cpus

    ps authored
    Summary: Instead of relying on -t to run on greater than 4 cpu systems, auto detect the number of active processors
    
    Reviewed By: ttung
    
    Test Plan: ran memcached on 8 core machine, 9 threads created.  ran on 4 core machine, 5 threads created.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@130064 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 16, 2008
  1. Bump version number of memcached

    ps authored
    Summary: Prepare to release 1.2.3e
    
    Reviewed By: ttung
    
    Test Plan: running in production
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@127004 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 15, 2008
  1. if item_storage_allocated is per-thread, we actually allocate (#threa…

    ttung authored
    …d) * memlimit
    
    Summary: accumulate item_storage_allocated before reading.  this should happen infrequently enough that it won't kill system performance.
    
    Reviewed By: ps
    
    Test Plan: ran it with mcblaster cmdline:
               ./mcblaster -p 15151 -u 15151 -c 32 -k 1000000 -x 4 -w 50000 -r 50000 -z 8192 127.0.0.1
               
               resident memory went to about 100mb, but dropped back to ~66mb after i killed mcblaster.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@126762 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 28, 2008
  1. remove unneeded assert for stats locking

    ps authored
    Summary: stats are gotten via TLS, so there's no reason to assert verify the threadid when locking.  This also resolves a bug where stats aggregation runs from any thread and therefore the assert would be wrong.
    
    Reviewed By: marc
    
    Test Plan: Ran in production before and got an assertion before.  Ran in production after this change and the assertion did not come up.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@123764 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 24, 2008
  1. make cache_lock adaptive

    ps authored
    Summary: the cache_lock in memcached is now our "giant" lock and is a source of much contention.  by making it adaptive, performance on an 8 core system with 8 server threads goes from 240,000 gets/s to over 300,000.
    
    Reviewed By: sgrimm
    
    Test Plan: ran stress test
               
               blasted with both tcp and udp requests
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@123267 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. move stats collection in memcached to per-thread

    ps authored
    Summary: Remove the global stats lock and collect stats per thread and aggregate only when the stats command is issued.
             
             The prefix, bucket and cost-benefit stats still use a global lock, but they are not enabled in normal operation, so they were not addressed here.
             
             Prior to this change, on an 8 core system, peak req/s is around 160,000 with 8 server threads.  With this change, peak req/s is 240,000.
    
    Reviewed By: marc
    
    Test Plan: Ran tester with asserts on and off.   Ran libmcc against server and verified stats matched what libmcc tester reports.
               
               Have not run in production yet.
               
               Have not blasted with UDP other than using libmcc
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@123229 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 18, 2008
  1. flat allocator broken by my last change

    ttung authored
    Summary: oops, i'm dumb.
    
    Reviewed By: ps
    
    Test Plan: compiled.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@122293 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. connection buffers are per-thread to avoid lock contention

    ttung authored
    Summary: - A connection buffer group is created per thread.  It gets the thread id so if asserts are compiled in, it can determine if it is being used by a different thread.
             - conn_new and dispatch_conn_new no longer needs an initial read buffer size.  it is meaningless in the context of connection buffers.
             
             Minor change:
             - udp support for stress test
    
    Reviewed By: ps
    
    Test Plan: ran stress test with asserts on for 1 hr with 4 clients that reconnect every 15 minutes.
               
               ran in production with asserts on for 10 minutes (and still going...)
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@122288 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 16, 2008
  1. killed non-threaded operation

    ttung authored
    Summary: self explanatory
    
    Reviewed By: ps
    
    Test Plan: ran stress test for 1 hr
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@121929 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 9, 2008
  1. crank up the udp receive buffer to avoid dropping gets

    ttung authored
    Summary: maximize_sndbuf was broken in that it was expecting setsockopt to fail when being asked to set the send buffer to a size larger than permitted.  it actually might not, so use getsockopt to observe the effect of the setsockopt.
    
    Reviewed By: ps
    
    Test Plan: ran with -vv
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@120646 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 4, 2008
  1. ip stamp not refreshed when reusing connection

    ttung authored
    Summary: when we reuse a connection, we don't save the ip address.  therefore, we write the ip address of a previous connection instead of the current ip address.
             
             some tweaking to the stress test loader to find the shared library.
    
    Reviewed By: ps
    
    Test Plan: ran with asserts on, spawned multiple connections and the assert did not fire.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@119996 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Aug 28, 2008
  1. fix slab allocator

    ttung authored
    Summary: slab allocator build was broken by last checkin (allowing keys to be split across chunks in flat allocator)
    
    Reviewed By: marc
    
    Test Plan: ran stress test for about 1 hour
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@118842 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Aug 21, 2008
  1. open the maps file before we drop root privileges

    ttung authored
    Summary: once we drop root privileges, we cannot open the maps file.
    
    Reviewed By: ps
    
    Test Plan: started memcached with sudo and -u nobody.  did a stats maps.  flooded memcached with sets to force it to do a brk.  did a stats maps.  diffed the outputs to verify that rewinding and rereading the file works.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@117510 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jul 29, 2008
  1. the flat allocator uses space inefficiently when there are keys that …

    ttung authored
    …don't fit in a small chunk
    
    Summary: If we size small chunks too big, we're wasting space on small items.  If we size small chunks too small, then keys that don't fit in a small chunk get autopromoted to a large chunk.  That's pretty wasteful too.  So this change allows us to split keys across multiple chunks.
             
             minor tweaks:
             - use memcmp instead of strncmp (so I can stop having to look up the definition of strncmp with respect to null termination).
             - #defined the format string for stats_prefix_dump so we can get compiler warnings when the format makes no sense.
    
    Reviewed By: ps
    
    Test Plan: libmcc test/test.py passes.
               flat storage unit tests pass.
               stress test passes.
               ran in production for a few days without any issues.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@112894 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jul 8, 2008
  1. merge LRU queues, other fixes.

    ttung authored
    Summary: What's life without a memcached diff in diffcamp? :)
             
             There are two key changes:
             
             1) here is that there is now only one LRU queue.  There's no benefit to maintaining two queues, but offers simpler code.
             2) do_add_delta takes a key/nkey pair instead of an item pointer.  this has two key advantages:
             a) it avoids us having to look up the item again when we don't do an update-in-place, since there is no race condition between the item_get and the add_delta locks.
             b) it is enormously helpful when we allow the splitting of keys across data chunks, as now we have a contiguous copy of the key.
             
             Other changes include:
             - warnings are now errors, except "deprecated" warnings (generated by OSX, ugh) which are ignored.
             - assoc_delete no longer requires that the item we delete be the one expected, since this assumption can be wrong when there are race conditions.
             - prefix stats now take a key/nkey pair instead of just key, since keys are no longer null-terminated.
             - append_to_buffer didn't properly reserve space for the terminator.
    
    Reviewed By: marc
    
    Test Plan: stress test passes.
               libmcc tests pass.
               flat allocator unit tests pass.
               ran in production.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@108366 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jun 16, 2008
  1. connection buffer sharing

    ttung authored
    Summary: - create a pool of connection buffers.  these are giant buffers that are just mmaped.  if we don't use it all, then we never fault in the page.
             - track the maximum usage of a buffer.  the client code must report how much of the buffer it used, otherwise the module assumes the entire buffer was touched.
             - if a buffer goes beyond a certain limit, we throw it away back to the OS.  if a global limit is hit, we also start reclaiming free buffers.
             - when getting a buffer, the module will always return the buffer that was used the most.  this allows us to minimize the amount of memory touched.
    
    Reviewed By: ps
    
    Test Plan: - ran the stress test against memcached with alloc_conn_buffer randomly returning NULL.  libmcc reported a lot of errors (not surprising) but memcached did not crash.
               - ran with freelist_check on, which ensures that the connection buffer free list is sane.
               - ran with conn_buffer_corruption_detection, which takes every buffer returned from memcached, and marks it unreadable/unwritable.  if memcached subsequently accesses this memory, it will segfault.  this ran fine until the OS refused to give us the same page back, but it was at least a few minutes.
               - the test/conn_buffer_test/* code is a stub that i never finished since the two checks embedded in the code are pretty thorough.  maybe one day. :)
               - this has run in production.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@104567 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jun 12, 2008
  1. fixes to item_walk, a unit test for item_walk

    ttung authored
    Summary: item_walk had some bugs for extracting the middle part of an item.  we never do that (we get the entire item for arithmetic/send/receive, and we get the end for memset/stamping), so this probably shouldn't actually make a difference.  still, it is good to fix.
             
             also added some unit tests for item_walk (which is how the bugs were found).
             
             This includes http://www.dev.facebook.com/intern/diffcamp/?tab=review&revisionID=13628
    
    Reviewed By: marc
    
    Test Plan: all flat allocator tests pass
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@103964 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. fixes to flat allocator, stats gathering, etc.

    ttung authored
    Summary: general fixes:
             1) account for expire items properly in stats.
             2) added a stat-bucket test.
             3) renamed gets to hits for stats-by-bucket.
             4) modified tests to check expire stats.
             
             flat allocator fixes:
             1) flat allocator is sometimes more efficient with large objects.  fixed evict tests so that they work with the flat allocator.
             2) item_memcpy_from, item_memcpy_to, item_memset, item_strtoul, add_item_to_iov, item_setup_receive all share the basic framework.  refactored the code into an ITEM_WALK macro so that fixes can be shared.
    
    Reviewed By: marc
    
    Test Plan: the following tests pass with the flat allocator:
               test/{cost_benefit_stats,metaget_test,prefix_stats,stats_buckets}.py pass
               libmcc/test/test.py -a 
               libmcc/test/test.py -a --binary
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@103932 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 28, 2008
  1. Do not hard code the use of -ldl

    ps authored
    Summary: Not all operating system have -ldl, so check for it's existence with autoconf.
    
    Reviewed By: ttung
    
    Test Plan: Builds on systems without -ldl and systems with -ldl.
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@100796 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 15, 2008
  1. Test for client support of udp reply ports

    ps authored
    Summary: Abuse the last 2 bytes of the UDP header to encode the number of udp reply ports the client supports.  If the client supports less ports than the server has configured, use the receive socket to transmit the reply on.
             
             While I'm here, make sure no dns requests are done in allocate_udp_reply_port.
    
    Reviewed By: marc
    
    Test Plan: Compiled libmcc without udp reply ports and responses were sent out on port 11300.  Compiled libmcc with udp reply ports and saw responses come from configured reply ports.  Set the # of reply ports to < # of threads in memcached, and the responses came from port 11300.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@98902 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 14, 2008
  1. re-add file lost in merge

    ttung authored
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@98519 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. move refactor tree to trunk

    ttung authored
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@98513 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  3. last port from trunk to storage refactor tree

    ttung authored
    Summary: Porting paul's multi-udp port changes to refactor tree.
             
             With this change, the storage refactor tree will have all of the changes in trunk.  Following this checkin, I plan on copying everything in the storage refactor tree back to trunk, i.e., https://trac.facebook.com/trac/browser/projects/memcached/branches/memcached-storage to https://trac.facebook.com/trac/browser/projects/memcached/trunk
             
             Please let me know if you have any objections.  I'm expanding the audience a bit on the CRR as to give potentially interested parties a chance to object.
    
    Reviewed By: ps
    
    Test Plan: configure and build.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98503 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 13, 2008
  1. port changes from trunk to storage refactor tree

    ttung authored
    Summary: most of these changes have been reviewed before.  the key change is for last-update time and ip address timestamping for flat allocator.
    
    Reviewed By: marc
    
    Test Plan: libmcc test/test.py -a passes
               libmcc test/test.py -a --binary passes
               
               test/prefix_stats.py passes
               test/cost_benefit_stats.py passes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98438 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. removed two SConstruct files

    ttung authored
    Summary: use auto* for stress test, moved metaget into tests directory.
    
    Reviewed By: marc
    
    Test Plan: ran bootstrap, configure, and make for stress test.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98412 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 8, 2008
  1. Allocate per-thread udp reply ports

    ps authored
    Summary: For UDP on Linux, the transmit path is heavily contended when multiple threads are transmitting on the same socket.  Allocate per-thread udp socket to sent response packets to.
             
             Feature is only used when configured with --enable-udp-reply-ports and the -x option given on the command line.
    
    Reviewed By: ttung
    
    Test Plan: modified libmcc to not ignore packets from different source ports.  ran load-mcc against memcache server and pushed server to 200,000 pps where before the limit was ~65,000 pps
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@97704 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. Fix build when stat buckets or cost-benefit-stats are not enabled

    ps authored
    Summary: memcached fails to build if not configured with --enable-stat-buckets   and  --enable-cost-benefit-stats
    
    Reviewed By: marc
    
    Test Plan: it builds.. no real functional changes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@97575 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 7, 2008
  1. add cost benefit stats to memcached

    ttung authored
    Summary: cost: slot-seconds
             benefit: hits
             
             additional tweaks:
             1) merged calls for size-buckets and cost-benefit stat collection
             2) removed \r\n from consideration in stat bytecounting
             3) removed the number of STAT_LOCK() and STAT_UNLOCK() calls by merging stat gathering blocks of code together
             4) "lifetime" -> "avrg lifetime" for size buckets
             5) reduced references to current_time, which is volatile
    
    Reviewed By: marc
    
    Test Plan: libmcc/test/test.py -a passes
               test/prefix_stats.py passes
               test/cost_benefit_stats.py passes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@97325 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. binary protocol reply was not getting the flags parameter

    ttung authored
    Summary: binary protocol reply was not getting the flags parameter.
             
             removed some references to current_time by only reading it once per function (it is a volatile).
    
    Reviewed By: marc
    
    Test Plan: test/test.py -a passes
               test/test.py -a --binary passes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@97323 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 6, 2008
  1. fixes to flat allocator, merge from trunk

    ttung authored
    Summary: you guys didn't think you would get off that easy, did you? :)
             
             main changes:
             1) merged from trunk.
             2) instead of using item* in the assoc table, use a generic typedef'ed type called item_ptr_t.  this allows us to use shrunken pointers for flat allocator for the h_next pointers.
             3) use shrunken pointers for h_next pointers.
             4) pass the pointer of the item we expect to delete to assoc_delete.  assert that the item we find based on key,nkey matches that item.
             5) got rid of the conn_t change (good idea, but terrible for merging in the future)
             6) use always_assert for checks that are one-time checks and should always be done.
             7) refactor out type casting from flat_storage module.
             8) added lifetime stat to flat_allocator
             9) added item_memcpy_from to support timestamp retrieval.
             10) added a memory pool interface to all alloc/free/realloc calls.  this allows us to track how much memory each class of buffers is taking.
             
             bug fixes:
             1) the previous implementation of find_unreferenced_broken_chunk used a master refcount per large broken chunk.  this does not work when the body of a small referenced chunk is in the large broken chunk.  now we properly follow the prev pointers to the head block to account for refcount.
             2) we were improperly clobbering the return value from find_unreferenced_broken_chunk
             3) migrating small chunks was always treating the blocks as body blocks, even though it could very well be title blocks.
             4) flushing expired items should handle both lru's.
             5) in item_get_max_riov for flat_allocator, provision an additional riov for the cr-lf.
             6) item_memcpy_to was totally buggy.
    
    Reviewed By: marc
    
    Test Plan: - libmcc test/test.py -a passes.
               - flat_storage unit tests pass.
               - ran stress test with more arithmetic ops to catch some more corner cases
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@97173 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 5, 2008
  1. include stdarg.h when required for varargs on newer gcc

    ps authored
    Summary: CentOS 5 uses gcc 4.1.2 and compiling memcached will not compile unless stdarg.h is included.
    
    Reviewed By: ttung
    
    Test Plan: It compiles, no code changes.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@96857 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 1, 2008
  1. flat memory allocator

    ttung authored
    Summary: Added a flat memory allocator and some tests (unit and stress).
    
             I know this is a huge diff, so maybe we should just get a conference room with all interested parties and walk through the code together.
    
    Reviewed By: marc
    
    Test Plan: existing libmcc tests pass.
               flat storage unit tests pass.
               ran stress tests for about a week.
               ran on production box for about two days.
    
               slab allocator code still works, but has not been pounded on as rigorously.
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@96181 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Apr 25, 2008
  1. timestamp memcached k:v pairs

    ttung authored
    Summary: it->time is last referenced time, not last updated time.  so write the last updated time in the same fashion as the ip address -- if the slack space permits it to be written.
             
             improved metaget test.
             
             changes so that memcached will build with osx < 10.5
    
    Reviewed By: marc
    
    Test Plan: libmcc test/test.py -a passes
               3-client stress test ran without errors for 90 minutes.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@95534 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. stat collection fixes, some general fixes

    ttung authored
    Summary: 1) stat by bucket collection is enabled by autotools.
             2) added arith cmds and arith hits counters
             3) refcount should not be blindly incremented, as it could potentially overflow (though unlikely).
             4) errno should be initialized to zero prior to strtol/strtoul
    
    Reviewed By: ps
    
    Test Plan: stress test passes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@95531 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Something went wrong with that request. Please try again.