Skip to content
Commits on Nov 5, 2008
  1. auto detect the number of cpus

    ps committed
    Summary: Instead of relying on -t to run on greater than 4 cpu systems, auto detect the number of active processors
    
    Reviewed By: ttung
    
    Test Plan: ran memcached on 8 core machine, 9 threads created.  ran on 4 core machine, 5 threads created.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@130064 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 24, 2008
  1. move stats collection in memcached to per-thread

    ps committed
    Summary: Remove the global stats lock and collect stats per thread and aggregate only when the stats command is issued.
             
             The prefix, bucket and cost-benefit stats still use a global lock, but they are not enabled in normal operation, so they were not addressed here.
             
             Prior to this change, on an 8 core system, peak req/s is around 160,000 with 8 server threads.  With this change, peak req/s is 240,000.
    
    Reviewed By: marc
    
    Test Plan: Ran tester with asserts on and off.   Ran libmcc against server and verified stats matched what libmcc tester reports.
               
               Have not run in production yet.
               
               Have not blasted with UDP other than using libmcc
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@123229 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 18, 2008
  1. connection buffers are per-thread to avoid lock contention

    ttung committed
    Summary: - A connection buffer group is created per thread.  It gets the thread id so if asserts are compiled in, it can determine if it is being used by a different thread.
             - conn_new and dispatch_conn_new no longer needs an initial read buffer size.  it is meaningless in the context of connection buffers.
             
             Minor change:
             - udp support for stress test
    
    Reviewed By: ps
    
    Test Plan: ran stress test with asserts on for 1 hr with 4 clients that reconnect every 15 minutes.
               
               ran in production with asserts on for 10 minutes (and still going...)
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@122288 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 16, 2008
  1. killed non-threaded operation

    ttung committed
    Summary: self explanatory
    
    Reviewed By: ps
    
    Test Plan: ran stress test for 1 hr
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@121929 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 9, 2008
  1. crank up the udp receive buffer to avoid dropping gets

    ttung committed
    Summary: maximize_sndbuf was broken in that it was expecting setsockopt to fail when being asked to set the send buffer to a size larger than permitted.  it actually might not, so use getsockopt to observe the effect of the setsockopt.
    
    Reviewed By: ps
    
    Test Plan: ran with -vv
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@120646 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 4, 2008
  1. ip stamp not refreshed when reusing connection

    ttung committed
    Summary: when we reuse a connection, we don't save the ip address.  therefore, we write the ip address of a previous connection instead of the current ip address.
             
             some tweaking to the stress test loader to find the shared library.
    
    Reviewed By: ps
    
    Test Plan: ran with asserts on, spawned multiple connections and the assert did not fire.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@119996 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Aug 21, 2008
  1. open the maps file before we drop root privileges

    ttung committed
    Summary: once we drop root privileges, we cannot open the maps file.
    
    Reviewed By: ps
    
    Test Plan: started memcached with sudo and -u nobody.  did a stats maps.  flooded memcached with sets to force it to do a brk.  did a stats maps.  diffed the outputs to verify that rewinding and rereading the file works.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@117510 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jul 29, 2008
  1. the flat allocator uses space inefficiently when there are keys that …

    ttung committed
    …don't fit in a small chunk
    
    Summary: If we size small chunks too big, we're wasting space on small items.  If we size small chunks too small, then keys that don't fit in a small chunk get autopromoted to a large chunk.  That's pretty wasteful too.  So this change allows us to split keys across multiple chunks.
             
             minor tweaks:
             - use memcmp instead of strncmp (so I can stop having to look up the definition of strncmp with respect to null termination).
             - #defined the format string for stats_prefix_dump so we can get compiler warnings when the format makes no sense.
    
    Reviewed By: ps
    
    Test Plan: libmcc test/test.py passes.
               flat storage unit tests pass.
               stress test passes.
               ran in production for a few days without any issues.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@112894 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jul 8, 2008
  1. merge LRU queues, other fixes.

    ttung committed
    Summary: What's life without a memcached diff in diffcamp? :)
             
             There are two key changes:
             
             1) here is that there is now only one LRU queue.  There's no benefit to maintaining two queues, but offers simpler code.
             2) do_add_delta takes a key/nkey pair instead of an item pointer.  this has two key advantages:
             a) it avoids us having to look up the item again when we don't do an update-in-place, since there is no race condition between the item_get and the add_delta locks.
             b) it is enormously helpful when we allow the splitting of keys across data chunks, as now we have a contiguous copy of the key.
             
             Other changes include:
             - warnings are now errors, except "deprecated" warnings (generated by OSX, ugh) which are ignored.
             - assoc_delete no longer requires that the item we delete be the one expected, since this assumption can be wrong when there are race conditions.
             - prefix stats now take a key/nkey pair instead of just key, since keys are no longer null-terminated.
             - append_to_buffer didn't properly reserve space for the terminator.
    
    Reviewed By: marc
    
    Test Plan: stress test passes.
               libmcc tests pass.
               flat allocator unit tests pass.
               ran in production.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@108366 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jun 16, 2008
  1. connection buffer sharing

    ttung committed
    Summary: - create a pool of connection buffers.  these are giant buffers that are just mmaped.  if we don't use it all, then we never fault in the page.
             - track the maximum usage of a buffer.  the client code must report how much of the buffer it used, otherwise the module assumes the entire buffer was touched.
             - if a buffer goes beyond a certain limit, we throw it away back to the OS.  if a global limit is hit, we also start reclaiming free buffers.
             - when getting a buffer, the module will always return the buffer that was used the most.  this allows us to minimize the amount of memory touched.
    
    Reviewed By: ps
    
    Test Plan: - ran the stress test against memcached with alloc_conn_buffer randomly returning NULL.  libmcc reported a lot of errors (not surprising) but memcached did not crash.
               - ran with freelist_check on, which ensures that the connection buffer free list is sane.
               - ran with conn_buffer_corruption_detection, which takes every buffer returned from memcached, and marks it unreadable/unwritable.  if memcached subsequently accesses this memory, it will segfault.  this ran fine until the OS refused to give us the same page back, but it was at least a few minutes.
               - the test/conn_buffer_test/* code is a stub that i never finished since the two checks embedded in the code are pretty thorough.  maybe one day. :)
               - this has run in production.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@104567 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jun 12, 2008
  1. fixes to item_walk, a unit test for item_walk

    ttung committed
    Summary: item_walk had some bugs for extracting the middle part of an item.  we never do that (we get the entire item for arithmetic/send/receive, and we get the end for memset/stamping), so this probably shouldn't actually make a difference.  still, it is good to fix.
             
             also added some unit tests for item_walk (which is how the bugs were found).
             
             This includes http://www.dev.facebook.com/intern/diffcamp/?tab=review&revisionID=13628
    
    Reviewed By: marc
    
    Test Plan: all flat allocator tests pass
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@103964 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. fixes to flat allocator, stats gathering, etc.

    ttung committed
    Summary: general fixes:
             1) account for expire items properly in stats.
             2) added a stat-bucket test.
             3) renamed gets to hits for stats-by-bucket.
             4) modified tests to check expire stats.
             
             flat allocator fixes:
             1) flat allocator is sometimes more efficient with large objects.  fixed evict tests so that they work with the flat allocator.
             2) item_memcpy_from, item_memcpy_to, item_memset, item_strtoul, add_item_to_iov, item_setup_receive all share the basic framework.  refactored the code into an ITEM_WALK macro so that fixes can be shared.
    
    Reviewed By: marc
    
    Test Plan: the following tests pass with the flat allocator:
               test/{cost_benefit_stats,metaget_test,prefix_stats,stats_buckets}.py pass
               libmcc/test/test.py -a 
               libmcc/test/test.py -a --binary
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@103932 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 15, 2008
  1. Test for client support of udp reply ports

    ps committed
    Summary: Abuse the last 2 bytes of the UDP header to encode the number of udp reply ports the client supports.  If the client supports less ports than the server has configured, use the receive socket to transmit the reply on.
             
             While I'm here, make sure no dns requests are done in allocate_udp_reply_port.
    
    Reviewed By: marc
    
    Test Plan: Compiled libmcc without udp reply ports and responses were sent out on port 11300.  Compiled libmcc with udp reply ports and saw responses come from configured reply ports.  Set the # of reply ports to < # of threads in memcached, and the responses came from port 11300.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@98902 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 14, 2008
  1. last port from trunk to storage refactor tree

    ttung committed
    Summary: Porting paul's multi-udp port changes to refactor tree.
             
             With this change, the storage refactor tree will have all of the changes in trunk.  Following this checkin, I plan on copying everything in the storage refactor tree back to trunk, i.e., https://trac.facebook.com/trac/browser/projects/memcached/branches/memcached-storage to https://trac.facebook.com/trac/browser/projects/memcached/trunk
             
             Please let me know if you have any objections.  I'm expanding the audience a bit on the CRR as to give potentially interested parties a chance to object.
    
    Reviewed By: ps
    
    Test Plan: configure and build.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98503 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 13, 2008
  1. port changes from trunk to storage refactor tree

    ttung committed
    Summary: most of these changes have been reviewed before.  the key change is for last-update time and ip address timestamping for flat allocator.
    
    Reviewed By: marc
    
    Test Plan: libmcc test/test.py -a passes
               libmcc test/test.py -a --binary passes
               
               test/prefix_stats.py passes
               test/cost_benefit_stats.py passes
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98438 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
  2. removed two SConstruct files

    ttung committed
    Summary: use auto* for stress test, moved metaget into tests directory.
    
    Reviewed By: marc
    
    Test Plan: ran bootstrap, configure, and make for stress test.
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@98412 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 6, 2008
  1. fixes to flat allocator, merge from trunk

    ttung committed
    Summary: you guys didn't think you would get off that easy, did you? :)
             
             main changes:
             1) merged from trunk.
             2) instead of using item* in the assoc table, use a generic typedef'ed type called item_ptr_t.  this allows us to use shrunken pointers for flat allocator for the h_next pointers.
             3) use shrunken pointers for h_next pointers.
             4) pass the pointer of the item we expect to delete to assoc_delete.  assert that the item we find based on key,nkey matches that item.
             5) got rid of the conn_t change (good idea, but terrible for merging in the future)
             6) use always_assert for checks that are one-time checks and should always be done.
             7) refactor out type casting from flat_storage module.
             8) added lifetime stat to flat_allocator
             9) added item_memcpy_from to support timestamp retrieval.
             10) added a memory pool interface to all alloc/free/realloc calls.  this allows us to track how much memory each class of buffers is taking.
             
             bug fixes:
             1) the previous implementation of find_unreferenced_broken_chunk used a master refcount per large broken chunk.  this does not work when the body of a small referenced chunk is in the large broken chunk.  now we properly follow the prev pointers to the head block to account for refcount.
             2) we were improperly clobbering the return value from find_unreferenced_broken_chunk
             3) migrating small chunks was always treating the blocks as body blocks, even though it could very well be title blocks.
             4) flushing expired items should handle both lru's.
             5) in item_get_max_riov for flat_allocator, provision an additional riov for the cr-lf.
             6) item_memcpy_to was totally buggy.
    
    Reviewed By: marc
    
    Test Plan: - libmcc test/test.py -a passes.
               - flat_storage unit tests pass.
               - ran stress test with more arithmetic ops to catch some more corner cases
    
    Revert: OK
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@97173 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on May 1, 2008
  1. flat memory allocator

    ttung committed
    Summary: Added a flat memory allocator and some tests (unit and stress).
    
             I know this is a huge diff, so maybe we should just get a conference room with all interested parties and walk through the code together.
    
    Reviewed By: marc
    
    Test Plan: existing libmcc tests pass.
               flat storage unit tests pass.
               ran stress tests for about a week.
               ran on production box for about two days.
    
               slab allocator code still works, but has not been pounded on as rigorously.
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@96181 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Mar 5, 2008
Commits on Jan 3, 2008
  1. re-factored memcached to accommodate alternative storage backends

    ttung committed
    Summary: 1) all receives and transmits of "value" data is now done using iovecs.  this allows for storage models in which data is broken up into chunks.
             2) abstracted access to ITEM_data by adding a shim layer in items_support.h.
             3) moved all slab code to slabs.c
             4) ensured that we're not trying to receive udp data in handle_direct_receive(..) for binary protocol.
             5) added a --enable-slab-allocator to auto*.  defaults to yes.
             6) instead of storing length+flags in ascii, store it in binary.  render
             it out to c->wbuf when transmitting in ascii.
             
             small changes:
             1) removed exptime field from binary_protocol.h for GETs.
             
             I plan on cleaning up the organization of the header files, but I don't want to introduce changes that should have zero impact on functionality into this diff, hopefully making it easier to review.
    
    Reviewed By: sgrimm
    
    Test Plan: ran the following tests:
               1) test/stats.py -a
               2) $libmcc/test/bp.py -a (simple binary protocol tests)
               3) $libmcc/test/test.py -a
               4) $libmcc/test/test.py -a --binary (#3, except all in binary protocol)
               5) $php_mcc/test/test.php
    
    Revert: if this is rolled out AND binary protocol is being used, it must be simultaneously rolled back along with the libmcc change that removes the exptime field in the binary protocol.  otherwise many bad things will happen.
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/branches/memcached-storage@76291 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Nov 12, 2007
  1. bucketize all mc transactions by size

    ttung committed
    Summary: -Create buckets by key+value size.
             -Record number and type of transactions.
             
             additional changes:
             -proper stat accounting in binary protocol
             -use PRINTF_INT64_MODIFIER instead of %ll
             -added test/pexpect.py so the stats code can be tested on machines that don't have pexpect installed.
             -fixed stats code for 64-bit machines
    
    Reviewed By: sgrimm
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@69245 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Nov 11, 2007
  1. Increase TCP listen queue depth to 4096

    sgrimm committed
    Reviewed by: marc
    
    Revert: ok
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@69210 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 31, 2007
  1. squashed warnings in memcached

    ttung committed
    Summary: - added -Wall to compile flags and squashed most warnings.
             - updated header files with #ifdef ... #endif
    
    Reviewed By: sgrimm
    
    Test Plan: test/stats.py passes
               
               libmcc/test/test.py -a passes with --enable-threads and without --enable-threads
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@66513 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 18, 2007
  1. UDP header now contains offset to first protocol data

    sgrimm committed
    Summary: This implements the "offset to first byte of protocol text"
             feature that should make UDP resynchronization easier.
    
    Reviewed By: marc
    
    Test Plan: Updated the UDP test cases in the built-in test suite.
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@64270 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 15, 2007
  1. [memcached] fixes for slab reallocation logic

    hzhao committed
    Summary: (1) fixed to call do_item_unlink() so not to deadlock; (2) fixed eviction count to count pushed out items without expiration time; (3) removed "rebalance" command
    
    Reviewed By: sgrimm
    Test Plan: mc078.sf2p
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@63668 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Oct 8, 2007
  1. Summary: Add byte counting by prefix to memcached. All keys that don'…

    ttung committed
    …t have a prefix are grouped as a wildcard class.
    
    Reviewed by: sgrimm
    
    Test plan: test/test.py -a in libmcc passes.  also wrote a set of directed
    tests that verify the counters.
    
    Revert plan: revertible
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@62726 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 19, 2007
  1. Update with the latest changes from the public memcached source

    sgrimm committed
    Summary: This is all cosmetic stuff and documentation, no functional change.
    
    Reviewed By: hzhao
    
    Test Plan: Compiles and passes test suite in both single and multithreaded modes.
    
    Revert: OK
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@60239 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Sep 12, 2007
  1. [memcache] added slab reallocation

    hzhao committed
    Summary: so we implemented an algorithm of doing slab reallocation, based on
      eviction rate and unique hit rate of different slab classses.
    
    Reviewed By: sgrimm
    
    Test Plan: mirroring test by setting three site vars:
    
            MEMCACHED_SERVER_MIRRORING = 1000 (if -1, mirroring dev servers)
            MEMCACHED_SERVER_MIRROR_TO = 10.8.110.106:11131
            MEMCACHED_SERVER_MIRROR_WHOM = 10.1.201.151:11130
    
      Then 1000 (MEMCACHED_SERVER_MIRRORING) out of 1000 web servers will mirror
      traffic to _WHOM to _TO.
    
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@59386 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Aug 22, 2007
  1. Summary: Added binary protocol support to memcached. There are a few …

    ttung committed
    …minor tweaks to be added later, but for the most part, this is complete.
    
    The future additions are:
    1) Some functions (bp_write_err_msg, bp_try_read_network, bp_try_read_udp, and bp_transmit) are very similar to their non-bp counterparts.  It's probably a good idea to merge them.  For this first iteration, I'm going to leave this alone.
    
    2) No support for writing UDP header hints (hints to allow the protocol to handle missing packets).
    
    3) Correct some misnomers (i.e., header vs empty header vs body).
    
    4) memcached without any arguments should start up listening to port
    11211.  If any binary port (TCP or UDP) is specified, the ascii port is
    not listened to.
    
    Reviewed by: marc
    
    Test plan: test/test.py -a in libmemcached passes.  test/test.py -a --binary-mode in an experimental branch of libmemcached passes.
    
    Revert plan: revertible
    
    
    git-svn-id: http://svn.facebook.com/svnroot/projects/memcached/trunk@56082 2c7ba8d8-a2f7-0310-a573-de162e16dcc7
Commits on Jul 8, 2007
Commits on Jun 20, 2007
  1. Solaris portability fixes from Trond Norbye

    plindner committed
    git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@570 b0b603af-a30f-0410-a34e-baf09ae79d0b
Commits on May 7, 2007
  1. some minor prototype fixes

    plindner committed
    git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@552 b0b603af-a30f-0410-a34e-baf09ae79d0b
  2. cleanup unistd.h, better spec file, remove warnings from threads.c

    plindner committed
    git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@551 b0b603af-a30f-0410-a34e-baf09ae79d0b
Commits on Apr 18, 2007
  1. Bug fix from Xueron Nee <xueron@gmail.com> for -l args problems.

    plindner committed
    Added test cases to t/00-startup.t to insure this doesn't break.
    
    
    git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@521 b0b603af-a30f-0410-a34e-baf09ae79d0b
Commits on Apr 17, 2007
Something went wrong with that request. Please try again.