Permalink
Commits on Jan 2, 2015
  1. Set optional 'static' for Quicklist+Redis

    This also defines REDIS_STATIC='' for building everything
    inside src/ and everything inside deps/lua/.
    committed Dec 20, 2014
  2. Add more quicklist info to DEBUG OBJECT

    Adds: ql_compressed (boolean, 1 if compression enabled for list, 0
    otherwise)
    Adds: ql_uncompressed_size (actual uncompressed size of all quicklistNodes)
    Adds: ql_ziplist_max (quicklist max ziplist fill factor)
    
    Compression ratio of the list is then ql_uncompressed_size / serializedlength
    
    We report ql_uncompressed_size for all quicklists because serializedlength
    is a _compressed_ representation anyway.
    
    Sample output from a large list:
    127.0.0.1:6379> llen abc
    (integer) 38370061
    127.0.0.1:6379> debug object abc
    Value at:0x7ff97b51d140 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718164 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:0 ql_uncompressed_size:1643187761
    (1.36s)
    
    The 1.36s result time is because rdbSavedObjectLen() is serializing the
    object, not because of any new stats reporting.
    
    If we run DEBUG OBJECT on a compressed list, DEBUG OBJECT takes almost *zero*
    time because rdbSavedObjectLen() reuses already-compressed ziplists:
    127.0.0.1:6379> debug object abc
    Value at:0x7fe5c5800040 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718109 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:1 ql_uncompressed_size:1643187761
    committed Dec 19, 2014
  3. Config: Add quicklist, remove old list options

    This removes:
      - list-max-ziplist-entries
      - list-max-ziplist-value
    
    This adds:
      - list-max-ziplist-size
      - list-compress-depth
    
    Also updates config file with new sections and updates
    tests to use quicklist settings instead of old list settings.
    committed Dec 16, 2014
  4. Add branch prediction hints to quicklist

    Actually makes a noticeable difference.
    
    Branch hints were selected based on profiler hotspots.
    committed Dec 16, 2014
  5. Cleanup quicklist style

    Small fixes due to a new version of clang-format (it's less
    crazy than the older version).
    committed Dec 30, 2014
  6. Allow compression of interior quicklist nodes

    Let user set how many nodes to *not* compress.
    
    We can specify a compression "depth" of how many nodes
    to leave uncompressed on each end of the quicklist.
    
    Depth 0 = disable compression.
    Depth 1 = only leave head/tail uncompressed.
      - (read as: "skip 1 node on each end of the list before compressing")
    Depth 2 = leave head, head->next, tail->prev, tail uncompressed.
      - ("skip 2 nodes on each end of the list before compressing")
    Depth 3 = Depth 2 + head->next->next + tail->prev->prev
      - ("skip 3 nodes...")
    etc.
    
    This also:
      - updates RDB storage to use native quicklist compression (if node is
        already compressed) instead of uncompressing, generating the RDB string,
        then re-compressing the quicklist node.
      - internalizes the "fill" parameter for the quicklist so we don't
        need to pass it to _every_ function.  Now it's just a property of
        the list.
      - allows a runtime-configurable compression option, so we can
        expose a compresion parameter in the configuration file if people
        want to trade slight request-per-second performance for up to 90%+
        memory savings in some situations.
      - updates the quicklist tests to do multiple passes: 200k+ tests now.
    committed Dec 11, 2014
  7. Add quicklist info to DEBUG OBJECT

    Added field 'ql_nodes' and 'ql_avg_per_node'.
    
    ql_nodes is the number of quicklist nodes in the quicklist.
    ql_avg_node is the average fill level in each quicklist node. (LLEN / QL_NODES)
    
    Sample output:
    127.0.0.1:6379> DEBUG object b
    Value at:0x7fa42bf2fed0 refcount:1 encoding:quicklist serializedlength:18489 lru:8983768 lru_seconds_idle:3 ql_nodes:430 ql_avg_per_node:511.73
    127.0.0.1:6379> llen b
    (integer) 220044
    committed Dec 11, 2014
  8. Remove malloc failure checks

    We trust zmalloc to kill the whole process on memory failure
    committed Dec 11, 2014
  9. Increase test size for migrating large values

    Previously, the old test ran 5,000 loops and used about 500k.
    
    With quicklist, storing those same 5,000 loops takes up 24k, so the
    "large value check" failed!
    
    This increases the test to 20,000 loops which makes the object dump 96k.
    committed Dec 10, 2014
  10. Convert quicklist RDB to store ziplist nodes

    Turns out it's a huge improvement during save/reload/migrate/restore
    because, with compression enabled, we're compressing 4k or 8k
    chunks of data consisting of multiple elements in one ziplist
    instead of compressing series of smaller individual elements.
    committed Dec 10, 2014
  11. Convert RDB ziplist loading to sdsnative()

    This saves us an unnecessary zmalloc, memcpy, and two frees.
    committed Dec 10, 2014
  12. Add sdsnative()

    Use the existing memory space for an SDS to convert it to a regular
    character buffer so we don't need to allocate duplicate space just
    to extract a usable buffer for native operations.
    committed Dec 10, 2014
  13. Add adaptive quicklist fill factor

    Fill factor now has two options:
      - negative (1-5) for size-based ziplist filling
      - positive for length-based ziplist filling with implicit size cap.
    
    Negative offsets define ziplist size limits of:
      -1: 4k
      -2: 8k
      -3: 16k
      -4: 32k
      -5: 64k
    
    Positive offsets now automatically limit their max size to 8k.  Any
    elements larger than 8k will be in individual nodes.
    
    Positive ziplist fill factors will keep adding elements
    to a ziplist until one of:
      - ziplist has FILL number of elements
        - or -
      - ziplist grows above our ziplist max size (currently 8k)
    
    When using positive fill factors, if you insert a large
    element (over 8k), that element will automatically allocate
    an individual quicklist node with one element and no other elements will be
    in the same ziplist inside that quicklist node.
    
    When using negative fill factors, elements up to the size
    limit can be added to one quicklist node.  If an element
    is added larger than the max ziplist size, that element
    will be allocated an individual ziplist in a new quicklist node.
    
    Tests also updated to start testing at fill factor -5.
    committed Nov 26, 2014
  14. Free ziplist test lists during tests

    Freeing our test lists helps keep valgrind output clean
    committed Nov 22, 2014
  15. Add ziplistMerge()

    This started out as #2158 by sunheehnus, but I kept rewriting it
    until I could understand things more easily and get a few more
    correctness guarantees out of the readability flow.
    
    The original commit created and returned a new ziplist with the contents of
    both input ziplists, but I prefer to grow one of the input ziplists
    and destroy the other one.
    
    So, instead of malloc+copy as in #2158, the merge now reallocs one of
    the existing ziplists and copies the other ziplist into the new space.
    
    Also added merge test cases to ziplistTest()
    committed Nov 21, 2014
  16. Add quicklist implementation

    This replaces individual ziplist vs. linkedlist representations
    for Redis list operations.
    
    Big thanks for all the reviews and feedback from everybody in
    antirez#2143
    committed Nov 13, 2014
Commits on Dec 23, 2014
  1. Cleanup ziplist valgrind warnings

    Valgrind can't detect 'memset' initializes things, so let's
    statically initialize them to remove some unnecessary warnings.
    committed Nov 13, 2014
  2. Fix ziplist test for pop()

    The previous test wasn't returning the new ziplist, so the test
    was invalid.  Now the test works properly.
    
    These problems were simultaenously discovered in #2154 and that
    PR also had an additional fix we included here.
    committed Nov 13, 2014
  3. Fix ziplistDeleteRange index parameter

    It's valid to delete from negative offsets, so we *don't*
    want unsigned arguments here.
    committed Nov 13, 2014
  4. Fix how zipEntry returns values

    zipEntry was returning a struct, but that caused some
    problems with tests under 32 bit builds.
    
    The tests run better if we operate on structs allocated in the
    caller without worrying about copying on return.
    committed Nov 14, 2014
  5. Allow forcing non-jemalloc build

    committed Nov 13, 2014
  6. Add simple ll2string() tests

    committed Nov 20, 2014
  7. Allow all code tests to run using Redis args

    Previously, many files had individual main() functions for testing,
    but each required being compiled with their own testing flags.
    That gets difficult when you have 8 different flags you need
    to set just to run all tests (plus, some test files required
    other files to be compiled aaginst them, and it seems some didn't
    build at all without including the rest of Redis).
    
    Now all individual test main() funcions are renamed to a test
    function for the file itself and one global REDIS_TEST define enables
    testing across the entire codebase.
    
    Tests can now be run with:
      - `./redis-server test <test>`
    
      e.g. ./redis-server test ziplist
    
    If REDIS_TEST is not defined, then no tests get included and no
    tests are included in the final redis-server binary.
    committed Nov 13, 2014
  8. Remove ziplist compiler warnings

    Only happen when compiled with the test define.
    committed Nov 8, 2014
  9. Add DEBUG JEMALLOC INFO

    Uses jemalloc function malloc_stats_print() to return
    stats about what jemalloc has allocated internally.
    committed Nov 16, 2014
  10. Add addReplyBulkSds() function

    Refactor a common pattern into one function so we don't
    end up with copy/paste programming.
    committed Nov 16, 2014
  11. INFO loading stats: three fixes.

    1. Server unxtime may remain not updated while loading AOF, so ETA is
    not updated correctly.
    
    2. Number of processed byte was not initialized.
    
    3. Possible division by zero condition (likely cause of issue #1932).
    antirez committed Dec 23, 2014
  12. Merge pull request #2227 from mattsta/fix/trib/assignment/master-replica

    Improve redis-trib replica assignment
    antirez committed Dec 23, 2014
  13. Merge pull request #2234 from mattsta/feature/sentinel-info-cache/age

    Add 'age' value to SENTINEL INFO-CACHE
    antirez committed Dec 23, 2014
Commits on Dec 22, 2014
  1. Merge pull request #2229 from advance512/spopWithCount

    Memory leak fixes (+ code style fixes)
    antirez committed Dec 22, 2014