Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Commits on Feb 24, 2012
  1. @dustin

    Introduced LevelDB.

    dustin authored
    Change-Id: I1353c9c86b4bfae86d98524cbcfe1a827ab5c703
  2. @chiyoung @trondn

    Delete uncommitted log items from hash table and data store.

    chiyoung authored trondn committed
    If a crash happens while writing items into the underlying data
    store and the mutation log, there might be uncommitted items from
    the mutation log. For consistency, we simply delete those
    uncommitted items from the hash table and underlying data store.
    
    Change-Id: Iedfedf93b5d17a3a80168529e9aa3d2a77690fe9
    Reviewed-on: http://review.couchbase.org/13574
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  3. @chiyoung

    MB-4828 Check backfill completion in TapProducer::nextFgFetched()

    chiyoung authored
    As backfill is performed by the separate dispatcher thread and
    replication/takeover is performed by the memcached tap thread,
    the tap thread should check the backfill completion before
    poping the next item from the queue.
    
    Change-Id: Ifff228a2e152f0472b1eb9c5bdbf0e14e5d76691
    Reviewed-on: http://review.couchbase.org/13562
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  4. @chiyoung

    Add missing flush params to cbflushctl help output.

    chiyoung authored
    Change-Id: I4419b7cedaa4cae7c056ea43defff70f7197f2a3
    Reviewed-on: http://review.couchbase.org/13508
    Reviewed-by: Michael Wiederhold <mike@couchbase.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  5. @alk @chiyoung

    make sure MAX_LOG_SIZE fits into 32 bit size_t

    alk authored chiyoung committed
    So that build doesn't fail on 32-bit target.
    
    I'm casting -1 to unsigned int first (assuming unsigned int is 32 bit
    large which is typical, i'm not sure we can rely on uint32_t being
    available), so that result is not larger than 2^32 which was seemingly
    original intention.
    
    Change-Id: I32746109307376e756aec129bfadf4cc35f1d777
    Reviewed-on: http://review.couchbase.org/13573
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Commits on Feb 23, 2012
  1. @chiyoung @trondn

    Log compactor that compacts a mutation log periodically.

    chiyoung authored trondn committed
    This is the initial implementation of the mutation log compactor.
    The compactor is scheduled periodically and checks the
    following conditions to see if the compaction is required for the
    current mutation log file:
    
    1) current mutation log size > max log size allowed,
    2) num of total logged items > max entry num allowed,
    3) write queue size < queue cap allowed,
    
    If all the above conditions are satisfied, the compactor suspends
    the persistence and then dumps the entire hash table into a new
    log file and finally renames it to the original log file.
    
    Change-Id: Ibfbe3bca339574bebd94de9ac0958005e1193e76
    Reviewed-on: http://review.couchbase.org/13507
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
Commits on Feb 22, 2012
  1. @kbatten

    add replica read to mc_bin_client

    kbatten authored
    Change-Id: Iba4304106f104e17cfdff015d3a11212128e7005
    Reviewed-on: http://review.couchbase.org/13200
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Reviewed-by: Jin Lim <jin@couchbase.com>
    Tested-by: Keith Batten <kbatten@gmail.com>
Commits on Feb 21, 2012
  1. @chiyoung

    Remove unnecessary notifyIOComplete calls in TAP BG fetches.

    chiyoung authored
    Everytime an item is fetch from disk for TAP replication,
    this event is always notified to the corresponding memcached
    connection through TapConnMap::performTapOp method.
    
    Change-Id: I2c64eedee664085e54dcdace806249f42e4dd94a
    Reviewed-on: http://review.couchbase.org/13391
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  2. @chiyoung

    Request an ack iff backfill queue is empty during backfill phase.

    chiyoung authored
    Change-Id: I2ca5d708d3d0375d6e8e39e4f92c9ecb8204e24b
    Reviewed-on: http://review.couchbase.org/13390
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Michael Wiederhold <mike@couchbase.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  3. @chiyoung

    Have only one method in TapProducer to process an BG fetched item

    chiyoung authored
    This change consolidates TapProducer::gotBGItem method into
    its completeBGFetchJob method to process an BG fetched item
    through a single method call.
    
    Change-Id: Ibbb5f931eb6eed74b312b0e3f733a34cb5c10d6e
    Reviewed-on: http://review.couchbase.org/13389
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  4. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      Backport: MB-4805 Fix items lost during startup
    
    Change-Id: Iff05b6565ffc322b64d36a8663e0c7160c154e0a
  5. @trondn @chiyoung

    Backport: MB-4805 Fix items lost during startup

    trondn authored chiyoung committed
    Change-Id: Ibb64c89f3bbe5ebbc436159bc447b15a6c06e1fe
    Reviewed-on: http://review.couchbase.org/13444
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Commits on Feb 20, 2012
  1. @bcui6611

    Add sqlite-kvstore/sqlite-vfs.c to build

    bcui6611 authored
    It will fix the build error on windows.
    
    Change-Id: I80c841f622743ccf19fa1a6a2f49429ca2d725ea
    Reviewed-on: http://review.couchbase.org/13400
    Tested-by: Bin Cui <bin.cui@gmail.com>
    Reviewed-by: Steve Yen <steve.yen@gmail.com>
  2. @trondn @chiyoung

    Remove duplicate code

    trondn authored chiyoung committed
    And fix up a wrong assumption that hrtime is usec in some of the
    tests.
    
    Change-Id: I3a6aaeee36e3d74b8fff47ad26e9fdad1209ccda
    Reviewed-on: http://review.couchbase.org/13382
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  3. @trondn

    Improve error message in exception

    trondn authored
    Change-Id: I9ab9bcab64217810c9ca1d117bb9090534bc1a5f
    Reviewed-on: http://review.couchbase.org/13383
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  4. @trondn

    Remove unused code

    trondn authored
    Change-Id: Id4a9dc502ded5f312212400ed5fea6b6a6617270
    Reviewed-on: http://review.couchbase.org/13381
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  5. @trondn

    Improve error handling for reading mutation log

    trondn authored
    Change-Id: I94406d85f10d4245ee7c8cc04890eaf93fb7eef9
    Reviewed-on: http://review.couchbase.org/13380
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
Commits on Feb 19, 2012
  1. @dustin @trondn

    Singly register VFS per file.

    dustin authored trondn committed
    We'd register the same VFS once every time we opened a given file
    (which is more than once with a concurrent reader).  We just need the
    one.
    
    Change-Id: Ia294cb870e5c51db151e51ca9a67b9edeb48ed42
    Reviewed-on: http://review.couchbase.org/13363
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
Commits on Feb 18, 2012
  1. @trondn @chiyoung

    MB-4805 Fix items lost during startup

    trondn authored chiyoung committed
    Change-Id: I1bec94225963bd21520947a8c2c98918cb17ea79
    Reviewed-on: http://review.couchbase.org/13331
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  2. @chiyoung @trondn

    Rename TapProducer's waitForBackfill to waitForBgFetches

    chiyoung authored trondn committed
    Change-Id: I8f88c76f1fce642e4d8ea16a377ac79342c35db5
    Reviewed-on: http://review.couchbase.org/13359
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  3. @chiyoung @trondn

    Include bg fetched items in the TAP live stream queue size.

    chiyoung authored trondn committed
    Change-Id: Ie5a958dcc0336d3233c0681a2f51f602ae713c86
    Reviewed-on: http://review.couchbase.org/13345
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  4. @chiyoung @trondn

    MB-4813 Maintain TAP backfill queue and backlog size separately.

    chiyoung authored trondn committed
    Backfill rate from disk is usually slower than the replication
    rate if disk IO becomes bottleneck. In this case, the backfill
    backlog stat is usually ranged in a very small number as it is
    currently based on the size of backfill queue in the TAP
    connection.
    
    To address the above issue, we maintain two separate stats for
    backfill. The first stat is the backfill queue size that was
    previously used to represent the total backfill backlogs.
    The second stat is the total backfill backlogs that are
    estimated right before memory/disk backfill task is scheduled,
    and then updated while sending backfill items to the destination.
    
    Change-Id: Icf25dd1cebd872a107646c30d65db084286eed0d
    Reviewed-on: http://review.couchbase.org/13344
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  5. @chiyoung @trondn

    Remove duplicate memory constraint checks on backfill.

    chiyoung authored trondn committed
    Memory constraints are already checked right before scheduling
    a single vbucket backfill task.
    
    Change-Id: I990357358b4d7e3837a32c726bb5012710731be0
    Reviewed-on: http://review.couchbase.org/13343
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  6. @chiyoung @trondn

    Don't reset vbucket stats on a dead vbucket.

    chiyoung authored trondn committed
    As dead vbucket could be set to active or replica vbucket
    during rebalance, its stats shouldn't be reset when its state is
    set to dead or changed from dead state.
    
    Change-Id: Ife4801bae741f458fbec8a34fb5c1722aa6b5305
    Reviewed-on: http://review.couchbase.org/13342
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  7. @chiyoung @trondn

    Minor refactoring in TapProducer class's member variables.

    chiyoung authored trondn committed
    This simply reorders the member variables of TapProducer class
    and does minor formatting in comments.
    
    Change-Id: I527f5faec3449469b7a39a743947efa5a9c1fea0
    Reviewed-on: http://review.couchbase.org/13341
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
  8. @chiyoung @trondn

    Checkpoint maintains keys and subset of meta data fields only.

    chiyoung authored trondn committed
    The previous checkpoint implementation requires each checkpoint
    to maintain the list of keys, their values, and all of their
    meta data fields, which can cause very high memory overhead
    in case disk write throughput or replication is very slow.
    
    Based on various use cases for checkpoint, it turns out that it
    is not necessary to maintain the strong consistency across
    checkpoints. This can simplify a checkpoint's item structure,
    which allows us to have keys and subset of their meta data fields
    only. Basically, a checkpoint only contains the information about
    which keys have been mutated and their operation types.
    
    With this simplication, the memory overhead of a single checkpoint
    item (i.e., QueuedItem instance) is reduced from 88 bytes to 40
    bytes. In addition, the closed checkpoint won't point to stale
    Blob value instances anymore.
    
    Change-Id: I8c0273f582a53014ea92dd94ca9194cb9f5bc387
    Reviewed-on: http://review.couchbase.org/13286
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
Commits on Feb 17, 2012
  1. @dustin @chiyoung

    SQLite low-level stats.

    dustin authored chiyoung committed
    Documentation is not included here yet, but the seek stats are
    approximately how far we asked the disk head to move from the position
    we last left it.  The read and write sizes are how big the chunks of
    data were that we were reading and writing.  The times are the amount
    of time it took us to do one of the things it says we did.
    
    Counters:
    
     close:       201
     lock:        1042
     open:        207
     sector_size: 512
     truncate:    0
    
    And timings:
    
     delete (406 total)
        16us - 32us   : (  0.49%)   2
        32us - 64us   : (  0.74%)   1
        64us - 128us  : ( 30.30%) 120 ##########
        128us - 256us : ( 43.60%)  54 ####
        256us - 512us : ( 54.93%)  46 ###
        512us - 1ms   : ( 87.19%) 131 ##########
        1ms - 2ms     : ( 99.75%)  51 ####
        2ms - 4ms     : (100.00%)   1
     readSeek (3710601 total)
        0 - 2         : (  0.01%)     411
        16 - 32       : (  0.01%)      89
        64 - 128      : (  0.01%)       5
        256 - 512     : (  0.07%)    2098
        512 - 1KB     : (  0.07%)      16
        1KB - 2KB     : (  9.40%)  346284 ##
        2KB - 4KB     : ( 11.27%)   69202
        4KB - 8KB     : ( 13.46%)   81508
        8KB - 16KB    : ( 14.77%)   48473
        128KB - 256KB : ( 14.94%)    6146
        256KB - 512KB : ( 15.17%)    8698
        512KB - 1MB   : ( 15.18%)     171
        1MB - 2MB     : ( 97.41%) 3051513 ########################
        2MB - 4MB     : ( 98.35%)   34781
        4MB - 8MB     : ( 98.36%)     231
        8MB - 16MB    : ( 98.37%)     456
        16MB - 32MB   : (100.00%)   60519
     readSize (3710604 total)
        8 - 16        : (  0.06%)    2187
        16 - 32       : (  0.08%)     689
        64 - 128      : (  0.08%)       5
        1KB - 2KB     : (100.00%) 3707723 #############################
     readTime (3710597 total)
        0 - 1us       : (  0.04%)    1387
        1us - 2us     : ( 79.08%) 2932976 #######################
        2us - 4us     : ( 97.87%)  697021 #####
        4us - 8us     : ( 99.57%)   63127
        8us - 16us    : ( 99.88%)   11589
        16us - 32us   : ( 99.97%)    3467
        32us - 64us   : (100.00%)     865
        64us - 128us  : (100.00%)     127
        128us - 256us : (100.00%)      13
        256us - 512us : (100.00%)      13
        512us - 1ms   : (100.00%)      12
     sync (4780 total)
        64us - 128us  : ( 29.10%) 1391 #########
        128us - 256us : ( 45.63%)  790 #####
        256us - 512us : ( 47.55%)   92
        512us - 1ms   : ( 47.87%)   15
        1ms - 2ms     : ( 49.00%)   54
        2ms - 4ms     : ( 51.84%)  136
        4ms - 8ms     : ( 90.17%) 1832 ############
        8ms - 16ms    : ( 92.07%)   91
        16ms - 32ms   : ( 94.23%)  103
        32ms - 65ms   : ( 95.96%)   83
        65ms - 131ms  : ( 97.82%)   89
        131ms - 262ms : ( 99.31%)   71
        262ms - 524ms : ( 99.90%)   28
        524ms - 1s    : (100.00%)    5
     writeSeek (14502993 total)
        4 - 8         : ( 49.99%) 7249692 ##############
        8 - 16        : ( 49.99%)     269
        16 - 32       : ( 50.00%)     954
        128 - 256     : ( 50.00%)      88
        256 - 512     : ( 50.00%)     181
        512 - 1KB     : ( 50.00%)     271
        1KB - 2KB     : ( 78.26%) 4099250 ########
        2KB - 4KB     : ( 78.27%)    1388
        4KB - 8KB     : ( 78.28%)     177
        8KB - 16KB    : ( 78.28%)     122
        16KB - 32KB   : ( 78.28%)      72
        32KB - 64KB   : ( 78.28%)     178
        64KB - 128KB  : ( 78.29%)    1202
        256KB - 512KB : ( 78.29%)      89
        512KB - 1MB   : ( 78.29%)      88
        1MB - 2MB     : ( 99.35%) 3054955 ######
        2MB - 4MB     : ( 99.59%)   34789
        4MB - 8MB     : ( 99.59%)     248
        8MB - 16MB    : ( 99.59%)      90
        16MB - 32MB   : (100.00%)   58890
     writeSize (14503006 total)
        4 - 8         : ( 49.99%) 7249969 ##############
        8 - 16        : ( 50.01%)    2456
        16 - 32       : ( 50.01%)     954
        512 - 1KB     : ( 50.03%)    2189
        1KB - 2KB     : (100.00%) 7247438 ##############
     writeTime (14502983 total)
        1us - 2us     : ( 37.00%) 5366472 ###########
        2us - 4us     : ( 89.60%) 7628245 ###############
        4us - 8us     : ( 97.53%) 1149346 ##
        8us - 16us    : ( 99.73%)  320212
        16us - 32us   : ( 99.92%)   26762
        32us - 64us   : ( 99.97%)    8117
        64us - 128us  : ( 99.99%)    2997
        128us - 256us : (100.00%)     351
        256us - 512us : (100.00%)     324
        512us - 1ms   : (100.00%)     144
        1ms - 2ms     : (100.00%)       9
        2ms - 4ms     : (100.00%)       3
        4ms - 8ms     : (100.00%)       1
    
    Change-Id: Ia8135bb810f242575c9498b559cc1e40906279cd
    Reviewed-on: http://review.couchbase.org/13270
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  2. @alk @dustin

    declare base add_casted_stat overload as inline and unbreak build

    alk authored dustin committed
    gcc with our paranoid warning levels complains that add_casted_stat
    function (not template) is not declared before being defined.
    
    Which kind of makes sense as duplicating trivial 'proxy' function in
    all compile units is not what we need. Inline fixes it.
    
    Change-Id: I28cef250b68ef4b4c93c3ba271953b2bd5bcb83f
    Reviewed-on: http://review.couchbase.org/13337
    Tested-by: Dustin Sallings <dustin@spy.net>
    Reviewed-by: Dustin Sallings <dustin@spy.net>
Commits on Feb 16, 2012
  1. @mikewied @chiyoung

    Add more comprehensive log messages to TAP-related modules.

    mikewied authored chiyoung committed
    Change-Id: Ic02e5f4257670b9ad2b386b7d39eaf30dd6e3712
    Reviewed-on: http://review.couchbase.org/12053
    Reviewed-by: Michael Wiederhold <mike@couchbase.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
  2. @dustin @chiyoung

    refactor: Moved stat writing template generator stuff into a header f…

    dustin authored chiyoung committed
    …ile.
    
    This makes it easier to delegate stat writing down to other stat
    containers.
    
    However, this has a nasty side-effect of identical duplicate template
    functions being generated in separate compilation units and then
    making the linker angry when bringing them back together.  I'm
    generating them in different namespaces with a gross mix of templates
    and cpp.
    
    http://www.youtube.com/watch?v=ruhFmBrl4GM
    
    Change-Id: I333ca02d42c78c50e71226aa82b73c99a12142b2
    Reviewed-on: http://review.couchbase.org/13269
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
    Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Commits on Feb 15, 2012
  1. @chiyoung @trondn

    Add more comprehensive log messages to dispatcher modules.

    chiyoung authored trondn committed
    Change-Id: I05d30db4fca4ac60a17e691c9c602087090978a1
    Reviewed-on: http://review.couchbase.org/13240
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  2. @chiyoung @trondn

    Add more comprehensive log messages to checkpoint modules.

    chiyoung authored trondn committed
    Change-Id: I2560f7073eefdb2c76b065e88b1934ddfe761ccb
    Reviewed-on: http://review.couchbase.org/13239
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  3. @mikewied

    Removed observe test cases

    mikewied authored mikewied committed
    The observe functionality was remove from the code a few
    months ago. This commit removes its test cases.
    
    Change-Id: I0419b5b2ba8ee15474ff020562ffe3cba14f603f
    Reviewed-on: http://review.couchbase.org/13222
    Reviewed-by: Michael Wiederhold <mike@couchbase.com>
    Tested-by: Michael Wiederhold <mike@couchbase.com>
  4. @mikewied

    Dump core immediately on mutex error

    mikewied authored mikewied committed
    I was looking at a bug today and lost a lot of stack trace
    info because we threw an exception which was caught so high
    up the stack that I wasn't able to figure out exactly what
    caused it.
    
    Change-Id: I1d90ceeae2755b60f3dada201ec5df76d7a52176
    Reviewed-on: http://review.couchbase.org/13205
    Reviewed-by: Dustin Sallings <dustin@spy.net>
    Tested-by: Michael Wiederhold <mike@couchbase.com>
Commits on Feb 14, 2012
  1. @chiyoung

    Disable the checkpoint extension mode if the memory usage is high

    chiyoung authored
    The cluster manager enables the checkpoint extension mode for
    active vbuckets that are affected by the maintenance mode, so that
    the full backfill from those active vbuckets can be avoided if
    possible when the node in the maintenance mode rejoins the
    cluster.
    
    However, if the memory usage is still above the high water mark
    and there were few items evicted even after the item pager runs,
    the extension mode on active vbuckets should be disabled, so that
    their open checkpoints can be closed and their non-dirty items
    can be evicted.
    
    Change-Id: Id44b6a91f3f3843bb4716e9ddcc662b06c71819f
    Reviewed-on: http://review.couchbase.org/13176
    Reviewed-by: Michael Wiederhold <mike@couchbase.com>
    Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Something went wrong with that request. Please try again.