Permalink
Commits on Aug 25, 2016
  1. @jimwwalker @daverigby

    MB-20598: AddressSanitizer problem in perfsuite

    This test was being OOM killed so was left out of
    the orginal set of fixes. Tap iterator mutation/deletions
    need to be released by the client.
    
    Change-Id: I3dced32ed08d4c9fc0c31b38728576df4929c423
    Reviewed-on: http://review.couchbase.org/67058
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    jimwwalker committed with daverigby Aug 25, 2016
  2. @Chippiewill @daverigby

    Initial Phosphor Support

    Adds initial support for phosphor. Can be tested by running the
    ep_unit_tests or using a Couchbase Bucket with memcached.
    
        $ PHOSPHOR_TRACING_START="save-on-stop:ep_unit_tests.%d.%p.json" ./ep-engine_ep_unit_tests
    
    Will dump a file of the form 'ep_unit_tests.<timestamp>.<pid>.json'
    in the current directory.
    
    In this patch all of the ep-engine tasks have trace points added
    as well as a few other occasionally called functions like
    CouchKVStore compaction.
    
    Change-Id: Ic0d79620226ac332a36358e520596d87bf4652d1
    Reviewed-on: http://review.couchbase.org/66320
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Chippiewill committed with daverigby Jul 29, 2016
  3. @daverigby

    Merge remote-tracking branch 'couchbase/watson' into master

    * couchbase/watson:
      MB-20645: Don't request stats from null DCP backfill manager
    
    Change-Id: Iecc8047ac48e97ba4aabf6ef0ccdd2c6283099ac
    daverigby committed Aug 25, 2016
  4. @daverigby

    MB-20645: Don't request stats from null DCP backfill manager

    If a DCP Producer has DcpProducer::addStats called on it after its
    been disconnected (but before it's removed from the connMap) then we
    end up dereferencing a null backfillMgr pointer.
    
    Fix by adding a guard that the manager is valid before including its
    stats.
    
    Change-Id: Idc97b447090f5390054a9c40f207dae5494e63b9
    Reviewed-on: http://review.couchbase.org/67025
    Well-Formed: buildbot <build@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    daverigby committed Aug 24, 2016
  5. @jimwwalker @daverigby

    MB-20649: Fix lock inversion between Processor and front-end

    The Processor task creates a lock-inversion when it processes
    setVbucketStateMessages.
    
    The Processor holds bufMutex and calls down, eventually taking
    ConnManager::connsLock.
    
    Front end ops are generally reversed in that they will take
    ConnManager::connsLock then call into PassiveStream and then taking
    bufMutex.
    
    To highlight this issue, an existing test-case is reused with a config
    tweak that forces all DCP consumer messages to be buffered and handled
    by the Processor task.
    
    A side-effect of addressing the lock-inversion by releasing bufMutex
    before processing the DCP messages is that another thread may close
    the stream and delete the DcpResponse pointers whilst the Processor
    is working on them. To address that problem we tweak the buffer
    code so that.
    
    1) We now use a bare std::deque allowing push_front
    2) Use of std::unique_ptr to tidy up the messy delete(s)
    3) During Processor remove from the buffer any object we're using
     so that another thread cannot delete it and push_front if we need
     to retry it.
    
    Change-Id: Ifde01a570f1fc16f8f5d61328c3a44f39246785c
    Reviewed-on: http://review.couchbase.org/67019
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    jimwwalker committed with daverigby Aug 23, 2016
Commits on Aug 24, 2016
  1. @abhinavdangeti @abhinavdangeti

    Estimating cummulative flushtimes & committimes

    Calculate the cumulativeFlushTime and cumulativeCommitTime
    with the same granularity of milli-seconds.
    
    Use gethrtime() to set the flush_start time, as gethrtime()
    is used to set the end, which is used to estimate transaction
    time for an item. Also the same start and end times can be
    used to calculate the cumulativeFlushTime as well.
    
    Change-Id: Iea77e49715303afbdbbd1fd7f9d8eb8f10155843
    Reviewed-on: http://review.couchbase.org/67010
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    abhinavdangeti committed with abhinavdangeti Aug 24, 2016
  2. @jimwwalker @daverigby

    MB-20598: AddressSanitizer problems

    Two leaks and one stack overflow.
    
    The forest-kvstore code should use dynamic_cast (like couch-kvstore)
    else when the incoming callback is not a RememberingCallback, we will
    access outside of the incoming object.
    
    ep_testsuite has a leak in tap code where we must release items
    during iteration.
    
    kvstore_test has a leak in that the callback doesn't free the requested
    item.
    
    Change-Id: I6a69bd5f3e95c7a7ad291f2c6dae147a293f3bca
    Reviewed-on: http://review.couchbase.org/66915
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    jimwwalker committed with daverigby Aug 19, 2016
  3. @jimwwalker @daverigby

    MB-20598: Use scoped locking

    A few more lines of code, but clearer as to the intention
    of what the point of the lock is.
    
    Also includes an extra check we need to perform as the stream can
    become dead whilst the bufMutex is relinquished.
    
    Change-Id: Ib384de8f2dc0d64333376f3caceb40e72580056f
    Reviewed-on: http://review.couchbase.org/66984
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    jimwwalker committed with daverigby Aug 23, 2016
  4. @jimwwalker @daverigby

    Make engine_testapp a dependency of testsuite libraries

    The libraries aren't much use without the memcached/engine_testapp
    binary, so mark it out as a dependency.
    
    Change-Id: Ib0528dd16cf236f416f9dfba0e11fb51758023fd
    Reviewed-on: http://review.couchbase.org/66983
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    jimwwalker committed with daverigby Aug 23, 2016
Commits on Aug 22, 2016
  1. @jimwwalker @owendCB

    MB-20585: Fix three memory leaks

    Recent changes have snuck some leaks in.
    
    1) CouchKVStore refactoring allocated two buffers when a single
    larger buffer should be used. The second buffer never gets freed.
    Also a new/free mistmatch between couchstore/ep-engine is fixed.
    
    Introduced by - 8350175
    
    2) The watson -> master merge fixed a TSAN issue, but introduced a leak
    in DcpConsumer code.
    
    Introduced by - 7ed33aa
    
    3) Changes in connmap mean test code can now leak cookies because the
    cleanup task runs every second allowing a test to finish with an
    allocated cookie.
    
    Introduced by dd2bf1e
    
    Change-Id: I8e63a51a4a6ec0c88782a6f487ced648d36afd2e
    Reviewed-on: http://review.couchbase.org/66913
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    jimwwalker committed with owendCB Aug 19, 2016
Commits on Aug 18, 2016
  1. @abhinavdangeti @abhinavdangeti

    Adding mmap/munmap interfaces to file ops

    Context:ForestKVStore
    Pre-req: http://review.couchbase.org/#/c/66763/
    
    Change-Id: I3bd1b6396a983b105fe4f29144c69ba213c910b7
    Reviewed-on: http://review.couchbase.org/66845
    Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
    Tested-by: abhinav dangeti <abhinav@couchbase.com>
    abhinavdangeti committed with abhinavdangeti Aug 18, 2016
Commits on Aug 17, 2016
  1. @daverigby

    ep_unit_tests: Use real memory tracking code

    Use the 'real' memory tracking hooks instead of alloc_hooks_dummy in
    the ep-engine unit tests. This more accurately reflects how our code
    used in the 'real world'
    
    Change-Id: I231a179e7765d46a63c72686c0279983db21cf0b
    Reviewed-on: http://review.couchbase.org/66733
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Jim Walker <jim@couchbase.com>
    daverigby committed Aug 12, 2016
Commits on Aug 16, 2016
  1. Add KV timings stats for only instances supported by underlying store

    CouchKVStore supports both read-only and read-write instances but
    ForestKVStore supports only a read-write instance. List only timings
    for the instances supported.
    
    Change-Id: I5a433c4b8b75ca691798a7844f5f886842cdc4db
    Reviewed-on: http://review.couchbase.org/66785
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Sriram Ganesan committed Aug 16, 2016
  2. @daverigby

    evp_store_test: Compare strings using EXPECT_STREQ

    Otherwise we may fail to compare equally if the compiler doesn't
    perform string literal canonicalization.
    
    Fixes intermittent failure in highly coverage tests.
    
    Change-Id: Idb9e907b60ebb2569f223ad60ea1e8ccc735160b
    Reviewed-on: http://review.couchbase.org/66760
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    daverigby committed Aug 15, 2016
Commits on Aug 12, 2016
  1. @jimwwalker @daverigby

    MB-19948: Refactor CouchKVStore meta-data handling

    Wrap the ep-engine metadata we store into couchstore in
    a class to provide a much cleaner management interface.
    
    This includes a fix for the orginal MB-19948 issue in that
    we correctly read V1 metadata (the 18byte version written
    by 3.x). The fix is kind of hidden in the refactor, but the
    new code explictly looks at the size of metadata and has
    construction paths for 16, 18 and 19 bytes.
    
    Tests in cab19c11ea95df30a119196078f7bc8bff683692 lend
    some confidence to this refactoring.
    
    Change-Id: I7d2042af519819737a6e4d070fd8185c84778d6a
    Reviewed-on: http://review.couchbase.org/64978
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    jimwwalker committed with daverigby Jun 13, 2016
Commits on Aug 11, 2016
  1. @daverigby

    Improve connection shutdown message when bucket is deleted

    Make it more explicit that this is occuring as part of bucket delete.
    
    Change-Id: I1548a28a2a5f0db4552b6ba94928fa8d390cc938
    Reviewed-on: http://review.couchbase.org/66689
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    daverigby committed Aug 11, 2016
  2. @owendCB @daverigby

    MB-19955: Reduce NOOP interval to one second

    ns_server plan to spy on dcp traffic to determine whether a
    node is healthy.  Therefore on an idle node they need to
    ensure a DCP NOOP message is sent at a one second interval.
    
    This patch reduces the NOOP interval to one second.
    It also modifies the ConnManager task to invoke the
    manageConnections function once a second.
    
    The connection timeout is changed from being twice the NOOP
    interval to being a separate dedicated configuration option.
    
    The decision of whether to disconnect is moved outside the
    maybeSendNoop function into a dedicated maybeDisconnect
    function.
    
    Change-Id: I1b47e46fcd84a8d7f6a2f885b0120863d031b251
    Reviewed-on: http://review.couchbase.org/65927
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Jim Walker <jim@couchbase.com>
    owendCB committed with daverigby Jul 6, 2016
Commits on Aug 8, 2016
  1. @daverigby

    Merge remote-tracking branch 'couchbase/watson' into master

    * couchbase/watson:
      MB-20425: Change options parameter to correct values
      MB-20425: Remove default options parameter from get functions
    
    Change-Id: If98ef0e46d34e39c98b096098d8130e2a363bd4d
    daverigby committed Aug 8, 2016
  2. @owendCB @daverigby

    MB-20425: Change options parameter to correct values

    Updates epstore get to use the options passed in.
    Requires the call to ep_engine get from ep_engine
    arithmetic to be updated to use the following
    options:
    QUEUE_BG_FETCH | HONOR_STATES |
    TRACK_REFERENCE | HIDE_LOCKED_CAS
    
    Requires the call to ep_engine get from epstore
    store to be updated to use the following options:
    QUEUE_BG_FETCH | HONOR_STATES | TRACK_REFERENCE |
    DELETE_TEMP | HIDE_LOCKED_CAS
    
    Also adds an associated test, where the bloom filter
    is disabled which in the presense of the bug will
    cause the test to hang.
    
    Change-Id: I8fd275c3e14b0050e172b32f15fb7ed555e4b0c2
    Reviewed-on: http://review.couchbase.org/66537
    Well-Formed: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    owendCB committed with daverigby Aug 5, 2016
  3. @owendCB @daverigby

    MB-20425: Remove default options parameter from get functions

    The ep_engine get function defaults the option parameter.
    The ep_store get function also defaults the option parameter.
    
    These multiple levels of defaulting has made it difficult to
    track the value of the options parameter for different calls.
    Therefore the use of defaults are removed for these cases.
    
    This will make the change that addresses the regression of
    MB-20425 much easier to understand.  This patch makes no
    functional change.
    
    Change-Id: I69aaa31a9a437f13299eb019956aa0488f13b95a
    Reviewed-on: http://review.couchbase.org/66531
    Well-Formed: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    owendCB committed with daverigby Aug 5, 2016
  4. @daverigby

    defragmenter_test: Run for fewer iterations under Valgrind

    Currently when this test is run under Valgrind it takes ~150s to
    complete; because we create a large (500,000) number of items to
    operate on for the benchmark tests. There's little value in
    benchmarking under Valgrind, however it is still useful to run the
    tests under Valgrind to validate functional correctness.
    
    Therefore reduce the item count to 10 when run under Valgrind. This
    reduces the runtime to <10s
    
    Change-Id: Idda46dff5963c29db6890b0c8841c80b1401c010
    Reviewed-on: http://review.couchbase.org/66450
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    Reviewed-by: Jim Walker <jim@couchbase.com>
    Tested-by: buildbot <build@couchbase.com>
    daverigby committed Aug 3, 2016
Commits on Aug 5, 2016
  1. MB-20242: Disable breakpad initialization in ForestDB

    Memcached already has breakpad initialized. So, disable breakpad
    initialization inside ForestDB.
    
    Change-Id: Iefc54b2ddc2352b59c0108d16e3f70b89b2fb934
    Reviewed-on: http://review.couchbase.org/66385
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Sriram Ganesan committed Aug 1, 2016
  2. @daverigby

    unmerged-commits: Improvements for usage in scripts

    Improve interoperability with scripting use-cases:
    
    * Only use ANSI colour codes if stdout is a TTY.
    * Use the total number of unmerged patches as the error code.
    * Don't print headers if there are no unmerged patches. This also
      means that if no unmerged patches are present, the script outputs
      nothing.
    
    Change-Id: Ief509bb9ec85df7205f3ad66961df28c95d3f066
    Reviewed-on: http://review.couchbase.org/66538
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    Tested-by: Dave Rigby <daver@couchbase.com>
    daverigby committed Aug 5, 2016
  3. @daverigby

    Merge "Merge remote-tracking branch 'couchbase/watson' into 'couchbas…

    …e/master'"
    daverigby committed with Gerrit Code Review Aug 5, 2016
  4. @daverigby

    Merge "Merge remote-tracking branch 'couchbase/watson' into 'couchbas…

    …e/master'"
    daverigby committed with Gerrit Code Review Aug 5, 2016
  5. @daverigby

    unmerged-commits: Fix import ordering error

    Order of __future__ imports and normal imports got messed up during
    review fixes - 'from __future__' import must be first.
    
    Change-Id: I86be6535b49dc48c2d25f99909fcc786adfa25d8
    Reviewed-on: http://review.couchbase.org/66533
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    daverigby committed Aug 5, 2016
  6. @daverigby

    Merge "Merge remote-tracking branch 'couchbase/watson' into 'couchbas…

    …e/master'"
    daverigby committed with Gerrit Code Review Aug 5, 2016
  7. @jimwwalker

    Merge remote-tracking branch 'couchbase/watson' into 'couchbase/master'

    * couchbase/watson:
      MB-20307: Re-enable dcp ep_dcp_dead_conn_count
      MB-20312: Initialise snapshot task priority
      MB-20330: ReaderLockHolder with no lvalue
      MB-19837: Increase number of NONIO threads
      MB-18453: Make task scheduling fairer
      [BP] MB-18452: Single threaded test harness improvements
    
    Change-Id: Iea7f8d42d9c1f01492e206132e5369d0f258d819
    jimwwalker committed Aug 5, 2016
  8. @jimwwalker

    Merge remote-tracking branch 'couchbase/watson' into 'couchbase/master'

    * commit '890f7994a319a251458531b07c5a40fb90711f04':
      MB-20182: Update checkpoint snapshot correctly during TAP backfill
      MB-20105: Ensure purge_seq is not reset when no items are purged in a compaction
      MB-20054: Fix windows build error by adding size() func in class AtomicQueue
      MB-20054: Fix windows build error by including a missing header file
      MB-20054: Regression test - bucket is deleted with DCPBackfill running
      MB-20054: Account for memory alloc/dealloc in unregisterBucket
      MB-20054: [BP] Add verbose (logging) output to ep_unit_tests_main
      MB-20054: Backport ep-engine_unit_tests from watson to 3.0.x
    
    Change-Id: Ie898b27a104ae24b735d0b0901f5ba1973516820
    jimwwalker committed Aug 4, 2016
  9. @daverigby

    Merge "Merge remote-tracking branch 'couchbase/watson' into 'couchbas…

    …e/master'"
    daverigby committed with Gerrit Code Review Aug 5, 2016
Commits on Aug 4, 2016
  1. @daverigby

    Add unmerged-commits.py

    Script to show which commit(s) are not yet merged between our release
    branches.
    
    Example output (branches slightly hacked to show interesting results):
    
        Commits in 'couchbase/3.0.x' not present in 'couchbase/sherlock':
    
        Commits in 'couchbase/sherlock' not present in '5edb7b1':
        + 297e7b9 MB-20330: ReaderLockHolder with no lvalue
        + 60f5d2f MB-20312: Initialise snapshot task priority
        + 1301ca6 MB-20307: Re-enable dcp ep_dcp_dead_conn_count
    
        Commits in '5edb7b1' not present in 'couchbase/master':
        + 6e38fe8 MB-19113: Address false positive lock inversion seen with test_mb16357
        + 3bd9fa4 MB-19153: Break circular dependency while deleting bucket
        + 6f7bfa3 MB-16656: Send snapshotEnd as highSeqno for replica vb in GET_ALL_VB_SEQNOS call
        ...
    
    Change-Id: I4887ab87cc7622a6b52089ca2aaf7d10a8ce71f1
    Reviewed-on: http://review.couchbase.org/66487
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Daniel Owen <owend@couchbase.com>
    Reviewed-by: Will Gardner <will.gardner@couchbase.com>
    daverigby committed Aug 4, 2016
  2. @jimwwalker

    Merge remote-tracking branch 'couchbase/watson' into 'couchbase/master'

    * commit '7013939548201d2a833099c0bfe0cb7a5cefe041':
      MB-18453: Give all tasks their own stats and priority
      [BP] MB-18580: Wait for VB state to be persisted before starting tests
    
    Change-Id: I0df59e4a2c0b6ce49d2cbc8930fec685678e4d20
    jimwwalker committed Aug 4, 2016
  3. @jimwwalker

    Merge remote-tracking branch 'couchbase/watson' into 'couchbase/master'

    * commit '20c768ec560ecfb39313f77c17ecae4c99a4ec9b': (89 commits)
      MB-20351: Fix lock-order inversion in ~CheckpointManager
      MB-20224: [BP] Replace ThreadLocal '#define' with a using
      Revert "MB-18453: Make task scheduling fairer"
      docs/Testing.md: Document the different test types
      MB-18453: Make task scheduling fairer
      MB-20046: ep_store_test: Use the correct dbname instead of 'test'
      MB-18452: Extra refactoring and single-threaded test
      MB-19948: enable disabled meta-data tests.
      MB-19892: Ensure backfills are terminated when closing DcpProducer's streams
      MB-19982: Don't hold connsLock for duration of dcp stats
      MB-19982: Fix potential deadlock between DcpConsumer::bufMutex & connsLock
      MB-19719: Cleanup reading of vBucket stats when couchstore file doesn't exist.
      MB-19948: Handle 18 bytes of metadata
      MB-19948: CouchKVStore metadata tests
      MB-14859: Handle quick successive BG Fetch of a key interleaved with exp pager
      MB-19897: Fix the data race on lastSendTime between stats and dcp worker threads
      MB-16337: Fix for intermittent test_access_scanner failure
      MB-19843: Modify the end_seqno in DCP stream request after checking for rollback
      MB-19815: Expand 19695 regression test to cover DCP
      MB-19886: Fix data race on ActiveStream::curChkSeqno by making it atomic
      ...
    
    Change-Id: I38a4b040da6267231c01d81fa71472236c74a734
    jimwwalker committed Aug 4, 2016
Commits on Aug 3, 2016
  1. @jimwwalker

    Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/wat…

    …son'
    
    * commit '1301ca609be559248af78d6fa52ce766dd8e4915':
      MB-20307: Re-enable dcp ep_dcp_dead_conn_count
      MB-20312: Initialise snapshot task priority
      MB-20330: ReaderLockHolder with no lvalue
    
    Change-Id: I5878d95f8d792971fbb4ab5342baf4b017b6614a
    jimwwalker committed Aug 3, 2016
  2. @jimwwalker

    Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/wat…

    …son'
    
    * commit '62f88138da834e216b953d3cf8064accb521c205':
      MB-19837: Increase number of NONIO threads
      MB-18453: Make task scheduling fairer
      [BP] MB-18452: Single threaded test harness improvements
    
    Change-Id: If16ed42aed060f94d3180e832aaae0a7f5c5f052
    jimwwalker committed Aug 3, 2016