Permalink
Commits on May 25, 2017
  1. MB-24322: Clean up protocol_binary.h

    trondn committed May 15, 2017
    Put the new class definitions into its own namespace and
    provide #defines for backward source compatibility.
    
    Change-Id: I7e0f17c2ed494ca220965585a8e15637e76595b7
    Reviewed-on: http://review.couchbase.org/78116
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  2. MB-24313: Add server to client communication

    trondn committed May 12, 2017
    Start discussing the "overall on the wire framing" used
    to make the communication pipe full duplex.
    
    Change-Id: Ia9b3235218eb497af1ce84b4c4352f415b5ce272
    Reviewed-on: http://review.couchbase.org/78068
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
Commits on May 24, 2017
  1. MB-23990: Fix intermittent hangs in unit test

    premkumr committed May 24, 2017
    Add a counter for no.of notifications so that we dont wait on a missed
    notification through the corresponding condition variable
    
    Change-Id: Id9d9404892e111d7d60760d0e9ce727b474c0997
    Reviewed-on: http://review.couchbase.org/78507
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  2. MB-24098: Add CMD_DELETE to subdoc multi mutation

    OliverMD authored and trondn committed May 17, 2017
    This patch allows a CMD_DELETE to be one of the wholedoc body mutations
    in a subdoc multi mutation. The goal of this is to allow one to delete a
    document and update the system Xattrs in a single atomic operation.
    
    The approach use here is similar to one used by the remove_context which
    handles regular CMD_DELETEs. If, after the Xattr mutations are
    processed, there are system Xattrs then these are preserved, user Xattrs
    are removed and a bucket_store with a document state of Dead is used. If
    there are no system Xattrs then we perform a bucket_remove instead.
    
    Change-Id: If70c5f43925a2550f3d6354da30a1987b79d10cc
    Reviewed-on: http://review.couchbase.org/78237
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: Build Bot <build@couchbase.com>
  3. MB-24293: Modify tests to use the new cb::io::rmrf API

    sriganes authored and trondn committed May 22, 2017
    cb::io::rmrf has been modified to throw exceptions. Modify the
    callers of the API accordingly
    
    Change-Id: I16e81f3572e0b7d58af3d5ee1f7849aec8cabf97
    Reviewed-on: http://review.couchbase.org/78424
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  4. MB-22602: Add gerrit_move_to_kv_engine.py

    daverigby committed May 24, 2017
    Add a script to assist in migrating patches from the old memcached &
    ep-engine repos to kv_engine.
    
    The script will checkout the given change from Gerrit (memcached or
    ep-engine), rebase into kv-engine, and then re-upload to Gerrit as a
    new change against kv-engine.
    
    The script automatically handles any rebases onto kv_engine which
    don't otherwise conflict with upstream changes on the original repo -
    i.e. if your patch would have rebased cleanly prior to the repo merge,
    it should cleanly rebase onto the merged repo.
    
    Example usage:
    
        scripts/gerrit_move_to_kv_engine.py drigby 78323
    
    Example output:
    
        ** Migrating change 78323 - 'Move Blob to its own file (blob.{cc.h})' to kv_engine
        >> Fetching refs/changes/23/78323/3
        From http://review.couchbase.org/ep-engine
         * branch            refs/changes/23/78323/3 -> FETCH_HEAD
        -- Checking out change
        Previous HEAD position was bd68264... MB-22602: Add 'ep' as a subdir to engines/
        HEAD is now at 5b9d5a4... Move Blob to its own file (blob.{cc.h})
        -- Checking out latest kv_engine/master
        From github.com:couchbase/kv_engine
         * branch            master     -> FETCH_HEAD
        -- Rebasing change onto kv_engine/master
        << Uploading to Gerrit under kv_engine
        ** Successfully moved change to kv_engine
    
    Change-Id: Ib84265529b2d4834d361c1b2db694fb73e9ab65a
    Reviewed-on: http://review.couchbase.org/78522
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Daniel Owen <owend@couchbase.com>
  5. Add moxi_hammer, a tool to hammer moxi with traffic

    trondn committed May 22, 2017
    moxi_hammer is a command line utility to drive load to a moxi server
    to see how it works under pressure. It works by populating a buffer with
    commands it wants to send to the server, and then loops over and over
    sending that buffer to the server. It spins up a configurable number
    of connections to the server and they all send the same buffer, but
    start at a different offset (all by sending a set command).
    
    This is currently a minimum implementation, and there is a fair
    amount of @todo's in the code for future enhancements... like
    adding more threads ;-)
    
    Change-Id: I06f89651f79a1171e00ea008f86eeb44e2c7e4b3
    Reviewed-on: http://review.couchbase.org/78389
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
Commits on May 23, 2017
  1. MB-24481: Print breakpad messages as FATAL

    daverigby committed May 22, 2017
    Use the FATAL log level instead of WARNING for Breakpad messages - if
    breakpad catches a crash then we are going down...
    
    Change-Id: I2e6204af227421c8fce26d80e1bd387ebcf5b54d
    Reviewed-on: http://review.couchbase.org/78404
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  2. MB-24481: Skip backtrace printing from terminate_handler if Breakpad …

    daverigby committed May 22, 2017
    …enabled
    
    If breakpad is enabled, then in the event of a crash the backtrace is
    printed twice - once from our terminate_handler, and then a second
    time from Breakpad itself.
    
    This is potentially confusing (have we crashed twice?), so supress
    printing the backtrace from terminate_handler when Breakpad is
    enabled.
    
    Change-Id: I9fb5c310b715157d13ee2aa9ee3f2b1ab5d0d0e7
    Reviewed-on: http://review.couchbase.org/78403
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  3. MB-24481: Re-initialize terminate_handler once file logger enabled

    daverigby committed May 22, 2017
    Previously we initialised our custom terminate handler (for unhandled
    exceptions) at the very start of memcached_main, before we have the
    file logger available. This has the advantage that we can capture even
    very early crashes, but it means details of such crashes are not
    written to our log file.
    
    Historically this has lack of information in the log file hasn't made
    much material difference, as file logger is generally only setup in
    production configurations, and there we have Breakpad which reports
    the same information. However, now record details of the uncaught
    exception in terminate_handler, and hence that reports useful
    informaiton not available to breakpad.
    
    As such, add the ability to re-install the terminate handler with a
    different logger. Re-install the terminate handler immediatly after
    the file logger is setup.
    
    Change-Id: I6bc9d7b4d3d78cb829af426703e6ba2aedb9976b
    Reviewed-on: http://review.couchbase.org/78402
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  4. MB-24481: Include exception what() in terminate handler

    daverigby authored and trondn committed May 22, 2017
    When an uncaught exception is thrown we only print the exception
    message to stderr (i.e. babysitter) - which is done by the
    default_terminate_handler. It would be more helpful is this was also
    included in the memcached.log (alongside the backtrace).
    
    Change-Id: I889368a96b5242b29d474842087be9bc8cd279e2
    Reviewed-on: http://review.couchbase.org/78397
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  5. Use the correct addrinfo entry in the connect loop

    trondn committed May 22, 2017
    Change-Id: Ic65ff53ff31d4f7f5530fd416a2accd524cc1f3f
    Reviewed-on: http://review.couchbase.org/78388
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
Commits on May 22, 2017
  1. Print 'Restarting file logging' message as NOTICE

    daverigby authored and trondn committed May 22, 2017
    Reducing this from WARNING to NOTICE.
    
    Change-Id: Ib2f335577cdb9d6619a2b7ccd161882e00be7e6d
    Reviewed-on: http://review.couchbase.org/78405
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Commits on May 19, 2017
  1. MB-24460: Subdoc: release item if doc_flag:Add and EEXISTS

    daverigby authored and trondn committed May 19, 2017
    When performing a multi-mutation with the doc_flag 'Add' set, if the
    item does exist ensure it is released before returning EEXISTS to the
    client.
    
    This addresses the following memory leak:
    
        ==33026==ERROR: LeakSanitizer: detected memory leaks
    
        Direct leak of 104 byte(s) in 1 object(s) allocated from:
            #0 0x7f43bc927532 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99532)
            #1 0x7f43b4c0191f in __unique_if<Item>::__unique_single std::make_unique<>() platform/include/platform/make_unique.h:43
            #2 0x7f43b4c0191f in StoredValue::toItem(bool, unsigned short) const ep-engine/src/stored-value.cc:166
            #3 0x7f43b4c2a3e9 in VBucket::getInternal(DocKey const&, void const*, EventuallyPersistentEngine&, int, get_options_t, bool) ep-engine/src/vbucket.cc:1576
            #4 0x7f43b4be1898 in KVBucket::getInternal(DocKey const&, unsigned short, void const*, vbucket_state_t, get_options_t) ep-engine/src/kv_bucket.cc:1434
            #5 0x7f43b4b9a5e1 in KVBucket::get(DocKey const&, unsigned short, void const*, get_options_t) ep-engine/src/kv_bucket.h:152
            #6 0x7f43b4b9a5e1 in EventuallyPersistentEngine::get(void const*, void**, DocKey const&, unsigned short, get_options_t) ep-engine/src/ep_engine.h:176
            #7 0x7f43b4b9a5e1 in EvpGet ep-engine/src/ep_engine.cc:268
            #8 0x7f43b1244d84  (<unknown module>)
            #9 0x6b38b0 in bucket_get(McbpConnection*, void**, DocKey const&, unsigned short, DocStateFilter) memcached/daemon/protocol/mcbp/engine_wrapper.cc:148
            #10 0x7723a3 in subdoc_fetch memcached/daemon/subdocument.cc:482
            #11 0x7723a3 in subdoc_executor memcached/daemon/subdocument.cc:394
            #12 0x7775fc in subdoc_multi_mutation_executor(McbpConnection*, void*) memcached/daemon/subdocument.cc:1568
    
    Change-Id: Ib1958fa7f697baf60f8367dce977be2b611e2dad
    Reviewed-on: http://review.couchbase.org/78347
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Commits on May 18, 2017
  1. Fix warnings reported by clang-scan-build

    trondn committed May 18, 2017
    Change-Id: I71c329c70fa258be0e32c168d66dd054ad4d644f
    Reviewed-on: http://review.couchbase.org/78317
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: Build Bot <build@couchbase.com>
Commits on May 17, 2017
  1. MB-24378: Skip currently broken XattrTest.OperateOnDeletedItem

    daverigby authored and trondn committed May 15, 2017
    Skip test if running under ep-engine as it currently fails.
    
    Change-Id: Ic53a175be2780742b0ca169ecea1a1f59b2b36e1
    Reviewed-on: http://review.couchbase.org/78142
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  2. MB-22602: Give memcached/include its own CMake project

    daverigby committed May 16, 2017
    In preparation for merging the memcached & ep_engine repos, create a
    separate CMake project for the memcached/include subdir - the external
    API of memcached. This is necessary to avoid a circular dependancy
    between the combined (memcached+ep_engine) Project and couchstore:
    
    - couchstore depends on memcached/protocol_binary.h (for couch_dbdump)
    - ep_engine depends on couchstore.
    
    By creating a separate CMake project (KVEngineAPI) for the memcached
    headers; couchstore can depend on that project, breaking the cycle.
    
    Change-Id: Iff8db50bd4653b985f7b6427084959fff6c2367b
    Reviewed-on: http://review.couchbase.org/78189
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Oliver Downard <oliver.downard@couchbase.com>
    Reviewed-by: Jim Walker <jim@couchbase.com>
Commits on May 16, 2017
  1. MB-19606: Allow test execution against ep-engine

    mnunberg authored and trondn committed Dec 20, 2016
    This allows the tests to be run against ep-engine. To run against
    ep-engine, supply -E ep to memcached_testapp.
    
    Note: a number of tests are initally skipped as part of this patch to
    allow the tests to at least run.
    
    Change-Id: I17af453cd400511ac6754267f643cd1a58037d60
    Reviewed-on: http://review.couchbase.org/72884
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  2. MB-19606: Use GET_AND_LOCK to lock items; allow 0 length stats.

    daverigby committed May 15, 2017
    Some tests written prior to get_and_lock() being part of the engine
    API used SetItemCas to emulate locking an item. While this works for
    default engine, it fails for ep engine as it handles CAS internally
    differently.
    
    To allow these tests to work against ep engine, refactor so the
    get_and_lock() function is used instead.
    
    Additionally, remove the check that a stats value is non-zero - some
    ep engine stats can be zero length.
    
    Change-Id: I5c10c46a4d8428f7a61826ed33b9541ddd2e56f3
    Reviewed-on: http://review.couchbase.org/78141
    Reviewed-by: Jim Walker <jim@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  3. MB-16181: Allow replication streams to be forced into collections mode

    jimwwalker committed May 15, 2017
    1. Add a memcached setting switch so that memcached can be told to
    do "special" things for collections.
    
    2. Based on that switch allow memcached to retry dcp.open if it sees
    a failure, it retries by forcing DCP_OPEN_COLLECTIONS. This allows the
    bucket to decide if needs collections for a specific DCP stream.
    
    Change-Id: I2cc383410d5eb00328c2bd92224746521bdf7074
    Reviewed-on: http://review.couchbase.org/78095
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Commits on May 15, 2017
  1. MB-23524: Add UTF8 subdoc test in testapp

    OliverMD authored and trondn committed May 15, 2017
    This test checks that UTF8 characters used in a subdoc path and value
    are handled correctly from end to end.
    
    Change-Id: Iad1e1fa8411504adbfb7f8c323a774b51465ded1
    Reviewed-on: http://review.couchbase.org/78129
    Reviewed-by: Dave Rigby <daver@couchbase.com>
    Tested-by: Build Bot <build@couchbase.com>
  2. Fix autogeneration of memcached log filenames in mclogsplit

    trondn committed May 15, 2017
    when running without -p (to preserve filenames) it would
    always use filename 000000 as the counter wasn't being
    incremented.
    
    Change-Id: I05a83ebdba4835fba9729e662fe2b6f4eb8bda3e
    Reviewed-on: http://review.couchbase.org/78125
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  3. MB-16181: Missing byte swap on system event constructor

    jimwwalker authored and trondn committed May 12, 2017
    Constructor is missing byte swaps on event and bySeqno
    
    Change-Id: I19d071c0f4d5f64ecddd48b1586441182f48ec00
    Reviewed-on: http://review.couchbase.org/78094
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  4. MB-16181: DCP open with collections

    jimwwalker authored and trondn committed Apr 21, 2017
    1. Add a new flag so that clients can signal they are collection
       aware.
    2. Allow DCP open to specify a value, the value will be used to carry
       collection filtering data.
    
    Change-Id: Id33306f0d663b263840f70c44a16deff6db0a89e
    Reviewed-on: http://review.couchbase.org/77196
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
Commits on May 12, 2017
  1. testapp: report sub-second test timings

    daverigby authored and trondn committed May 12, 2017
    engine_testapp currently reports test runtimes as integer seconds, so
    tests are often reported as having 0seconds duration(!)
    
    Increase the fidelity to milliseconds (same as GoogleTest).
    
    Change-Id: Ifefa4a052e9944116cf30a6eb0fa5abbdff9a340
    Reviewed-on: http://review.couchbase.org/78078
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Jim Walker <jim@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
  2. MB-24309: Remove "Protocol::Greenstack" related code

    trondn committed May 11, 2017
    Change-Id: Iea03a6579e9fe5ffb281cf2210bd974cc4da9d77
    Reviewed-on: http://review.couchbase.org/78027
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  3. MB-24309: Remove "command" from the cookie (greenstack leftover)

    trondn committed May 11, 2017
    Change-Id: I0de5c61b704d360666c1fdb9092b965224a4575c
    Reviewed-on: http://review.couchbase.org/78026
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
Commits on May 11, 2017
  1. MB-24324: Don't use multiline error for SSL error data

    trondn committed May 11, 2017
    Change-Id: Ic7683ace35d21e11e7dbf883b8655306712dcc85
    Reviewed-on: http://review.couchbase.org/78020
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Patrick Varley <patrick@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  2. MB-24152: Allow CMD_GET in multi-lookup to access empty, deleted docs

    daverigby authored and trondn committed May 10, 2017
    When performing a subdoc multi-lookup with includes a WholeDoc spec
    (e.g. CMD_GET + an XATTR lookup), we currently fail if the document
    being accessed is deleted and has an empty body - i.e. a "traditional"
    delete.
    
    We should allow this to partially succeed (allow access to the XATTR);
    to support the use-cases which SyncGW has (they want to issue a
    CMD_GET+XATTR, and get back whatever partial information exists for
    the key).
    
    To support this, check if the input document for WholeDoc:CMD_GET is
    of zero length (indicating there is no body), and if so return
    SUBDOC_PATH_ENOENT - indicating that "path" (i.e. the doc body)
    doesn't exist.
    
    Change-Id: If35344264c17739b94617b8a47b8a170621a5c90
    Reviewed-on: http://review.couchbase.org/78013
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  3. MB-23808: Add SUBDOC_MULTI_PATH_FAILURE_DELETED status code

    daverigby authored and trondn committed May 10, 2017
    Add a new status code - SUBDOC_MULTI_PATH_FAILURE_DELETED - which is
    used to indicate that a multi-lookup operation partially
    succeeded/failed, and that the looked up document was deleted.
    
    Without this code, there is an ambiguity when a client makes a request
    with the AccessDeleted flag set - was the state of the document alive
    or dead?
    
    Change-Id: Ifbdbf4eacf4e429734595314094ca0b1c6a818d2
    Reviewed-on: http://review.couchbase.org/78012
    Reviewed-by: Daniel Owen <owend@couchbase.com>
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  4. MB-24152: Allow CMD_GET in multi-lookup to access non-JSON documents

    daverigby committed May 10, 2017
    When using CMD_GET as part of a sub-document multi-lookup request,
    allow the document to be read even if it is non-JSON.
    
    The reason for allowing this is twofold:
    
    1) Given CMD_GET returns the whole thing, there's no good reason why
       it needs to be limited to specific datatypes (JSON).
    
    2) If a multi-lookup is performed against a Deleted document which has
       an empty body (i.e. the "traditional" form of a delete), then the
       datatype is 'Raw' for the (non-existent) body. As such, to allow
       multi-lookups against empty, deleted documents we need to skip the
       JSON check for them.
       Note: due to another bug we don't currently support accesing an
       empty deleted body, making this change simplifies the work needed
       to support it (in a follow-up patch).
    
    To allow access to non-JSON (for WholeDoc operations), we defer the
    check that the input data is JSON to 'subdoc_operate_one_path' - i.e.
    the function which actually does the JSON manipulation - allowing
    WholeDoc commands to continue on any type.
    
    Change-Id: Ie5fc27265e721c24ba976ec9b73dd9ed2495a927
    Reviewed-on: http://review.couchbase.org/78011
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Build Bot <build@couchbase.com>
  5. MB-24309: Remove cas_t (only used once)

    trondn committed May 11, 2017
    Change-Id: If4c07f91b71e1bf7bf2e06ee87217da83d31fca8
    Reviewed-on: http://review.couchbase.org/78017
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  6. MB-24309: Use enum class for MutationType in tests client

    trondn committed May 11, 2017
    Change-Id: Ia68c94c326aa0810d5468b041b82b9c257c9f74e
    Reviewed-on: http://review.couchbase.org/78016
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  7. Add ability to schedule tasks to be runnable in the future

    Chippiewill authored and trondn committed May 5, 2017
    Tasks can be schedule to be made runnable in the future via the
    `makeRunnable(time)` method. The daemon periodically checks
    per-executor lists of future scheduled tasks and sets them as
    runnable when the designated time has passed.
    
    Change-Id: I6d456e23ebc7ade1b8742d2ee8e22ee7d0b30050
    Reviewed-on: http://review.couchbase.org/77735
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>
  8. MB-24309: Rip out Greenstack code [2/n]

    trondn committed May 11, 2017
    This patch rips our the remaining definitions used from
    libgreenstack and moves them to mcbp and internal to
    the test library. Further refactor should be performed
    to clean up all of the usage.
    
    Change-Id: I25a9e20194b5b74ba06f9ed04c9396cddb886225
    Reviewed-on: http://review.couchbase.org/77997
    Tested-by: Build Bot <build@couchbase.com>
    Reviewed-by: Dave Rigby <daver@couchbase.com>