Permalink
Commits on Jan 27, 2014
  1. cmake changes for couchdb

    Change-Id: If0f29acdd08ff8b148a5362f3a6fb4d6a5be901c
    nimishzynga committed with Jan 24, 2014
Commits on Jan 23, 2014
  1. MB-7629 Prevent sorting compactor delta log files

    Send a flag to the native btree updater to
    turn off sorting compactor delta log files.
    
    Change-Id: I64293a045e73dba9b80c87bfb7321c65e00791af
    Reviewed-on: http://review.couchbase.org/32313
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 10, 2014
  2. MB-9951: Trim rev-id to 16 bytes

    With adding the datatype and flexible metadata to the the
    metadata field [1], it is no longer fixed to 16 bytes, but
    is at least 2 bytes bigger, one for the flex meta code and
    one for the datatype field.
    
    Hence trim the revision ID down to a maximum of 16 bytes.
    
    [1] http://review.couchbase.org/31046
    
    Change-Id: I25898dc4e70c1d7d071c228e2bcdd2c03e06ff6a
    Reviewed-on: http://review.couchbase.org/32743
    Tested-by: abhinav dangeti <abhinav@couchbase.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
    vmx committed with vmx Jan 22, 2014
Commits on Jan 22, 2014
  1. MB-7629 Use native updater for applying compactor delta logs

    Replaced erlang btree updater with native btree updater to
    apply delta log files to the compacted btrees.
    
    Change-Id: Ib2e09a450cfe0924484574b5e0680f0d6725912c
    Reviewed-on: http://review.couchbase.org/32312
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 10, 2014
  2. Refactor native btree updater wrapper to make it reusable

    Refactored native btree updater helper code to be
    reusable by incremental updater as well as view compactor
    delta log apply methods.
    
    During a test run (on Mac/SSD) with 7M documents and 3 simple views,
    the taken taken before and after integration of native sequential sorting
    and btree updater is as follows:
    
    Before:
    Indexing time: 657.884 seconds
    Sorting + Btree updates: 603.450 seconds
    
    After:
    Indexing time: 386.529 seconds
    Sorting + Btree updates: 346.332 seconds
    
    Change-Id: I23172de5fd5a09b8a9f94f37476599f13f41a71e
    Reviewed-on: http://review.couchbase.org/32311
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 10, 2014
Commits on Jan 21, 2014
  1. MB-7629 Fix view compactor stopper to stop native process

    Compactor stopper has to send a stop message to the
    native compactor process and it has to die gracefully.
    It should be waiting for native process to die after
    sending stop message.
    
    Change-Id: I07c02bd5585538b51e05e9d59595f5b684d8cc3c
    Reviewed-on: http://review.couchbase.org/32310
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 9, 2014
  2. MB-7629 Integrate native view compactor

    Invoke native view compactor program, couch_view_group_compactor
    to perform btree compactions.
    
    During a test run (on Mac/SSD) with 7M documents and 3 simple views,
    time taken for view compaction before and after native compactor is
    as follows:
    
    Before:
    86.739063s
    
    After:
    49.133651s
    
    Change-Id: I8c7526a5f2c46423e370d0823d34d2311d0ce14b
    Reviewed-on: http://review.couchbase.org/32309
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 8, 2014
Commits on Jan 20, 2014
  1. MB-9771 Use port stream option to read stdout response

    For native incremental index updater, use stream method to
    read lines from stdout and parse view group header.
    Earlier {line, N} option was used to receive stdout response.
    It has limited control for receiving binary raw data (header data)
    since the port driver itself tries parse lines.
    
    Change-Id: I4104d494a7e56031e4e60cb7599d8382c10d0f3a
    Reviewed-on: http://review.couchbase.org/31551
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 2, 2014
Commits on Jan 17, 2014
  1. Add helper method for gracefully stopping external programs

    Change-Id: I781e295596bd908c913184a4074f34b1f31c8836
    Reviewed-on: http://review.couchbase.org/32308
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 9, 2014
  2. Remove an unwanted enclosing of a variable in list

    Change-Id: I2f705c510c49aab6bbc06ed93e683f9d0324e23d
    Reviewed-on: http://review.couchbase.org/31662
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Jan 7, 2014
  3. MB-9771 Remove sorting step for incremental mapreduce view updates

    For mapreduce views, native incremental index updater performs sorting
    of btree ops before performing the update. Hence, sorting is not required
    to be done from erlang side.
    
    Change-Id: I433731c7bd9f6e1b75a5e120bfb1c98c39f7e7c9
    Reviewed-on: http://review.couchbase.org/31550
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 31, 2013
Commits on Jan 15, 2014
  1. MB-9771 Integrate native incremental index updater

    The main updater-writer process creates batches of incremental
    changes to be applied to the view btrees and id btree. The list
    of operations for each batch are written into a sorted file (one
    for each btree in the index file). The viewgroup info along with
    the sorted file paths are passed to an external native process.
    The external process reads the operations from the sorted file and
    applies to the btrees in the index file efficiently.
    
    Change-Id: I122252a3803c29ac9351b8902e939c8c17c02825
    Reviewed-on: http://review.couchbase.org/31362
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 25, 2013
Commits on Jan 7, 2014
  1. Merge remote-tracking branch 'origin/2.1.1'

    * origin/2.1.1:
      Checkpoint progress for docs with _design prefix and too large bodies
      MB-9209: hibernate idle couch_file readers
    
    Conflicts:
    	src/couch_set_view/src/couch_set_view_updater.erl
    
    Change-Id: Ida38d0c32d6c1b12d70e8ed9af8b0438b9c40a23
    fdmanana committed Jan 7, 2014
  2. Checkpoint progress for docs with _design prefix and too large bodies

    If the last document in the sequence tree of a database file had its ID
    with the "_design/" prefix, we would completley ignore it and skip
    checkpointing. This was a problem if no other partition database files
    had any updates and this document was associated with the last sequence
    of the database - in this case each indexer run would process that sequence
    number over and over, which would block forever stale=false queries - unless
    a new document was added after to that database, or an existing one updated
    or deleted. So just treat a document with a _design/ prefix in its ID as
    any other document, as it shouldn't have any special meaning in regular
    databases.
    
    Similarly, if a document with a too large body is skipped, ensure that we
    don't end in the same loop condition as above, by ensuring its sequence
    number is checkpointed.
    
    This was diagnosed in CBSE-893.
    
    Change-Id: I7390db0329b00b137a0d9f1ff8e29971a58af143
    Reviewed-on: http://review.couchbase.org/31341
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Sarath Lakshman <sarathlakshman@slynux.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    fdmanana committed with fdmanana Dec 23, 2013
Commits on Jan 6, 2014
  1. Remove an unused variable in rollback unit test

    Remove an unused variable since it is violating the fact
    that couch_file process pid can change in between updates since
    viewgroup cleaner resets the couch file process for the viewgroup.
    
    Change-Id: I8e3f9020e769c195970ae2984ea4bcaf2d5afee6
    Reviewed-on: http://review.couchbase.org/31040
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 12, 2013
  2. MB-9730 Add support to stop cleaner native process

    stop_cleaner() function needs to ensure that the
    viewgroup cleanup native process has exited.
    Added a mechanism to send exit message to native viewgroup
    cleanup process and wait for process to exit and send exit status.
    
    Change-Id: If187e2114a0608a8f704f6b43a3c36a4817035ab
    Reviewed-on: http://review.couchbase.org/30988
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 5, 2013
  3. MB-9730 Integrate native viewgroup cleanup support

    An external native program `couch_view_group_cleanup`
    is invoked to perform viewgroup cleanup. On successful cleanup,
    viewgroup couch_file processes are refreshed to use new views
    after cleanup.
    
    A comparison of time taken for cleanup before and after this change
    benchmarked by running `test/11-updates-cleanup-many-views.t` with
    6400000 documents are as follows:
    
    Before:
    Removed 30000 values from the index in 4.468 seconds
    
    After:
    Removed 30000 values from the index in 2.364 seconds
    
    Change-Id: I1ef010a5c1dad8366e90dbbba4bf93117a92377f
    Reviewed-on: http://review.couchbase.org/30787
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 3, 2013
Commits on Jan 2, 2014
  1. MB-9730 Add support for couch_file to refresh EOF

    The couch_file process and its child writer process keeps track of
    EOF information. This EOF position is incremented when data is
    appended to the file by the writer process. The reader also
    makes use of stored EOF position for reading latest header from
    the current file process.
    
    When an external process opens the same file and appends data into
    the file, the EOF information held by the couch_file process becomes
    invalid. Hence, we need to update the EOF information held by the
    couch_file process explicitly.
    
    Added couch_file:refresh_eof(Pid) method to refresh the EOF position
    by reopening the backing write fd for the file.
    
    Change-Id: I427646e0a0707a2ccaca5af8a354552dc184dfc8
    Reviewed-on: http://review.couchbase.org/30989
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
    t3rm1n4l committed with fdmanana Dec 10, 2013
Commits on Dec 21, 2013
  1. Fix post 3.19 v8 support related wrong typedef

    Change-Id: I275fcfdf502224725337221a6caf36026dbd701a
    Reviewed-on: http://review.couchbase.org/31127
    Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
    Tested-by: Trond Norbye <trond.norbye@gmail.com>
    Reviewed-by: Matt Ingenthron <matt@couchbase.com>
    t3rm1n4l committed with ingenthr Dec 13, 2013
Commits on Dec 19, 2013
  1. MB-9307: Add typespecs to newly added functions

    Change-Id: I7d48215ef6fb760236446f5bfb4006075ba092d9
    Reviewed-on: http://review.couchbase.org/30958
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Sarath Lakshman <sarathlakshman@slynux.com>
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Dec 9, 2013
Commits on Dec 16, 2013
  1. CBD-485 fix descriptors leak in jump_to_another_version

    Change-Id: Idef3c2d31cc12483b27015e0ea7bb6848d39f512
    Reviewed-on: http://review.couchbase.org/31123
    Tested-by: Artem Stemkovski <artem@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    vzasade committed with fdmanana Dec 15, 2013
Commits on Dec 5, 2013
  1. MB-9307: Better log message

    Change-Id: Idc07ed0947b54d1224d8ed36e548d8f787aa4a3b
    Reviewed-on: http://review.couchbase.org/30867
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: buildbot <build@couchbase.com>
    vmx committed with vmx Dec 5, 2013
  2. MB-9307: Add test for indexable/undindexable switch

    Add a test for the rollback when a partition changes from
    unindexable to indexable.
    
    Change-Id: I99414a2d81c8c90da3281e59aa925e1e55856c52
    Reviewed-on: http://review.couchbase.org/30860
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Dec 4, 2013
  3. MB-9307: Rollback with multiple partitions

    Instead of rolling back taking a single partition and sequence
    number into account, it's now possible to rollback with supplying
    multiple partitions and sequence numbers at the same time. It
    is only rolled back if a header satisfies all given partition/
    sequence number pairs.
    
    Change-Id: I9492fd894a8b1fcc06d81aef93fe960322cfade6
    Reviewed-on: http://review.couchbase.org/30058
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Nov 5, 2013
  4. MB-9307: Don't include new partitions due to rollback

    When a rollback is performed, the new header might contain
    partitions that are not part of the current index. Mark those
    for cleanup and update the header accordingly.
    
    Change-Id: If5ddc95a1ef8866418d0bffcc4eff4773ac227b0
    Reviewed-on: http://review.couchbase.org/30023
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Nov 4, 2013
  5. MB-9307: Don't rollback if partition didn't exist

    Don't roll back if a partition doesn't exist in a
    previous header (or has sequence number 0).
    
    Change-Id: Idb99d557aa27339989d3e5c2b9c3a28c158e16c9
    Reviewed-on: http://review.couchbase.org/30021
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 31, 2013
  6. MB-9307: Take unindexable partitions into account

    Partitions might be set to unindexable, but that doesn't mean
    that we can't roll back to them and use them as a starting point
    for future updates.
    
    Change-Id: I401e4ea0de6f5c8f95c823cd67f780b159860413
    Reviewed-on: http://review.couchbase.org/30020
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 31, 2013
  7. MB-9307: Find header in block size increments

    To find a header, don't scan the file byte by byte, but
    only on block boundaries where the headers are stored.
    
    Change-Id: Ia6fd86bc0b707db723b86b4d76488ba0d2b78b51
    Reviewed-on: http://review.couchbase.org/30019
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 30, 2013
  8. MB-9307: Use timeout for rollback call

    Change-Id: I1f515e66cfcfb49c7b8af539c51433a4d21d8ccb
    Reviewed-on: http://review.couchbase.org/30018
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 30, 2013
  9. MB-9307: Abort compaction during rollback

    Change-Id: I810a69d7625b59b4364e0786f6360bfb3c56c156
    Reviewed-on: http://review.couchbase.org/30017
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 30, 2013
  10. MB-9307: Don't replace the full header

    After a rollback, don't use the full header information
    from the on-disk header, but only the state of the
    ID-btree and the views trees.
    
    Also update the sequence numbers of the indexable and
    unindexable partitions to match the ones of the on-disk
    header it was rolled back to.
    
    Change-Id: Iecf95d38f5111923cb6239adb0b7d824e625b5a3
    Reviewed-on: http://review.couchbase.org/30016
    Reviewed-by: Volker Mische <volker.mische@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 30, 2013
  11. MB-9307: Add indexer rollback

    Add the functionality to rollback to a certain sequence
    number of a certain partition. It rolls back to a previous
    header, which means that other partitions might also be
    rolled back.
    
    "make check" is epected to fail and is fix with the next
    commit.
    
    Change-Id: I62cae905fd3e6e733310057e36a3f66c7200d315
    Reviewed-on: http://review.couchbase.org/29732
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    vmx committed with vmx Oct 22, 2013
  12. MB-9307: Size of couch_file:read_header_bin/2 is not used

    Returning the size of the header is not needed anywhere.
    
    Change-Id: I9f7f97d0a4e00fcb35635332da3be75dd83b3a01
    Reviewed-on: http://review.couchbase.org/30859
    Tested-by: buildbot <build@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    vmx committed with vmx Dec 4, 2013
Commits on Nov 19, 2013
  1. MB-9209: hibernate idle couch_file readers

    During xdcr those processes are reading tons of binaries. And after
    burst of activity tend to go idle for long time. It makes sense to
    hibernate those folks in this state. Releasing unused
    memory. Especially given that it's state is just two args which is
    cheap to gc/hibernate.
    
    Change-Id: I6793dd7fcd29457f711a871bc12f3b7e275b56ff
    Reviewed-on: http://review.couchbase.org/30357
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Volker Mische <volker.mische@gmail.com>
    Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Aliaksey Kandratsenka committed with alk Nov 16, 2013
  2. CBD-485 jump_to_another_version to adjust the vbucket

    version in couchdb after the ep-engine compacted it
    
    Change-Id: Id1c1e375d142788e473b32e618feb5ae97963f15
    Reviewed-on: http://review.couchbase.org/30326
    Tested-by: Artem Stemkovski <artem@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    vzasade committed with fdmanana Nov 14, 2013