Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Commits on Mar 8, 2012
  1. @alk

    Schedule vbucket replications in small batches

    Srinivas Vadlamani authored alk committed
    Previously, xdcr manager would trigger couch replications for all
    active vbuckets simultaneously which was causing performance to
    suffer. The new approach, instead, schedules replications for only a
    small number of vbuckets (32 by default and configurable through an
    environment variable called MAX_CONCURRENT_REPS_PER_DOC) at any given
    time.
    
    This change set only supports xdcr continuous mode replication. We
    start by scheduling the specified number of vbuckets (in couch mode
    non-continuous) and periodicaly monitor them. As one or more of them
    are done, new vbuckets are scheduled. After we've run through all of
    them in this manner, we repeat the process.
    
    An alternate approach is likely better where we can assign a time
    slot for each vbucket instead of letting it run through completion
    before scheduling another. Implementing this approach would be
    greatly simplified if the Couch replicator API supported pause and
    resume functions. Until then, we can stick to the current approach.
    
    Also, I found it harded to break this change set into smaller pieces.
    If reviewing seems hard or if there's a better way to organize stuff,
    please let me know.
    
    Change-Id: Ifc905becb41296e68ee66dbdc73b912bc055d15a
    Reviewed-on: http://review.couchbase.org/13663
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  2. @alk

    Set timeout for set_with_meta

    Srinivas Vadlamani authored alk committed
    The gen_server call was using the default timeout of 5 secs which is
    too small and was causing unnecessary timeouts. The new value is set
    to ?TIMEOUT (30 secs).
    
    Change-Id: I614a0ae70973f35f0307af6a2ea0885d1febe386
    Reviewed-on: http://review.couchbase.org/13661
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  3. @alk

    Implementation of delete-with-meta in capi

    Srinivas Vadlamani authored alk committed
    Change-Id: I1ac7959e701a1bb94c4358383acea5166a4cd0d9
    Reviewed-on: http://review.couchbase.org/13659
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  4. @alk

    Add checkpointing support to xdcr

    Srinivas Vadlamani authored alk committed
    We leverage Couch's native replication checkpointing support which
    works as follows: At the beginning of replication, the source and
    destination are looked up to check whether a previous checkpointing
    document exists. This is done to determine the sequence number at
    which to start replication. While replication is going on,
    checkpointing is done at regular intervals by a separate process.
    Each time, a new checkpointing record (consisting of, among other
    things, the last replicated sequence number and the session id) is
    created and and appended to the history, and the document is sent for
    an update to both the source and the destination.
    
    Given this scheme of things, the following approach for xdcr
    checkpointing seems to work best i.e., one involving the least changes
    to the existing Couch replicator code (this may change in the future
    if we decide to adopt a different approach).
    
    We bypass ep-engine for opening and updating checkpoint documents. In
    order to make this safe (since there's no coordination between Couch
    and ep-engine), we store the documents in the master database instead
    the individual vbucket databases. We also prepend the document id with
    the vbucket id to avoid conflicts.
    
    Opening a checkpoint doc is no different from opening a regular doc.
    Updating a checkpoint doc requires us to wait for the in-memory items
    (as of the time of receiving the doc) to be persisted to Couch before
    updating the doc itself. This is a little conservative as the wait
    times may end up being longer for nodes with slow disks. It is,
    however, the simplest approach right now. Later, we can evaluate
    whether we need another approach.
    
    Change-Id: I8ba141cefc54e368071e1326ac0bb81e24c72d07
    Reviewed-on: http://review.couchbase.org/13772
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  5. @alk

    Use set_with_meta instead of add_with_meta for new items

    Srinivas Vadlamani authored alk committed
    Even for an add through xdcr, ep-engine creates a temp item and
    returns its CAS, which is supposed to be used while issuing the add
    later. This is similar to issuing a set_with_meta command. Hence,
    add_with_meta is redundant and can be removed.
    
    Change-Id: Ic7339c74f7c2dd09f5b5e0516809e93085015567
    Reviewed-on: http://review.couchbase.org/13658
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  6. @alk

    Make get-meta returns the CAS

    Srinivas Vadlamani authored alk committed
    Return the CAS returned by ep-engine along with enoent for a non-
    existent item.
    
    Change-Id: I85e9d6eecbec38ce31cfedf2c2fabb5a78e85fdb
    Reviewed-on: http://review.couchbase.org/13657
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  7. @alk

    ns_server changes towards fixing xdcr

    Srinivas Vadlamani authored alk committed
    Change-Id: I55cfd8ae40812f9e9cb3fb06b3b93b7b2e7f3001
    Reviewed-on: http://review.couchbase.org/12292
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Reviewed-by: Damien Katz <damien@couchbase.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  8. @alk

    Set couch_replicator connnection timeout

    Srinivas Vadlamani authored alk committed
    The default timeout was too small and was causing the xdcr source to
    retry requests too frequently. If the destination was already busy
    servicing the original request, resending requests frequently would
    make it worse.
    
    The new value is 5 mins. Not sure if this is too conservative but if
    it turns out to be so, it can be changed later or made an environment
    variable.
    
    Change-Id: I4b074ddd65f09afb8f43f936c8889620a06e0871
    Reviewed-on: http://review.couchbase.org/13660
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  9. @alk

    Move getenv_int() from to misc.erl

    Srinivas Vadlamani authored alk committed
    This function will also be used by xdcr manager. Hence moving it to a
    common place.
    
    Change-Id: Ic5726b8b56fb2d42923f186a886a93bed954ef40
    Reviewed-on: http://review.couchbase.org/13662
    Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Commits on Mar 7, 2012
  1. @daleharvey @alk

    Added 7 stats to bucket analytics to cover couch disk usage

    daleharvey authored alk committed
    Document data size, view data size, total view and docs disk
    size, view and docs fragmentation percentage and total size
    of view + docs data.
    
    Change-Id: I18708d895a8a20a27c2222a2fa3fde918cbe2851
    Reviewed-on: http://review.couchbase.org/13672
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Commits on Mar 2, 2012
  1. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      don't try to force local ddoc replication on janitor pass.MB-4825
    
    Change-Id: I2eb27ec54eacf80ae4ab0b731597ae5befb5b888
  2. @alk @aartamonau

    don't try to force local ddoc replication on janitor pass.MB-4825

    alk authored aartamonau committed
    The original idea was to ping ddoc replication srv that some vbuckets
    may have been created. This was when we replicated ddocs to each
    vbucket database.
    
    It was broken even from day 0. Because it only touches local ddoc
    replication srv. Even if this node doesn't run bucket yet/anymore.
    
    Good thing is that it's not needed anymore. Because we don't replicate
    ddocs to vbucket databases. So ddoc_replication_srv doesn't need any
    notifications from janitor.
    
    Change-Id: I759e457c984f94384eecfc673a7a0921111de99f
    Reviewed-on: http://review.couchbase.org/13752
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Commits on Mar 1, 2012
  1. @fdmanana

    Compact vbuckets by their ID in ascending order

    fdmanana authored fdmanana committed
    When updating an index, the updater opens all the vbucket
    databases, reads all changes from each, starting with the
    vbucket with lowest ID, and then closes each before reading
    the changes from the next one.
    Compacting the vbucket databases in order decreases the
    amount of time pre-compaction database files are kept open.
    
    Change-Id: Iff01a792b08bdbeb7afc9d2263d4e21db23a0ee3
    Reviewed-on: http://review.couchbase.org/13735
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Commits on Feb 27, 2012
  1. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      MB-4839 Don't ignore errors in failover safeness multicall.
    
    Change-Id: Ia6995a66bf38beddf94e0b4a7f49fd8536a87ae6
  2. @aartamonau @alk

    MB-4839 Don't ignore errors in failover safeness multicall.

    aartamonau authored alk committed
    Change-Id: I323fa902ce1f08d8a11a56e32bb5e89858346785
    Reviewed-on: http://review.couchbase.org/13637
    Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  3. @alk

    Merge remote-tracking branch 'origin/branch-18' into master

    alk authored
    * origin/branch-18:
      use wall clock time instead of now() for stats timestamps.MB-4554
      implemented ns_memcached:deregister_tap_client/2
      disable email alerts by default
    
    Conflicts:
    	include/mc_constants.hrl
    	src/mc_client_binary.erl
    
    Change-Id: I7398ef80533c2afb6e8797c057e1929e53c8551b
  4. @alk @aartamonau

    use wall clock time instead of now() for stats timestamps.MB-4554

    alk authored aartamonau committed
    Change-Id: Ie8f1806af78f33718c4ce7f356238e323cf4365d
    Reviewed-on: http://review.couchbase.org/13611
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  5. @alk @aartamonau

    implemented ns_memcached:deregister_tap_client/2

    alk authored aartamonau committed
    We'll use it soon in order to fix that "don't start rebalance 5
    minutes after failed rebalance" issue.
    
    Change-Id: Ia374b85226f5d91fe9954933af42f87df86ef57a
    Reviewed-on: http://review.couchbase.org/13610
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  6. @alk @aartamonau

    disable email alerts by default

    alk authored aartamonau committed
    Because sending email to root@localhost (our default setting) makes no
    sense at all. So lets make sure somebody configures it first.
    
    Change-Id: Ib37d004135238e8c6e4f553a9bd149703f93e1fc
    Reviewed-on: http://review.couchbase.org/13609
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  7. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      implemented REST API for max_parallel_indexers setting.MB-4831
      sync max_parallel_indexers from ns_config to couch_config
    
    Change-Id: I6e8de831a2d568e8ba5f017b0d16a2924b56245d
  8. @alk @aartamonau

    implemented REST API for max_parallel_indexers setting.MB-4831

    alk authored aartamonau committed
    GETting max parallell indexers URL (taken from /pools reply json path
    $.settings.maxParallelIndexers) will return json with field
    globalValue reflecting global setting.
    
    Current implementation will also return per-node value in
    corresponding nodes field.
    
    POSTing urlencoded globalValue parameter into same URL will change
    global value. Invalid or missing globalValue will result in 400 reply.
    
    Change-Id: I3c5c99a4c6dce6e410cdd8d94ae56db7f257d38b
    Reviewed-on: http://review.couchbase.org/13613
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  9. @alk @aartamonau

    sync max_parallel_indexers from ns_config to couch_config

    alk authored aartamonau committed
    Change-Id: Ie981881405760f1f09ba2e9d5fc4a523d9e24cb4
    Reviewed-on: http://review.couchbase.org/13612
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Commits on Feb 25, 2012
  1. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      subtract grace period when configuring autofailover.MB-4830
      fixed UI error when refreshing sample buckets settings fails
    
    Change-Id: I1d33540b135f51b756cfe6ceb6808fbf4570faa7
  2. @alk @aartamonau

    subtract grace period when configuring autofailover.MB-4830

    alk authored aartamonau committed
    Autofailover needs 2 extra hearbeat intervals during which it
    double-checks that we have only one down node. If it is we
    auto-failover it.
    
    This 10 seconds needs to be subtracted. So that UI setting of 30
    seconds means ~30 seconds to autofailover.
    
    Change-Id: If241c2cb64bd8b757419200c475c8d0ff75690cf
    Reviewed-on: http://review.couchbase.org/13603
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  3. @alk @aartamonau

    fixed UI error when refreshing sample buckets settings fails

    alk authored aartamonau committed
    $.get doesn't handle network problems IOCenter.performGet does. So we
    use it.
    
    It's still not proper error handling because entire settings section
    lacks it, but at least it's not hard error anymore.
    
    Change-Id: Ic07ccadcda8476a6ca09761c7c62bde95a7ab5c1
    Reviewed-on: http://review.couchbase.org/13599
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  4. @alk

    Merge remote-tracking branch 'origin/branch-20dp4' into master

    alk authored
    * origin/branch-20dp4:
      fixed CSS hiding rebalance_empty notice
      fixed bucket_name/node_name border overlap issue
    
    Change-Id: Ib9eabf1f0de14a866f0a67e2ea48ae1b1f9600af
  5. @BigBlueHat @alk

    fixed CSS hiding rebalance_empty notice

    BigBlueHat authored alk committed
    Also did some general CSS cleanup in this area
    per Alk's request.
    
    Change-Id: I7c10a2a891a876cd5c25335c040206e5be3d3dc0
    Reviewed-on: http://review.couchbase.org/13598
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  6. @BigBlueHat @alk

    fixed bucket_name/node_name border overlap issue

    BigBlueHat authored alk committed
    Before/After screenshots:
    http://www.evernote.com/shard/s61/sh/1e633fff-3280-4415-9511-597a17de68a4/fc2a098eb6d6c6cc045337ea67fdc444
    
    Change-Id: Iaaa2f55b7bcdbd29d28b7f407e50412e7fc92820
    Reviewed-on: http://review.couchbase.org/13597
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Commits on Feb 24, 2012
  1. Merge branch 'branch-20dp4'

    Gerrit authored
    * branch-20dp4:
      [bp] Consume exit message in ns_pubsub:unsubscribe.
      [bp] Link parent instead of monitoring in ns_pubsub.
    
    Change-Id: I6ef80a35770365803824245ad383d88728dc7381
  2. @aartamonau @alk

    [bp] Consume exit message in ns_pubsub:unsubscribe.

    aartamonau authored alk committed
    Originally reviewed on http://review.couchbase.org/13584
    
    Change-Id: Ie3b81157cc1f74cc189dfd8c6548a6efa0223a08
    Reviewed-on: http://review.couchbase.org/13593
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
    Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  3. @aartamonau @alk

    [bp] Link parent instead of monitoring in ns_pubsub.

    aartamonau authored alk committed
    This ensures that if handler dies for some reason then parent will die
    as well.
    
    Originally reviewed on http://review.couchbase.org/13583
    
    Change-Id: Ide8bdf300c3846f5c7128c2f955f658ad820bc5b
    Reviewed-on: http://review.couchbase.org/13592
    Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
    Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
  4. @aartamonau

    Merge remote-tracking branch 'couchbase/branch-18' into merge

    aartamonau authored
    * couchbase/branch-18:
      Consume exit message in ns_pubsub:unsubscribe.
      Link parent instead of monitoring in ns_pubsub.
      Start sasl explicitly.
      Remove compatible mode from mb_master.
      Always run ns_single_vbucket_mover on master node.
      Add misc:start_link functions.
    
    Change-Id: I060d70c6f496670cca79f9a9ff0633fcdba83fcc
  5. @aartamonau @alk

    Revert "Ensure that ns_single_vbucket_mover is loaded on the remote n…

    aartamonau authored alk committed
    …ode."
    
    This reverts commit 367eebb.
    
    Conflicts:
    
    	src/ns_single_vbucket_mover.erl
    
    Change-Id: Iae36401e4733c032f13e2597dd70a6109893722a
    Reviewed-on: http://review.couchbase.org/13588
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  6. @aartamonau @alk

    Revert "Load required modules on remote nodes at start of rebalance."

    aartamonau authored alk committed
    This reverts commit 40dc6ee.
    
    Conflicts:
    
    	src/ns_vbucket_mover.erl
    
    Change-Id: I453d4db7134084b682f91b6ad77ec30506fc8266
    Reviewed-on: http://review.couchbase.org/13587
    Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
    Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  7. @aartamonau

    Merge commit '4435eaca8dca81c65854087f9ad85e82049ad25d' into merge

    aartamonau authored
    * commit '4435eaca8dca81c65854087f9ad85e82049ad25d': (32 commits)
      [bp] fixed logger crash while logging start of engage_cluster
      [bp] Debug macros for rebalance-related messages.
      [bp] Add macro for debug level log messages.
      [bp] Fix log browsing.
      [bp] MB-4802 Parse versions of the 2.0.0dp4r form.
      [bp] made ns_pubsub:unsubscribe synchronous
      [bp] made ns_pubsub:subscribe_link handle normal termination of parent
      [bp] ns_pubsub:subscribe_link uses intermediate process.
      [bp] Add a possibility to specify certain log to browse.
      [bp] diag and sasl_logs for specific log file.
      [bp] Log all error messages to a separate file.
      [bp] Separate logger for ns_cluster.
      [bp] Adapt ns_log_sink to the messages without a code.
      [bp] Separate logger for rebalance-related messages.
      [bp] Separate logger for stats.
      [bp] Separate logger for ns_doctor.
      [bp] Use separate logger for menelaus messages.
      [bp] Use separate logger for user-visible messages.
      [bp] Read loggers loglevels from static config.
      [bp] Update diag_handler and ns_log_browser to new log format.
      ...
    
    Conflicts:
    	Makefile
    	etc/static_config.in
    	include/ns_common.hrl
    	rebar.config
    	src/cb_init_loggers.erl
    	src/menelaus_web.erl
    	src/misc.erl
    	src/ns_cluster.erl
    	src/ns_janitor.erl
    	src/ns_rebalancer.erl
    	src/ns_server.erl
    	src/ns_server_cluster_sup.erl
    	src/ns_single_vbucket_mover.erl
    	src/ns_vbucket_mover.erl
    	src/uuid.erl
    
    Change-Id: Iab134597ed965d7b10a67983d56d43e6cd8a5c75
Something went wrong with that request. Please try again.