Skip to content

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also .

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also .
Commits on Nov 14, 2012
Robert Newson Bump dependencies dfddfeb
Robert Newson Support jsonp in externals.
Allow ?callback= for any external that returns json (i.e,
uses "json":{} instead of "data":"data".

BugzID: 12748
@kocolosk kocolosk Fix spurious declarations of new merge conflicts
This patch also adds extra tests of the key tree merging logic as well
as edoc-formatted documentation for the module and a few of the merge
functions.  Closes COUCHDB-902.

Thanks Paul Davis, Bob Dionne, Klaus Trainer.

git-svn-id: 13f79535-47bb-0310-9956-ffa450edef68
@kocolosk kocolosk Skip ddoc pull, use middleman for validation reset
We were pulling a list of design documents and then ignoring the result
when the #db was a partition of a clustered database.  Also, the call to
fabric:reset_validation_funs/1 can occasionally cause a stray rexi_EXIT
message to arrive in the db_updater mailbox (and subsequently kill the
server) if a worker fails.  I don't think that's desired behavior,
though it's a debatable point.  This patch spawns a middleman process to
act as a sink for those stray messages.

BugzID: 13087
Bob Dionne Expose latest=true option in calls to retrieve revisions
When a call is made to retrieve a specific revision, latest=true will
retrieve any descendent leaves instead. This enables the replicator to
better keep up with edits that occur whilst it's retrieving revisions

BugzID: 14241
@kocolosk kocolosk Add an index keyed on LRU for faster candidate ID
Our current implementation for closing an LRU DB involves a full scan
of a public ets table.  This scan blocks all other activity in
couch_server and can become a serious bottleneck when the LRU cache hit
rate drops too low.  In the worst-case all_dbs_active scenario we end up
with O(N**2) algorithmic complexity.

This patch adds a new index keyed on LRU for faster access to the least
recently used databases.  It also moves the ets table to a dict on the
couch_server heap.  The downside is an increased message rate inbound on
the couch_server, as clients are no longer allowed to update the LRU
data structures without sending a message.

BugzID: 12879

@kocolosk kocolosk Fork new OS processes outside proc_manager loop 90c656f
@davisp davisp Allow processes to idle before exiting
When system load exceeds the ability of os_process_soft_limit to keep
up with demand we enter a fork-use-kill (FUK) cycle. The constant
spawning and destruction os these processes thrashes system resources
and causes general instability.

This patch changes the behavior from killing each process as its
returned to letting it idle for a configurable amount of time (default
five minutes) which allows it to be reused by other clients. This way we
can avoid adding unnecessary load when demand for couchjs processes
exceeds os_process_soft_limit.

As a happy benefit this should also allow os_process_soft_limit to be
set much lower since the number of processes will now more closely
follow actual demand (instead of provisioning for the worst case



@kocolosk kocolosk Trigger a GC when OS process is returned e9440ae
@kocolosk kocolosk Mark proc as unused in ret_proc
Otherwise we'll never reuse it
@davisp davisp Avoid scanning the entire table of procs
For large numbers of os processes its possible that we have a slowdown
when requesting a new process. The old code matches all possible
processes out of the table to find an appropriate candidate.

We avoid the issue by using ets:select_reverse to also prefer keeping
newer processes and releasing longer lived processes. Length of life is
based on the implicit sorting of pids having newer pids sorting larger.
@davisp davisp Use a match_spec() instead of a match_pattern()
I misread the docs on what was expected for ets:select_receive/3.
@davisp davisp Two last bugs in couch_proc_manager
* Design doc languages are converted to lists
* Make sure to monitor every client correctly
@davisp davisp Remove the client ref when it dies
We have observed periods of couchjs processes spiking into the hundreds
and thousands for short periods of time since the new couch_proc_manager
was released. Today I happened to catch one in the act and poked at
couch_proc_manager's ets table. There seemed to be a few more couchjs
processes with clients than I would have expected so I skimmed the code
looking for a place where we didn't clear the client value (which would
prevent it from being reused so that it would eventually just timeout).

I found a case where if the Pid that checked out the process dies
without the OS process dying, we were forgetting to clear the client in
the ets table. This patch refactors the two places we return processes
into a single function call which clears the OS process client.
@davisp davisp Only close idle processes that have no client
Previously we didn't check if an os_process was in use by a process
before closing it. This ended up generating noproc errors in the
couch_view_updaters which would then spider out to the couch_view_group
processes causing client errors and resetting compaction.

BugzId: 13798
@kocolosk kocolosk Allow 202 as a valid status code
BugzID: 13940
@kocolosk kocolosk Don't touch the old couch_lru ets table
It doesn't exist.

BugzID: 13299
Commits on Nov 15, 2012
Robert Newson Backport new /_active_tasks API
Improved _active_tasks API

Tasks are now free to set any properties they wish (as an
Erlang proplist). Different tasks can have different properties
and the status string doesn't exist anymore - instead client
applications can build it using more granular properties from
_active_tasks. Some of these properties are:

1) "progress" (an integer percentage, for all tasks)
2) "database" (for compactions and indexer tasks)
3) "design_document" (for indexer and view compaction tasks)
4) "source" and "target" (for replications)
5) "docs_read", "docs_written", "doc_write_failures",
   "missing_revs_found", "missing_revs_checked", "source_seq",
   "checkpointed_source_seq" and "continuous" for replications

BugzID: 14269

Robert Newson Fix design_document inconsistency
Always report it as "design_document":"_design/foo"

BugzID: 14562

Robert Newson Add total_changes to view compaction status
@kocolosk kocolosk Use twig for logging
@kocolosk kocolosk Suppress duplicate access logs b2bf557
Robert Newson Fix travis build? 0c4e0ec
Robert Newson disable cover bbf0032
Bob Dionne Add twig to path for etap tests
Required to run 030-doc-from-json.t

BugzID: 14419

Robert Newson use chttpd/master during development 7af6efe