Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
  • 160 commits
  • 103 files changed
  • 0 comments
  • 11 contributors
Jul 31, 2012
Filipe Manana fdmanana MB-6056 Fix distributed skip and limit
When used together, limit can not be propagated as it is without
summing the value of skip to it. We don't know in advance how
many rows are discarded from each node, so in worst case N (where
N is limit value) rows are discarded from a node, effectively
counting as a limit of 0 for that node.
Regression introduced in cf25a4b.

Change-Id: Ia129cc3463d79e2b1e5f04ca8e5ec69320734db9
Reviewed-on: http://review.couchbase.org/19034
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
167127e
Chris Anderson jchris Add a dateToArray() function to map context
This converts from a date object, or from a string like
this: "2012-07-30T23:58:22.193Z" or anything else the V8
date parser can handle.

The output is the familiar time segmented array for using in
keys, something like this:

[2011,9,27,1,46,40]

Usage of the whole thing is like this:

function(doc) {
	if (doc.date) {
		emit(dateToArray(doc.date), null)
	}
}

Change-Id: Ided5d8e0dbfb79424404c47c72b92bb1fc444d73
Reviewed-on: http://review.couchbase.org/19048
Reviewed-by: Chris Anderson <jchris@couchbase.com>
Tested-by: Chris Anderson <jchris@couchbase.com>
e27b8f3
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-100: include ddoc signature into get_group_data_size reply
Which is required for new ns_server per-ddoc stats gathering
implementation.

Change-Id: Id9679d1298f814635af35cfcecfde428e20a743b
Reviewed-on: http://review.couchbase.org/19111
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
bf5666e
Damien Katz Damienkatz More space efficient KeyDocId for view indexes
Move 16bit docid length prefix. It is unnecessary as we already know
how many remaining bytes are left after the emitted json key.

Change-Id: I4486e9c0d29435bd7f1285621058956d933dbc79
Reviewed-on: http://review.couchbase.org/19049
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
632415b
Aug 01, 2012
Filipe Manana fdmanana MB-6071 Fix _all_docs collation with ?include_docs=true
Regression introduced by CBD-32, as row tuples have variable
sizes since then.

Change-Id: Id91f5ea3d6d854acbaf54a1545e131acaf3a98da
Reviewed-on: http://review.couchbase.org/19135
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
0e7e44f
Filipe Manana fdmanana MB-100 Don't log debug message so often
The compaction daemon in ns_server calls the old CouchDB index file
cleanup function very often, causing too much useless output in the
log when no files are deleted.

Change-Id: I77b4cac2d24e776a0d2a3e472c2ea794713a856b
Reviewed-on: http://review.couchbase.org/19140
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
4cd84cf
Filipe Manana fdmanana CBD-426 Reject writes if keys/values/reductions are too large
Rather than accept the accept the writes, which will overflow the
size fields, and then find later hard to debug failures at read
time - which can be very similar to file corruption issues and
therefore difficult to distinguish.

Change-Id: Ia2f54c185ceff1b1ca7078aaabdd979bb2dea52c
Reviewed-on: http://review.couchbase.org/19014
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
37a93af
Filipe Manana fdmanana CBD-426 Update ejson collator
This is the latest version, which is thread safe (ICU collators are not
guaranteed to be thread safe nor reentrant) and has several other
improvements.

This was reviewed previously and merged, but somehow not present anymore
(maybe there was some git/gerrit issue that caused the rollback?).
Fortunately this last and better version was still intact in Apache CouchDB.

This is used for old CouchDB views (development views) and part of this
will be re-used by the new raw JSON collator (CBD-426).

Change-Id: Ibc65dfb6c71ae2af2af6ad64acf0790bc5d72e14
Reviewed-on: http://review.couchbase.org/19016
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
2a3b85f
Filipe Manana fdmanana CBD-426 Add raw json collator
This allows comparing JSON keys in their raw form, without the
need to decode them into Erlang terms.
Algorithm and original implementation by Jens Alfke (TouchDB).

Change-Id: I2b6f7222cf10c72a63b3ab08589ac52589e4df90
Reviewed-on: http://review.couchbase.org/19121
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
2933d67
Filipe Manana fdmanana CBD-426 Use raw collator for index builds
For the same dataset (2M docs, 3 views), with the raw collator
index build time was 6m9.554s and before was 14m15.181s.
Without the new file format (CBD-426) it took 5m30.656s.

Change-Id: I87f441a35a24ad29ef67b620f597721e09b61eff
Reviewed-on: http://review.couchbase.org/19158
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c490e60
Filipe Manana fdmanana CBD-426 Use less space for temporary files
4 bytes length prefix for values is not needed because they are
the rest of the binary.

Change-Id: If320ad4337950c5385b10801555c3dc33d216ab2
Reviewed-on: http://review.couchbase.org/19159
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
e387087
Aug 06, 2012
Filipe Manana fdmanana CBD-426 More efficient query path for new index format
Restore the performance/efficiency of the query path to at
least match the one we had prior to introducing an Erlang
termless index file format.

Change-Id: I168cfac36784c209117e07d2ccdbc27ef710bd12
Reviewed-on: http://review.couchbase.org/19212
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
45b099b
Filipe Manana fdmanana CBD-279 Remove unnecessary header/check
No need to make application/json content type required
for view merge requests between nodes. This was inherited
from CouchDB codebase and not needed.

Change-Id: I004d5cb891f1c60d79c834dcb2b60bb1aca7b441
Reviewed-on: http://review.couchbase.org/19251
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
30fe50c
Filipe Manana fdmanana MB-100 On shutdown, delete group's ets entry
If the ddoc is deleted, or updated, when the view group
shutdowns just delete its entry from the ets table.
Otherwise a too many ddoc deletes/updates could lead to
a memory leak.

Change-Id: I8a7e8bfd1dd3f15356f8fb951374340dff8b70f8
Reviewed-on: http://review.couchbase.org/19252
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
fae2683
Aliaksey Artamonau aartamonau CBD-181 Allow passing user status to set view compactor.
Currently ns_server has two types of compactions: scheduled and
triggered manually. A progress of certain compaction is tracked using
the status announced via couch_task_status. Sometimes it's really
useful to discriminate two aforementioned cases. But currently it's
not possible since announced status is not controlled by
ns_server. This change allows to pass arbitrary initial status to
set view compactor. So that ns_server could pass some initial status
that would allow it to tell manual and scheduled compactions apart.

Change-Id: I3b7569ba73e74b55b31ff6301d67f0b9e3519e3e
Reviewed-on: http://review.couchbase.org/19269
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
95d3cc6
Filipe Manana fdmanana CBD-423 Add couch_set_view:trigger_update/3
Change-Id: Ic732c72d013529d38d1abb93d3433e1039f6221a
Reviewed-on: http://review.couchbase.org/19295
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
809f580
Aug 07, 2012
Filipe Manana fdmanana CBD-279 More efficient updater/compactor task creation
Getting a list of design doc IDs with a specific signature is
now much cheaper, as it avoids reading all the design documents
from the master database (and opening the database), decompressing
and JSON decoding the document bodies, and finally compute their
signatures.

Change-Id: I9beaaafaf22fd87f7590742e8af5d7cc68987051
Reviewed-on: http://review.couchbase.org/19299
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
04e3dab
Filipe Manana fdmanana MB-6126 Correctly split database name into components
If the bucket name had forward slashes in it (/), then all the
code that splitted a database name into bucket name (set name)
and vbucket Id (partition Id) didn't work correctly.

Change-Id: I6b266f8d16b30a97c9ddafa2ce1bb70ae5408875
Reviewed-on: http://review.couchbase.org/19301
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
e024aaf
Aug 08, 2012
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-100: ignore mark-indexable calls on non-unindexable partitions
Because that's more convenient for stateless capi_set_view_manager
inside ns_server.

Change-Id: I90c7799a7e6dc5a7555fd76895a5dbd72531d0e7
Reviewed-on: http://review.couchbase.org/19350
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
8e8d963
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-100: don't add already added active/passive/cleanup partitions
So that 'unindexable' check is not falsely triggered.

capi_set_view_manager in ns_server is always passing all vbuckets it
has for all vbucket states. If any of those have already been added
and then marked as 'unindexed', then previous code would incorrectly
fail.

Change-Id: Ic2412c23895402915ae6f6ca42ce719ece523187
Reviewed-on: http://review.couchbase.org/19351
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
01d7e11
Filipe Manana fdmanana MB-100 Avoid unncessary list iterations
Change-Id: Ib34dccdc8c4b9e9a70459c816b46f71a18993e82
Reviewed-on: http://review.couchbase.org/19360
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
5fafeb3
Filipe Manana fdmanana MB-100 Make test 16-pending-transition.t run faster
Without compromising coverage.

Change-Id: I1194c0feff19fbacb112fd00f6826a3f8bdda606
Reviewed-on: http://review.couchbase.org/19361
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
baa5f04
Aliaksey Artamonau aartamonau CBD-423 Kill couch_set_view_update daemon.
The idea is to resurrect it in ns_server so that we could depend on
some concepts that are unknown to couchdb (like development views
etc.).

Change-Id: I6ed51434566abd92a8068cda75e8fbaf410ac50a
Reviewed-on: http://review.couchbase.org/19268
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
67e28a2
Aliaksey Artamonau aartamonau CBD-423 Don't include design document options in signature.
We're going to store per design document updateMinChanges there. And
it's not desirable to rebuild index just because that option changes.

Change-Id: I3186af9f7ae440ac482daa292934e61fe922502d
Reviewed-on: http://review.couchbase.org/19356
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
4af0f39
Filipe Manana fdmanana MB-6154 Correctly update view ets tables on ddoc deletion
Issue introduced by 04e3dab,
which attempts to be more efficient finding the IDs of all the
design documents that have a particular index signature.

Change-Id: I2d7841d95bdc2074c4ec811722812afd1033bd83
Reviewed-on: http://review.couchbase.org/19367
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
7fed5be
Aug 09, 2012
Filipe Manana fdmanana MB-6158 Trigger another update if there are listeners
Update listeners might have been added after the updater
started and already processed the requested partitions.

Change-Id: Ibdccba7fecd80f5fcbc14e31b84772513b8dda12
Reviewed-on: http://review.couchbase.org/19383
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
e69676e
Filipe Manana fdmanana MB-6165 Ignore index pausing of already paused partitions
If the same partition is requested twice to be unindexable, at
the second request the view group process crashes, as it expected
the requested partition to be currently indexable.

Change-Id: Iabcd3e6f9ca392169ab6fd9f7892ed0a2a566594
Reviewed-on: http://review.couchbase.org/19403
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
fc3a87e
Aug 10, 2012
Filipe Manana fdmanana CBD-426 Index headers are stored in a termless format
The index headers are no longer written as serialized Erlang
terms. Instead they are now written in a format easy to parse
and for any environment/language.

Change-Id: Ief5cd900d2d1f87fc6db1e42ce3be1c0b02648b7
Reviewed-on: http://review.couchbase.org/19332
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
8aae876
Filipe Manana fdmanana MB-6096 Fix too high memory consumption during indexing
Several things where causing the indexer to consume too much
memory:

1) The use of binary mode btrees, introduced by CBD-426, was
   creating #doc_info records with IDs and Revs as sub-binary
   references, which were preventing the binaries read from disk
   to be released soon, as these records in turn were passed
   to several view processes (work queues, etc). The solution
   here is to copy those sub-binary using binary:copy/1.

2) Reduce the size of the map queue. Doing it helps reducing the
   memory consumption and doesn't seem to affect the index build
   time;

3) Use enif_alloc() and enif_free() in the mapreduce NIF. Using
   C++'s new operator or C's malloc/calloc/etc is perfectly fine
   but less efficient. Using enif_alloc() overall memory consumptiom
   is significantly lower;

4) Some minor optimizations regarding handling and matching of
   binaries, as explained by:
   http://www.erlang.org/doc/man/binary.html

Change-Id: I49cbfbd4c4be06cc6de1451c18aabcdb5b100581
Reviewed-on: http://review.couchbase.org/19430
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c8003e5
Filipe Manana fdmanana CBD-503 Eliminate temporary memory allocations
Have the map and reduce handlers in mapreduce.cc directly
allocate the Erlang binaries with the JSON key/value pairs
and reductions that are returned to the Erlang code.

Change-Id: I3eee936e2e7026b5ae39e245ad38292257fa02df
Reviewed-on: http://review.couchbase.org/19456
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
079a4bd
Filipe Manana fdmanana CBD-504 Use enif_alloc()/enif_free() in couch_view_parser
These are more efficient, since are Erlang VM specific, in
opposition to C++'s new and delete operators and C's malloc()
and free() allocators.

Change-Id: I4470fa75ac5f1d37587c13c3e7b51739f066fe33
Reviewed-on: http://review.couchbase.org/19457
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
75072cd
Filipe Manana fdmanana CBD-504 More extensive couch_view_parser tests
The tests now do multiple iterations of the same test cases.
The goal here is to make it much more likely to detect errors
such as segmentation fauls in the C++ NIF.

Change-Id: Ic01618f59cd6f4e7f1cfaa6064d46c047798ef0a
Reviewed-on: http://review.couchbase.org/19464
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
919f9e5
Aug 13, 2012
Filipe Manana fdmanana MB-6096 Use enif_alloc()/enif_free() in ejson decoder
Change-Id: I783148245b59b2134322f8a3c69fa251a86a475b
Reviewed-on: http://review.couchbase.org/19553
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
2c4b88c
Volker Mische vmx MB-6204: Fix build issue with GCC-4.6
The error checking in GCC-4.6 got stricter, which lead
to the error:
error: 'ptrdiff_t' does not name a type

To fix this issue, just include <cstddef>.

Change-Id: Iffd67fd5db1a4265007ea6b87f2e6e8a4724c107
Reviewed-on: http://review.couchbase.org/19554
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
24139f1
Aug 14, 2012
Junyi Xie MB-100: skip dumping all docs replicated to log files
Today the logger dumped every replicated doc to logfiles, making them
significantly bloated and XDCR unnecessarily slow. We can dump docs
for debugging purpose but not in production system.

Change-Id: Idfb57bd2e397ea05f687c77d251e1c11cf366638
Reviewed-on: http://review.couchbase.org/19583
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
66812f1
Filipe Manana fdmanana MB-6218 More explicit error on database open failure
Change-Id: I8f68f75841f37218fd6ab2761ff2b64e52da2409
Reviewed-on: http://review.couchbase.org/19598
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
eae7370
Bin Cui bcui6611 MB-6198: Build couch_ejson_compare on windows
Change-Id: Ic432f982e25f836d8647594dcb0f6c79ffac55df
Reviewed-on: http://review.couchbase.org/19609
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
dfb997d
Aug 15, 2012
Filipe Manana fdmanana MB-6208 Don't crash in before_database_delete call
A crash in the synchronous before_database_delete would lead to
the terminate callback be invoked which in turn can lead to
deadlocks, as it it synchronously shutdowns view groups.
Crash caused by the fact that when a view group dies, the ets
entries for its aliases were not deleted.

Change-Id: I7f04ff324711f17e948058bb54d2f727c29de0b6
Reviewed-on: http://review.couchbase.org/19658
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
ac91861
Damien Katz Damienkatz MB-5941: Fix to replicate doc data with full fidelity
Change-Id: Ibcd85c4b9d896b6cf03f7a2a988ba521779d3d68
Reviewed-on: http://review.couchbase.org/19608
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
3c95238
Aug 16, 2012
Aaron Miller MB-6236 Don't destroy DB handle on compaction fail
This fixes an issue in which any error from compaction would replace a
db record with a new uninitialized one, causing all further operations
to that db to fail.

Change-Id: Icc4c69949ad6a1c2821f0f88dc8f16b3df188163
Reviewed-on: http://review.couchbase.org/19672
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
96ddca4
Filipe Manana fdmanana CBD-511 Get all ddoc aliases on view group restart
If a view group restarts, grab the id of all design documents
with the same signature (aliases). We were only populating the
ets tables when ddocs are created (or deleted), so in the case
of a view group restart (or server restart) we would no entries
for the aliases in the ets table.

Change-Id: Ibfa913ba49f9585c205e46cbc7f69032bb011e85
Reviewed-on: http://review.couchbase.org/19700
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1a64800
Aug 17, 2012
Filipe Manana fdmanana CBD-422 Write header when initial index build finishes
Also no need to truncate the file when the updater is
stopped while doing initial build.

Change-Id: I2069c55801583a3e87383091546265fe9acee3a5
Reviewed-on: http://review.couchbase.org/19775
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
d30ec5b
Filipe Manana fdmanana MB-100 Minor updates to get_group_info (used for debugging)
Change-Id: Idac7b1d472597fb52c623aa2f46be6a25ef732d1
Reviewed-on: http://review.couchbase.org/19776
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
ef19472
Aug 18, 2012
Filipe Manana fdmanana MB-6281 Simplify ddoc update notification
A doc record, representing the design document, is now part
of the ddoc update notification message. This avoids the event
listeners from opening the database and read the document.
In certain cases this could lead to a deadlock, such as in MB-6281:

1) ddoc update listener trying to open a database (calls couch_server);

2) couch_server tries to call the listener with the sync event
   before_database_delete

Change-Id: I7b3d47410f78a8078d1822a5d92307950e33a884
Reviewed-on: http://review.couchbase.org/19822
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
b5834a5
Filipe Manana fdmanana MB-100 Add some more logging on ddoc updates
Change-Id: I0af30748c6c6e4856757563884e43d1ec287c738
Reviewed-on: http://review.couchbase.org/19830
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
395c3e1
Aug 20, 2012
Filipe Manana fdmanana MB-6311 Don't delete tmp index files for older still active view groups
Change-Id: I70afa1b788b8ac17fb42cb5d191c5cbc1a1c2590
Reviewed-on: http://review.couchbase.org/19845
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1dbf3c2
Filipe Manana fdmanana CBD-422 Don't delete tmp files when stopping updater
It can take a significant time, blocking the index partition
state changes caller. The tmp files will soon be delete when
the updater is restarted, in a different process.

Change-Id: I4033889fd13f40751defa2831d5abfd31e1699b9
Reviewed-on: http://review.couchbase.org/19867
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
33c20fe
Filipe Manana fdmanana MB-6295 More explicit error message when keys are too long
Instead of propagating exception from btree module (storage
engine), assemble a more friendly message containing the respective
document ID and the first 100 characters of the key.

Change-Id: Ia6e409f6991eefc9c85dc9681d2dd7dde072c996
Reviewed-on: http://review.couchbase.org/19868
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
b2686ae
Aug 22, 2012
Filipe Manana fdmanana CBD-512 Allow replica group updates to be triggered by ns_server
Through couch_set_view:trigger_replica_update/3.

Change-Id: Ia2a00a1b0e675d3213c06103f2c644270bef06d2
Reviewed-on: http://review.couchbase.org/19701
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
9b4edfd
Aliaksey Artamonau aartamonau MB-100 Fix couch_set_view:start_compact type spec.
Change-Id: I90025970f2ada7517e611af6769a8aa28cd78852
Reviewed-on: http://review.couchbase.org/20019
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
65b31d6
Filipe Manana fdmanana MB-6361 Update task status on view compaction retry
Sometimes when server is under very heavry load, the view
compaction retries can take a significant time.

Change-Id: I6c1463ddc9ac8b6dfde6eff04a59e52a1d24f0d5
Reviewed-on: http://review.couchbase.org/20020
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
3deaad7
Filipe Manana fdmanana MB-6361 Bump gen_server call timeout to infinity
Change-Id: I6933081532888c2b8cf31829c4d424eb77750600
Reviewed-on: http://review.couchbase.org/20021
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
310df4f
Filipe Manana fdmanana CBD-536 Add blocked indexers info to _active_tasks
It's now possible, externally via _active_tasks, to know
how many and which indexers are blocked by the barriers.

Change-Id: Iecc462dd36d097ce84c1f62ec62b05fe68ae410b
Reviewed-on: http://review.couchbase.org/20023
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
4bff19a
Aug 23, 2012
Filipe Manana fdmanana MB-6399 Don't crash when ddoc doesn't map to a view group pid
This was introduced by CBD-279 (04e3dab)
and CBD-511 (1a64800).

Change-Id: I03ad70399ad7705a21664ba31bd1b00ccb7c8c68
Reviewed-on: http://review.couchbase.org/20075
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
9c06815
Filipe Manana fdmanana MB-6380 Add more tests and logging for compaction retries
Failure in MB-6380 might be related to MB-6371, which was
encountered right before.

Change-Id: Iccd217b44a5b20c4448d0e748298470600d87359
Reviewed-on: http://review.couchbase.org/20076
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
9b7e70a
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-100: got rid of max_dbs_open
Because we don't need it at all.

Change-Id: I3f384d284e619476a58df41e3ead31c43026bad0
Reviewed-on: http://review.couchbase.org/19737
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
4930e3e
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6049: implemented all_known_databases
So that we can quickly and racelessly see what databases are there.

Change-Id: I6eb79943c1a62714265045029f60ef9f16150a48
Reviewed-on: http://review.couchbase.org/19754
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
9f75374
Aug 24, 2012
Filipe Manana fdmanana CBD-75 Minor tweaks to replica index queries
Don't send the request to the main group with stale=false, after
all we want to query the replica group, not the main one.
Make error term more consistent with others errors, by being a
tuple with the atom 'error' as first element.

Change-Id: I16ba0989be8509a544bcf547add02eb886b7affd
Reviewed-on: http://review.couchbase.org/20119
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1709981
Filipe Manana fdmanana MB-6399 Add missing unit test
Change-Id: Ib90c1a273b8b6fd6fa75f4c19469fdc252fefe80
Reviewed-on: http://review.couchbase.org/20120
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
075f695
Filipe Manana fdmanana CBD-416 Add update frequency to view compaction tasks
So that we ensure minimal overhead.

Change-Id: I50a29ee45855793b3d632f6ded9dd76d5b846459
Reviewed-on: http://review.couchbase.org/20150
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
e08354a
Aug 25, 2012
Filipe Manana fdmanana MB-6415 Delete index files on view group terminate
A quick succession of ddoc updates, updating the ddoc back
to a previous version (while we're here in terminate), may lead
us to a case where it should start clean but it doesn't, because
couch_set_view:cleanup_index_files/1 was not invoked right before
the last reverting ddoc update, or it was invoked but did not finish
before the view group got spawned again. Problem here is during
rebalance, where one of the databases might have been deleted after
the last ddoc update and while or after we are in terminate callback,
so we must ensure we don't leave old index files around.

Change-Id: Ieb77ed59ac4de4a62882c711604bf360d6f46dfd
Reviewed-on: http://review.couchbase.org/20164
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
ddfada3
Aug 27, 2012
Filipe Manana fdmanana MB-6415 Delete index files on terminate after ddoc updated
Change-Id: If29c35736befbfb098601585241e0c879b046417
Reviewed-on: http://review.couchbase.org/20181
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
badcdf3
Filipe Manana fdmanana CBD-545 Don't restart compactor on index state changes
Each state change of an indexer (partition states, pausing/unpausing
of indexing for specific vbuckets, etc), stop and restarts the
compactor (if currently running). Successive state changes, as it's
common during rebalance, therefore cause lots of unncessary work.

Instead of stopping and restarting the compactor, just stop it, so
incoming state changes are more lightweight.

The compaction daemon in ns_server will trigger the compaction again on
its next cycle. Also, with the restart we currently do, due ns_server's
compaction daemon unawareness of the restart, we can get 2 index compactions
running in parallel, one for a different index triggered by the compaction
daemon and other caused by the restart.

Change-Id: Ie9c9caf44bede80b99e4a0bf6b49e86482386948
Reviewed-on: http://review.couchbase.org/20188
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
5262bca
Filipe Manana fdmanana CBD-546 Use ordsets of partition IDs instead of lists
To avoid creating temporary ordsets on each state transition.

Change-Id: I0f6627b6e0cf8388d0eb7e36b4002402b3aebfab
Reviewed-on: http://review.couchbase.org/20189
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
2deb6fd
Aug 28, 2012
Filipe Manana fdmanana MB-100 Fix race condition in test 02-old-index-cleanup.t
Change-Id: I769a3ee0a8bddc0de873b6a282dd85fc58d7a7a8
Reviewed-on: http://review.couchbase.org/20245
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
ca8eb92
Filipe Manana fdmanana CBD-546 Assert lists are ordsets
Dialyzer is unable to distinguish a list from an ordset.
Also at runtime, ordsets operations against lists that are
not ordsets don't raise an error, just return unpredictable
output.

Just assert via ordsets:is_set/1 that the input lists are
ordsets, otherwise hard to find bugs might happen in case
someone passes lists that are not ordsets.

Change-Id: Iba7267a9da2b33550134683bac5dd1d908af317b
Reviewed-on: http://review.couchbase.org/20244
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1f074c7
Filipe Manana fdmanana MB-100 Remove unnecessary list conversion
couch_set_view_util:decode_bitmask/1 already returns an
ordset.

Change-Id: Id363b8ad3856d52f8742819bdd2efb813042dd3c
Reviewed-on: http://review.couchbase.org/20246
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
5e81176
Filipe Manana fdmanana MB-100 Delete compaction file if index compactor dies
So disk space is freed and useful for other operations until
the same index is compacted again.

Change-Id: Icc4b33caf7204970c7c27555f6b43e9f3ee39aa7
Reviewed-on: http://review.couchbase.org/20257
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
6a91f5b
Filipe Manana fdmanana CBD-475 Add test 21-updater-cleanup.t
Verify that the updater is able to do cleanup as well.
Replaces the no longer working Python test:

test/python/set_view/update_cleanup.py

(not working anymore after REST API cleanup)

Change-Id: I48eed5f5bc2da69880cff3919488b9ed442b3452
Reviewed-on: http://review.couchbase.org/20258
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
49622a2
Filipe Manana fdmanana CBD-475 Add test 22-compactor-cleanup.t
Verify that the compactor is able to do cleanup as well.
Replaces the no longer working Python test:

test/python/set_view/compaction.py

(not working anymore after REST API cleanup)

Change-Id: I59d45230f05044a7fd90168bfe6ddb772af3d3d5
Reviewed-on: http://review.couchbase.org/20261
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c5d4867
Aug 31, 2012
Filipe Manana fdmanana MB-6492 Fix race condition in test 200-view-group-no-db-leaks.t
Only check db ref counters after the compactor process died,
otherwise they might not yet have been decremented.

Change-Id: I6dec4e89ecfa1bd4f67a0f09c4345cc1136dcb49
Reviewed-on: http://review.couchbase.org/20359
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
cc29f25
Filipe Manana fdmanana MB-6489 Add couch_btree:stats/1 for debugging
Traverses the entire tree to collect several useful stats
for developers, meant for debugging only. Example of stats
computed:

    btree_size                     117741553
    file_size                      208359643
    fragmentation                  43.49119085407532
    chunk_threshold                5120
    kp_nodes                       3155
    kv_nodes                       188596
    max_depth                      7
    min_depth                      3
    max_reduction_size             8
    min_reduction_size             8
    max_elements_per_kp_node       172
    min_elements_per_kp_node       1
    avg_elements_per_kp_node       60.776545166402535
    max_elements_per_kv_node       102
    min_elements_per_kv_node       38
    avg_elements_per_kv_node       58.32573331353793
    kv_count                       11000000
    min_kp_node_size               44
    min_kp_node_size_compressed    46
    max_kp_node_size               7397
    max_kp_node_size_compressed    2755
    avg_kp_node_size               2614.391442155309
    avg_kp_node_size_compressed    971.2161648177496
    min_kv_node_size               1045
    min_kv_node_size_compressed    389
    max_kv_node_size               2805
    max_kv_node_size_compressed    1011
    avg_kv_node_size               1604.9576661222932
    avg_kv_node_size_compressed    599.7721160576046

Change-Id: Ic10067f339f88c04738e65af4ef480b79a02a23c
Reviewed-on: http://review.couchbase.org/20350
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
f465366
Filipe Manana fdmanana MB-6489 Add debug URI to get view btree stats
Example output:

$ curl -s 'http://localhost:9500/_set_view/default/_design/test3/_btree_stats' | python -mjson.tool
{
    "id_btree": {
        "avg_elements_per_kp_node": 19.93181818181818,
        "avg_elements_per_kv_node": 75.00750075007501,
        "avg_kp_node_size": 3170.159090909091,
        "avg_kp_node_size_compressed": 454.0511363636364,
        "avg_kv_node_size": 2101.2100210021,
        "avg_kv_node_size_compressed": 884.929492949295,
        "btree_size": 3058201,
        "chunk_threshold": 5120,
        "file_size": 11866307,
        "fragmentation": 74.22786213098988,
        "kp_nodes": 176,
        "kv_count": 250000,
        "kv_nodes": 3333,
        "max_depth": 4,
        "max_elements_per_kp_node": 27,
        "max_elements_per_kv_node": 100,
        "max_kp_node_size": 4294,
        "max_kp_node_size_compressed": 619,
        "max_kv_node_size": 2801,
        "max_kv_node_size_compressed": 1161,
        "max_reduction_size": 133,
        "min_depth": 4,
        "min_elements_per_kp_node": 8,
        "min_elements_per_kv_node": 75,
        "min_kp_node_size": 1273,
        "min_kp_node_size_compressed": 206,
        "min_kv_node_size": 2101,
        "min_kv_node_size_compressed": 849,
        "min_reduction_size": 133
    },
    "view1": {
        "avg_elements_per_kp_node": 17.96416938110749,
        "avg_elements_per_kv_node": 23.99923202457521,
        "avg_kp_node_size": 3127.825732899023,
        "avg_kp_node_size_compressed": 498.3436482084691,
        "avg_kv_node_size": 3024.903235096477,
        "avg_kv_node_size_compressed": 805.7447441681866,
        "btree_size": 8789820,
        "chunk_threshold": 5120,
        "file_size": 11866307,
        "fragmentation": 25.92623804524862,
        "kp_nodes": 614,
        "kv_count": 250000,
        "kv_nodes": 10417,
        "max_depth": 5,
        "max_elements_per_kp_node": 21,
        "max_elements_per_kv_node": 24,
        "max_kp_node_size": 3676,
        "max_kp_node_size_compressed": 606,
        "max_kv_node_size": 3025,
        "max_kv_node_size_compressed": 852,
        "max_reduction_size": 141,
        "min_depth": 5,
        "min_elements_per_kp_node": 2,
        "min_elements_per_kv_node": 16,
        "min_kp_node_size": 357,
        "min_kp_node_size_compressed": 108,
        "min_kv_node_size": 2017,
        "min_kv_node_size_compressed": 577,
        "min_reduction_size": 137
    }
}
$

Change-Id: I0edf2da5dbd946ebaa13d4d7e31b5de13b608491
Reviewed-on: http://review.couchbase.org/20351
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
da9c98d
Sep 01, 2012
Filipe Manana fdmanana MB-6495 Fix view merging error on Windows
On Windows the old http streamer codepath is used,
therefore it must pass keys as raw json binaries,
otherwise the collator raises an error.

Change-Id: I47fe77ee00f012eac6d320ae8e0a9e789f499c9a
Reviewed-on: http://review.couchbase.org/20407
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
162d1eb
Filipe Manana fdmanana MB-6502 Ignore enoent errors on index files cleanup
Some index files might just have been deleted by the group
processes themselves when they terminate due to a ddoc update.
Therefore just ignore the posix error enoent (file not found).

Change-Id: I2826c24b1ea757c8f9718173b91aba0bd133fcff
Reviewed-on: http://review.couchbase.org/20408
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
00d291d
Sep 05, 2012
Filipe Manana fdmanana MB-6517 Delete index files only if there are no aliases
Regression introduced with MB-6415.

Change-Id: Ie6b5d0c61234d67d17a2767867b871ff765a7e82
Reviewed-on: http://review.couchbase.org/20481
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
884ba59
Filipe Manana fdmanana MB-6480 Ensure workers die asap when updater dies
So that when the updater is stopped or exits due to an
unexpected error, the workesr terminate immediately, not
spending more time doing useless IO on temporary files
that will be deleted right after the updater is started
again.

Change-Id: Ibfa56095c8cbb93152a8f313069929d994652000
Reviewed-on: http://review.couchbase.org/20459
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
88f0539
Filipe Manana fdmanana MB-6480 Optimize KV batch sizes for initial index build
Instead of assuming each dequeue operation will give lists
of KV pairs for each btree of approximate sizes, ensure
that before sorting and flushing to disk each list, each has
an optimal size.

This impacted performance of building indexes where different
views produced different numbers of KV pairs or with very
different sizes per KV pair (id btree also counts).

For a design document with 1 view, emitting 1 kv pair per
document, and a dataset of 25M documents, the indexing times
were:

    75m42.984s before this change
    59m11.640s after this change

Change-Id: I039a0681ea81025a97ebd3c7b2512d6976b0cdb8
Reviewed-on: http://review.couchbase.org/20460
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
cf6cd7a
Filipe Manana fdmanana MB-6480 Prioritize merging of files with similar sizes
To avoid reading the remains of large files just to write them
to other temporary files when they get merged with significantly
smaller files, just attempt at each iteration to merge files
with similar sizes/number of kv pairs.

For a design document with 1 view, emitting 1 kv pair per
document, and a dataset of 25M documents, the indexing times
were:

    59m11.640s before this change
    44m47.411s after this change

Change-Id: I4df32a1d2fe22d448055d200f2b1ca3fb6f81aaf
Reviewed-on: http://review.couchbase.org/20461
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
cc43407
Filipe Manana fdmanana MB-6480 More parallelism for initial index build
Instead of having the main writer process wait for each
merge worker to finish before spawning a new one, let the
new worker wait for the previous to finish before starting
its merging task. While the new worker is waiting for the
previous to finish, the main writer process keeps preparing
the next batch and sorting it.
This also allows to reduce the size of the writer queue
without affecting performance, and significantly reducing
memory usage while the inital index is being built.

Test with a 20M documents dataset, design document with
4 views, each emitting a KV pair for each document:

Before this change:  82m19.728s
After this change:   68m23.159s

Average memory consumptiom reduced by 50 to 100Mb.

Change-Id: I73ad255c753d82ecbc19cbc06f19fd22937e15fa
Reviewed-on: http://review.couchbase.org/20536
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c82ccb3
Damien Katz Damienkatz MB-6413 More reliable server startup when testing
When running test suites in parallel, ensure couch_server from
each test suite can never start with a common database/index prefix.
Because otherwise, when attempting to open databases on startup,
it can delete compaction files created by other test suites.

Change-Id: I5f0c7d1ad884c1b629ca23184aa5ebfb6d0a4a54
Reviewed-on: http://review.couchbase.org/20382
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
9419f75
Filipe Manana fdmanana MB-6413: Fix race condition in test 12-errors.t
Because couch_server now opens all databases on startup,
often it opened the database before the test deleted its
file.

Change-Id: Ifa84823992cbd9d873649f0e73167330986c3d58
Reviewed-on: http://review.couchbase.org/20402
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
39ae383
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6413: open all databases on startup
ns_server needs to know all present vbuckets. In order to do it
quickly and correctly we want to use ets table of all couch dbs. But
if we don't populate it early, there are problems, thus we do.

Change-Id: I604a8168d556581db964cd050e3fafdf974c911d
Reviewed-on: http://review.couchbase.org/20222
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
7db41d6
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6413: re-implemented all_known_databases/0
First, we've made couch_dbs_by_name ets table protected to enable read
access without having to call into couch_server. Second, we've made it
ordered_set. Third, we've changed API to list not all known databases,
but all known databases with given prefix. Because this is exactly
what's needed by ns_server and we can do it efficiently via
ordered_set ets table.

Change-Id: I2c1462e9d49868b7e4f225c5a22196fa0d833770
Reviewed-on: http://review.couchbase.org/20223
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
c4ba56a
Sep 06, 2012
Filipe Manana fdmanana MB-6532 View compactor continues if updater finishes too soon
When the compactor starts, if the index updater finishes too soon,
it exited on purpose with reason {updater_died, noproc}, and eventual
retries would succeed, as this was not expected to happen very often.
However, in some test environments it just started to happen too
often.

Change-Id: Iecebcc895c7fd200c93686652c131247e59798b9
Reviewed-on: http://review.couchbase.org/20582
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
8ccc607
Filipe Manana fdmanana MB-6537 Don't ignore unindexable partitions
When building list of all monitored partitions, don't
ignore the ones in the unindexable meta state.

Change-Id: Ie2b13b5b6d5597c9d5a30a10fdf34c93b10941a7
Reviewed-on: http://review.couchbase.org/20583
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
9424184
Filipe Manana fdmanana MB-6541 Notify update listeners when view compaction finishes
Due to view compaction retries, it's possible the compacted
group has caught up with the updater's group and it's actually
more recent than the view group's current group snapshot.
In this case notify the update listeners, as a new index update
might not be triggered until new changes are applied to the
vbucket databases and a subsequent query with stale=false or
stale=update_after arrives.

Change-Id: I455f699bb5ca5c31ab30c365df960e115ab0d448
Reviewed-on: http://review.couchbase.org/20587
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
4ede1a2
Sep 10, 2012
Damien Katz Damienkatz MB-5611: Reduce replication http retries from 10 to 2
Because XDCR now has better high level error handling and retry logic
via supervisors, reduce the http retries to 2, so that during
rebalance or other transient conditions that can cause errors, other
replicators may continue.

Change-Id: Icbda799cc87be7cbe155c6d344666700d4c9d391
Reviewed-on: http://review.couchbase.org/20665
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
a672a1d
Filipe Manana fdmanana MB-6556 Ignore effectless index partition state transitions
For state transition requests that do not change any partition
states at all, just ignore them, saving some processing time,
avoiding restarting the updater, cleaner, stopping the compactor,
commiting a new header, etc. Also avoids unncessary logging,
which makes log analysis harder and longer.

This previously used to work, before ns_server started passing
all partitions to all desired states for each state transition
call (passing partitions for cleanup that are not currently
marked as active nor passive).

Change-Id: Ieecaec03804ed97d0536694a601b5f68f054c2af
Reviewed-on: http://review.couchbase.org/20618
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
9b33243
Filipe Manana fdmanana MB-6480 Use file:advise/4 when finishing initial index build
When build the btrees at the end of the initial index build,
we're reading sequentially from a file which we will delete
when this step finishes. Reading this file will make the OS
page in all its blocks into the page cache, which can trash
the entire page cache if the file is too big, or we have
many indexes building or indexes with many views for large
datasets.

This change simply periodically tells the OS it can purge
pages from the cache that map to previous data we read from
the file. Calling file:advise/4 after every read significantly
slows down the btree build step, therefore we call it after
every 100Kb bytes were read.

This is a simple adaptation of the technique employed in rsync:

    http://insights.oetiker.ch/linux/fadvise/

Verified that page cache size doesn't grow by ~500Mb when building
a btree from a 500Mb file (via /proc/meminfo and vmstat utility).

The file_advise/4 wrapper was added in OTP R14A:

http://www.erlang.org/download/otp_src_R14A.readme (OTP-8637)

Change-Id: Icb96cf5560d83027a0468cd974f3c45f622d05fa
Reviewed-on: http://review.couchbase.org/20623
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
17cb544
Filipe Manana fdmanana MB-6480 More fair selection of files to merge
Take into account the number of input files when computing
the generation number for the output (merge result) file,
which ensures all future merges will have as input files with
approximate sizes.
For datasets of 20M kv pairs (1 view), indexing time gains in
the order of 4%.

Change-Id: I18f49adedf0eb52a740c5af30b037719e76c2c63
Reviewed-on: http://review.couchbase.org/20688
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
03da4b9
Filipe Manana fdmanana MB-6480 Only wait for oldest worker before spawning new one
Increases the parallelism, allowing the parent to proceed and
do more CPU bound tasks while the new worker will wait for
previously spawned workers to finish before starting its merge
task.
Gains in the order of 3% for a 20M kv pairs dataset with 4 views
in the same design document.

Change-Id: I95b55e9b397104b2e9bc1c27925101a98871f1dc
Reviewed-on: http://review.couchbase.org/20689
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
5282953
Sep 19, 2012
Filipe Manana fdmanana MB-6672: Sleep for right amount of time when not on Windows
The UNIX function 'usleep' takes a quantity in microseconds as a
parameter, but we were wrongly passing it a quantity in milliseconds.

Instead of multiplying the period by 1000, to get microseconds, use
intead the UNIX function 'nanosleep', which receives a timespec
structure, as 'usleep' in many implementations doesn't allow values
greater than 1000000us.

Change-Id: I3ee539e2f86d7088f28968650da41bbdb09c2572
Reviewed-on: http://review.couchbase.org/20863
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
1998314
Filipe Manana fdmanana MB-6653 Fix ddoc cache cleanup on master database delete
Change-Id: Ia216e018987539de58ff691dc870faf394c6b69b
Reviewed-on: http://review.couchbase.org/20858
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
34ae79b
Sep 24, 2012
Filipe Manana fdmanana MB-6689 Stop cleaner before updating unindexable partitions
Before toggling partition between the states indexable and unindexable,
stop the cleanup process. Not doing this was leading us into a bad and
permanent state.

Change-Id: Ib671c56a1410453e6142d72a0c25557032a0e9da
Reviewed-on: http://review.couchbase.org/21041
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c2b69e5
Filipe Manana fdmanana MB-6689 Avoid any work if list of partitions is empty
For calls that request a transition of partitions between the
indexable and unindexable states, totally ignore the call if the
list of target partitions is empty, since it has no effect.
This avoids unncessarily restarting the updater, compactor and
cleanup processes.

It was observed in the logs from MB-6689 that ns_server quite
often passes an empty list of partitions into the functions
couch_set_view:mark_partitions_unindexable/3 and
couch_set_view:mark_partitions_indexable/3.

Change-Id: I59365525e2349fb216711b6cc9fde00ba6e917c8
Reviewed-on: http://review.couchbase.org/21042
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
bcef21d
Sep 25, 2012
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6612: stop maintaining open dbs lru
Because that's long unused and simplifies things for actual MB-6612
fix.

Change-Id: Ie8ac7e5f6d03e962f181d84461cc26ee53cb7309
Reviewed-on: http://review.couchbase.org/20811
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
7587aa0
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6612: open databases synchronously
Problem that we see in MB-6612 is that all_known_databases_with_prefix
is scanning couch_dbs_by_name ets table. Given that previously we also
stored dbs being opened that lead to condition where some vbuckets
would be thought as present while in fact they are simply being tried
to be opened and do not actually exist.

Given that we don't need async db opening in couchbase fork of couchdb
for quite some time it seems logical to just get rid of async open and
thus never have onopened dbs in ets table.

Change-Id: I5d1dad5f60c64d197e143cf4a7be1996a4fc4ea2
Reviewed-on: http://review.couchbase.org/20812
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
fa5b6fe
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6612: just store pid in couch server ets tables
Instead of {opened,..} tuple.

Change-Id: Ie88c1137fa55f55e8b9f1128c12f15750d4b36bd
Reviewed-on: http://review.couchbase.org/20813
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
2613486
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6612: exclude opening dbs from all_known_databases_with_prefix
Because db being opening doesn't imply that it actually exists.

Change-Id: I5e36b16d3c8ec69c5f355341f18fc9e0e3a4b2d2
Reviewed-on: http://review.couchbase.org/20837
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
b945921
Damien Katz Damienkatz CBD-453 Exclude non-utf8 doc ids from views
Change-Id: I9fbe3e6a875f75e838b4fae6cbce098aec2217a2
Reviewed-on: http://review.couchbase.org/20210
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
8f678a3
Sep 26, 2012
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6736: Revert "MB-6612: exclude opening dbs from..."
I.e. 6612 was fixed by preceding 3 commits. I'm uploaded commit that
I'm reverting now in case we could not agree on merging those 3
commits.

Interestingly gerrit cherry-picked this "semantically conflicting"
change without any problems.

So fix is mere revert.

This reverts commit b945921.

Change-Id: Ia819f5cd1c2890642d178ca7f2ab5ce1e205ca6c
Reviewed-on: http://review.couchbase.org/21113
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
804ef00
Oct 03, 2012
Filipe Manana fdmanana MB-6706 Trigger update after defining indexable partitions
Change-Id: If6eeb1e0ed0aeb71e4081cdf74fa1b651190435a
Reviewed-on: http://review.couchbase.org/21070
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
9098ff0
Filipe Manana fdmanana MB-6490 Fix race condition in test 20-debug-params.t
After deleting the databases, wait for the view manager
server to process the view group process exit before
attempting to configure/define a new group. This ensures
we don't get the PID of the old view group that was just
shutdown but had its exit message not yet processed by
the view manager.

Change-Id: I95e86a85203f4c4ea9ddc8ca1a3a4717be6dc581
Reviewed-on: http://review.couchbase.org/21266
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
80cbce1
Filipe Manana fdmanana MB-6490 Allow unindexable partitions in the pending transition
When definining unindexable partitions, it's possible some of them
may not yet be active or passive, but are instead part of the existing
pending transition. Therefore, check and manipulate the pending
transition if necessary.

Change-Id: If2c7c8bfed49649b0694de0fdba2008be190b23e
Reviewed-on: http://review.couchbase.org/21262
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
63a94eb
Filipe Manana fdmanana MB-6490 Don't error when monitoring partitions in pending transition
Don't send an error to the caller if the requested partition is in the
pending transition, as the caller as no way to know when the pending
transition is applied (neither should it know about it, as it's a pure
implementation detail).

Change-Id: If9b9c6ab5bdd1397edc2842b9a7a7f9a43debfa3
Reviewed-on: http://review.couchbase.org/21308
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
780f5c8
Filipe Manana fdmanana MB-6804 Ignore effectless indexable/unindexable calls
Because during rebalance ns_server just keeps asking to set a
group of partitions as unindexable/indexable and they are all
already in the requested state. This avoids restarting updater,
compactor and cleaner processes, as well as writing new headers.
Such effectless and useless calls should be ignored and be a
no-op (node side effects).

Change-Id: I7454bb9b6445db98f7d9111e769f5b566cc2831e
Reviewed-on: http://review.couchbase.org/21304
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
c11c200
Oct 05, 2012
Filipe Manana fdmanana MB-6490 Add missing checks to state transition requests
It's possible the requested state transition will only toggle
partitions states between the active and passive sets in the
pending transition.

Change-Id: I7542b1b1d0f0a5f2dc5e31202a0473d3318e7086
Reviewed-on: http://review.couchbase.org/21347
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
bf5c23b
Oct 08, 2012
Filipe Manana fdmanana MB-6807 Account for exception stringification failure
If stringification of a V8 exception fails, it returns a
NULL pointer, which causes a segmentation fault when it's
used as the parameter for std::string's constructor.

Change-Id: Ibf3d7d84f5eac9b436412608a15acf7b528ce2f8
Reviewed-on: http://review.couchbase.org/21365
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
2db28bf
Filipe Manana fdmanana MB-6807 Fix some harmless warnings
Comparisons between signed and unsigned integers. Harmless
in this context as there can never be more than 256 map
functions (views) per design document.

Change-Id: Ia3fef7e4c5d867e8aed16937788b0d8422feca7a
Reviewed-on: http://review.couchbase.org/21366
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
314270c
Filipe Manana fdmanana MB-6805 Notify update listeners of index updater error
It's possible that several classes of errors prevent the
indexer from progressing (like failures opening database
files for example), which means it blocks update listeners
forever, and this prevents rebalance from progressing.
For this scenarios just send an error message to the
update listeners, so that rebalance doesn't get blocked
forever.

Change-Id: I3c97296d4a06c698917b085b713c547649f18563
Reviewed-on: http://review.couchbase.org/21382
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
e16e611
Oct 09, 2012
Aaron Miller MB-6773 Follow new JSON rules
 * There are no longer reserved field prefixes.
 * We allow any JSON value, not just arrays or objects.

from_binary is also used mostly by the CAPI layer, in order to get a doc
record from a ID/Body pair, and should not bother compressing docs, as
they will immediately be decompressed anyway.

Change-Id: Ib1d1dbeffd0d05cd91c86078ec84cbd4410b05dc
Reviewed-on: http://review.couchbase.org/21324
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
6b9fa5f
Filipe Manana fdmanana Don't release binary after it's transferred to a term
If compression is successful, the binary containing the
compressed content is transferred to an Erlang term.
Yet, the destructor of the compression sink class will
release the binary - while this doesn't seem to cause
any problem, the OTP man pages don't mention if it's safe
to release a binary that was transferred to an Erlang term.
All the OTP man pages say is that a binary is freed by
either calling enif_release_binary or enif_make_binary,
giving the idea they are mutually exclusive operations.

Change-Id: I341f685973031ae13b71daacd3febaf358c2f3be
Reviewed-on: http://review.couchbase.org/21339
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
e385cae
Filipe Manana fdmanana Don't allocate binary buffer with 0 bytes
For the standard C library, the behaviour of calling malloc
to allocate a bufer with 0 bytes is implementation specific.
Some implementations return a valid pointer while others
return a NULL pointer.

It's not clear if the same problem can happen directly with
enif_alloc_binary(), or if its implementation can result in
an malloc call for 0 bytes.

Nevertheless it's safer to start with an initial binary buffer
size greater than zero.

Change-Id: I282d4fab6d86be478d0cd5233c433e4b25b6e5eb
Reviewed-on: http://review.couchbase.org/21340
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
55973d4
Oct 10, 2012
Filipe Manana fdmanana MB-6860 Release index file ref counter
After serving a query through the optimized path, release
the index ref counter, so that after compaction the file is
effectively deleted by the OS.

Change-Id: Iff956a0ee38ccfbe4fcf3d5edfddde1fee06cfc2
Reviewed-on: http://review.couchbase.org/21468
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
091f7f4
Oct 11, 2012
Aliaksey Kandratsenka (aka Aliaksei Kandratsenka) alk MB-6841: [lhttpc] Un-urlencode credentials in urls
Comments in split_credentials correctly indicate that any special
chars need to be 'percent-encoded'. So in order to get their true
values we need to un-encode.

Otherwise there's simply no way to use usernames/passwords with
special characters when doing basic auth via urls.

NOTE: This is same patch as was recently accepted upstream except we don't
have lhttpc's tests. I.e. esl/lhttpc#19

Change-Id: I1ea38d890d3b41e33e813c62deb52166ae7794fe
Reviewed-on: http://review.couchbase.org/21461
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
ce405fb
Filipe Manana fdmanana MB-6868 Add couch_set_view:get_group_signature/2
So that other components, such as ns_server, can get the
signature of a view group in a very cheap way (get_group_info
is ex+ensive, meant only for debugging).

Change-Id: Ie000a25e2b828a0d9de042701808c15b7f9260ad
Reviewed-on: http://review.couchbase.org/21490
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
d2992d2
Filipe Manana fdmanana Add missing file flush call to test 020-btree-basics.t
This caused intermitent failures on slower environments
(example, often in the buildbot).

Change-Id: I5a07bd4fe9115d2cfe37b5c048c43560db81e4fb
Reviewed-on: http://review.couchbase.org/21538
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
b6454ca
Filipe Manana fdmanana MB-6866 Use different thresholds for kp and kv nodes
For views, since values are mostly raw JSON, compression
gains are high, therefore chunk threshold (uncompressed
size) can be raised, which decreases the number of kv nodes
in the trees. For kp nodes, because they all have a 1024
bits bitmask in their reductions, their branching factor was
a bit low, which made the trees deeper.

These changes made queries and compaction faster both for
generic btree tests and evperf lucky8 views (lucky8.conf and
lucky8-3d.conf). Inserts and updates also get a small speedup.

Change-Id: I2a94ec6005b881c81fd5295a2c89c7657cc6b19f
Reviewed-on: http://review.couchbase.org/21484
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
c62a4b2
Filipe Manana fdmanana MB-100 Change log level and unify messages
Ensure all errors are logged at error level and unify
the error and stacktrace messages so that it's easier to
link them when there are multiple requests in parallel
producing errors.

Change-Id: I6380d896ac36513e15cf0df5a2e949fc4ea94aa5
Reviewed-on: http://review.couchbase.org/21541
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
6cb0fa7
Filipe Manana fdmanana MB-6629 Faster index compaction retries
Revert to initial approach where compactor keeps
reading log file until it reaches EOF, instead of
coordinating with the updater regarding where the
EOF is and stop once it reaches that EOF, which
very often no longer matches the real EOF.

Verified with evperf lucky8 test that with this
approach there's no more than 2 to 3 retries, as
opposed to up to ~20 retries with previous approach.
Total compaction time average, and 99% percentile,
is almost half of what it was before during the
lucky8 test run.

Change-Id: I3c893a780a7878080fb366ba72b15e544bb36834
Reviewed-on: http://review.couchbase.org/21516
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
719e315
Oct 12, 2012
Filipe Manana fdmanana Revert "MB-6629 Faster index compaction retries"
This reverts commit 719e315.

Change-Id: I48a351f929d37121682d7303cdaaf530ede59221
Reviewed-on: http://review.couchbase.org/21582
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
4a1882b
Filipe Manana fdmanana MB-6799 Process group update
Before applying the pending transition, process the new
group received from the updater, which is clean. This
ensures we can apply the pending transition immediately.

Change-Id: I2962ae17bd15a7b8a2f13b38f0813632323067f5
Reviewed-on: http://review.couchbase.org/21551
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1b21587
Oct 14, 2012
Paul J. Davis davisp MB-6591: Use raw collation for _all_docs query params
_all_docs are always sorted in raw order, not Unicode. Hence
the query parameters (startkey and endkey) also need to
take this raw collation into account. The query validation checks
whether startkey is smaller than endkey, hence we need to take
the collation type (Unicode or raw) into account.

An example for raw collation is:

    Foo < bar < foo < zoo

With Unicode collation it would be:

    bar < foo < Foo < zoo

This patch is based on a patch for Apache CouchDB [1].

I've kept the original commit message below.

[1] apache@46d2dc6

Fix empty range check for raw collation.

The check for empty ranges was not taking into account the
view option for raw collation. This fixes that by passing
the couch_btree:less/2 function into the check.

Patch by: Jason Smith
Re: COUCHDB-1228 4/4

Change-Id: I4569683ff0253221f2f265e287b7a52d2ad32ac1
git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1156509 13f79535-47bb-0310-9956-ffa450edef68
Reviewed-on: http://review.couchbase.org/21359
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
fd03c3a
Oct 15, 2012
Volker Mische vmx MB-6860: Only delete .view files
When the mapreduce indexes are cleanup, only delete files that
have ".view" as an extension. This prevents cleanup_index_files/1
from deleting spatial indexes (which have a ".spatial" extension
and reside in the same directory).

Change-Id: I0ff1ea99687560816ae63d1aac3ca93e39a54776
Reviewed-on: http://review.couchbase.org/21588
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
23cec99
Oct 16, 2012
Farshid Ghods farshidce MB-6890 remove doc-from-binary test from distdir
Change-Id: I9f39a1780ad7d3614786cbf4744e5ea90648cff0
Reviewed-on: http://review.couchbase.org/21570
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
9ffd974
Filipe Manana fdmanana MB-6860 Don't open dev indexes during cleanup
For old couchdb indexes, used for development views with
single vbucket database, don't open a view group to get
its signature during cleanup. This avoids creating extra
Erlang processes, which live forever and keep 2 file
descriptors open for empty, and old, couchdb index files.

This was causing 2, useless, file descriptors open forever
per design document until the bucket was deleted

Change-Id: I7ae6a0531d2b60121c6ceb491a7c84babf87ce98
Reviewed-on: http://review.couchbase.org/21643
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
255e3a6
Filipe Manana fdmanana Return correct max_size value for allocators
As it's done in the standard/default allocator that
uses the operators new and delete:

http://gcc.gnu.org/onlinedocs/gcc-4.6.3/libstdc++/api/a00958_source.html

Change-Id: Ib6e2580a1e9f6a55a3cf220af808a374449f5a24
Reviewed-on: http://review.couchbase.org/21585
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
03cee9a
Filipe Manana fdmanana Update allocator's allocate method
So that it conforms more to the way the default/standard
allocator based on the operators new and delete:

http://gcc.gnu.org/onlinedocs/gcc-4.6.3/libstdc++/api/a00958_source.html

Also use static_cast instead of plain old C casts, and
avoid using the NULL constant, as these are best practices
in C++.

Change-Id: I7d5b24c1cae8bd0f3bcfcae4ed276a2d0a4fb0cc
Reviewed-on: http://review.couchbase.org/21586
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
69032c6
Oct 19, 2012
Filipe Manana fdmanana MB-6860 Fix old couchdb view cleanup when there are no ddocs
If there were no design documents, it wouldn't delete existing
useless index files.

Change-Id: I88b1f5950eed8bb0b04287fd14a4ccdf90083e55
Reviewed-on: http://review.couchbase.org/21745
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
5353fd9
Filipe Manana fdmanana MB-6860 Shutdown outdated dev index processes
After development design documents get updated or deleted,
ensure the corresponding view group processes (and files)
are shutdown/closed. This is used to work over an year ago,
and got fixed in Apache CouchDB, but since then Couchbase
uses the old couchdb views in a different way - the design
documents are always located in separate database (master)
from the database where data comes from (vbucket database).
This Couchbase specific way of using couchdb views broke
the shutdown logic of outdated and unnecessary view group
processes and their respecive index files.

This only affected development views that don't use the
full_set=true parameter.

Change-Id: I42d66cd1fdd9798ce43debb7b6d6579cd72ae273
Reviewed-on: http://review.couchbase.org/21746
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
eaa9847
Oct 22, 2012
Volker Mische vmx MB-100: Don't throw error on non-existing directory
Don't throw an error if the directory you want to delete
doesn't exist.

Change-Id: Id7e4dbfacc952ac3a2a2069b66c61c29303be4fe
Reviewed-on: http://review.couchbase.org/21711
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
33f201e
Oct 23, 2012
Filipe Manana fdmanana MB-6860 Ensure dev index file deleted after db deletion
Often view group would receive a db process monitor DOWN
message before couch_view received the database deleted
event, in which case the index file would end up not being
deleted. The solution here is to shutdown the view group
only through couch_view when the database is deleted.

Change-Id: Ife6311b3a50d5c9b5576bcb553b207c5b162c3d0
Reviewed-on: http://review.couchbase.org/21828
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
0ee5236
Filipe Manana fdmanana MB-6988 Add more data to output of queries with ?debug=true
Add some more index meta data, from all nodes, to the output
of queries with ?debug=true. Also make the same debug information
available for reduce queries as well, which was not working
after some optimizations done long ago.

Change-Id: I975130d95de196a29528ec061b11e1c89a080520
Reviewed-on: http://review.couchbase.org/21869
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
b4391e4
Farshid Ghods farshidce MB-6969 add snappy-stubs-public.h.in to snappy dist target
Change-Id: I111d01a73aa44df2defb477ee4c0f1a5d1d31888
Reviewed-on: http://review.couchbase.org/21801
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
c2629cd
Farshid Ghods farshidce MB-6969 add lhttpc.hrl to DIST target
Change-Id: I717ef85de9398eeafa31a950ea249efeb46e327f
Reviewed-on: http://review.couchbase.org/21802
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
68e8e55
Farshid Ghods farshidce MB-6969 add missing .erl file to couchdb Makefile.am
also removed couch_file.erl_config_writer.erl from
source_files as it was causing make dist to fail since
this file does not exist anymore

Change-Id: I33ab1fb23cef072ae0dc440c7096a7ee5dae4f4c
Reviewed-on: http://review.couchbase.org/21804
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
8bde00d
Oct 24, 2012
Volker Mische vmx MB-6990: Add validation of the spatial function
Design Docs that contained invalid spatial functions weren't
rejected, now they are.

This commit depends on http://review.couchbase.org/21890

Change-Id: If775539e965f1f3f8f1af807559c61e57ca15054
Reviewed-on: http://review.couchbase.org/21891
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
ebe5a3a
Oct 25, 2012
Filipe Manana fdmanana MB-6990 Ignore missing module during make check
During make check, or when running CouchDB standalone for
testing purpose, the spatial modules (GeoCouch) are not in
the Erlang path, whence just ignore the function call in
this case.

Change-Id: Ia02b37eef52960852189c2dc2b689198cecf89dc
Reviewed-on: http://review.couchbase.org/21962
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
ab2b0c2
Filipe Manana fdmanana MB-6947 Don't log set_view_outdated errors in httpd layer
These are already logged by the index/view merger, with a
log level of 'info' and with more useful details.

Change-Id: Id919d4ffedc60f58bf708f209689d0ab7eb2408a
Reviewed-on: http://review.couchbase.org/21963
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
24ef158
Aliaksey Artamonau aartamonau MB-6995 Don't restart couch_{server,set_view} on dir changes.
If some external (meaning ns_server) process calls anything from
couchdb at the time of restart, it may cause this process to terminate
in the hard way with reached_max_restart_intensity error. To avoid
this we're going to stop couchdb from ns_server in coordination with
all the other ns_server processes.

Change-Id: Ifea0c6e3bbebbdd36efedd4fe10bb2a7ccb27892
Reviewed-on: http://review.couchbase.org/21946
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
18bce93
Aliaksey Artamonau aartamonau MB-6995 Stop related apps when couchdb application is stopped.
When ns_server needs to restart couchdb, we want it to restart not
only couch app but also all the related apps.

Change-Id: Ic8e1d316fd1c2e765c7783e5009cbf9448cc9b13
Reviewed-on: http://review.couchbase.org/21947
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
a5a033c
Oct 26, 2012
Aliaksey Artamonau aartamonau MB-7025 Shutdown databases in couch_server:terminate correctly.
Since recently couch_dbs_by_name contains only PID and nothing
more.

It's not clear though if this actually fixes referred bug.

Change-Id: Ifb3a0bd9a9c011c4836e0513a4adbaba94c3c8c1
Reviewed-on: http://review.couchbase.org/22021
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
7a0f869
Filipe Manana fdmanana MB-6107 Do less fsyncs on index compaction
When reaching the retry phase, there's really no need to
do an fsync for each retry iteration. Instead just do a
single one at the end.

Also change the order of some flush calls (without affecting
correctness) just to increase parallelism.

Change-Id: I7021cde3aa54b5c9c7a04b0513301bce6c30ee8d
Reviewed-on: http://review.couchbase.org/21964
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
e29b945
Filipe Manana fdmanana MB-6107 More efficient index compaction retry phase
Ensure that during the index compaction retry phase, we
don't insert/remove many small batches into the btrees.
Instead do less batch operations but with larger batches,
reducing IO and causing less fragmentation.

Change-Id: Idc9a8059c2e74f4f3cfe19a7be98e3a9d556500d
Reviewed-on: http://review.couchbase.org/21965
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
09b4e43
Filipe Manana fdmanana MB-6107 Make index btree chunk thresholds configurable
Allow via couch_config to change the thresholds for the
btrees created for views. The purpose is just to allow
for evperf team to easily do tests with different values
for these thresholds.

Change-Id: I0e68127159b38081dd64391d4dc2b43004233113
Reviewed-on: http://review.couchbase.org/21968
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
77e563f
Filipe Manana fdmanana MB-7030 Always send new group snapshot to parent
Instead of sending it with a minimum periodicity of 5 seconds,
every time a batch of changes is applied to the btrees, send the
new view group snapshot to the parent. This prevents discarding
some indexing progress when there's a quick succession of partition
state transitions, as it happens during rebalance.

This restriction was added in the past with the goal of reducing
memory copying (group snapshots were structures with much more
fat).

Change-Id: I4a56b1eaf1019e3203dcaf4e23422145a0b64d21
Reviewed-on: http://review.couchbase.org/22006
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
2f17fe5
Filipe Manana fdmanana MB-7030 Use some cheaper BIFs for list and orddicts
lists:member/2 is cheaper than ordsets:is_element/2, and
lists:keyfind/3 is cheaper than orddict:is_key/2 and
orddict:fetch/2.

Change-Id: I657f644d7b675097cec46cfbabb0aba4ff02fcd7
Reviewed-on: http://review.couchbase.org/22007
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
a2bef2f
Filipe Manana fdmanana MB-7030 Check mail box for new group snapshot
When stopping the updater, check if there's currently
any new group snapshot sent by him to us. If so, process
it.
Before this change it was ignored, meaning that the next
time the updater was restarted, it would repeat some work,
wasting CPU and IO.

Change-Id: I840bc797567a1d7c81f9970aaf4e1b9d20271b1e
Reviewed-on: http://review.couchbase.org/22008
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
c2dd289
Filipe Manana fdmanana MB-7030 Avoid not useful anymore header commit
When toggling partition between the indexable and unindexable
states, don't build and write an header (and fsync it). This is
doesn't compromise reliability anymore, as if a crash happens,
there's no important state information loss - on restart ns_server
always attempts to configure indexes correctly (since MB-6310).

Change-Id: I992d3543ceef233b2694489fbc981511e4922df0
Reviewed-on: http://review.couchbase.org/22009
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
a0b3bb6
Filipe Manana fdmanana MB-7030 Don't fsync when doing some header commits
Fsync after writing an header only if the new cleanup bitmask
differs from the old cleanup bitmask. This is safe nowadays
as ns_server on startup configures view groups (MB-6310), and
sets the correct vbucket states. The fsync when cleanup bitmask
changes is necessary because otherwise on restart after a crash,
the view group might attempt to open a vbucket database that was
deleted right before the crash but before the header hit the disk.

Change-Id: I6b70cb7fae55f6a856c56aae8659e8072a64b351
Reviewed-on: http://review.couchbase.org/22010
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
aa361e5
Filipe Manana fdmanana MB-7030 Don't restart cleanup process
When doing the transition of partitions between the indexable
and unindexable states, don't restart the cleanup process.
When this process finishes, update its header's list of
indexable and unindexable partitions.

This saves some CPU, IO and can save up to 5 seconds per state
transition request (one which asks to toggle partitions between
the indexable and unindexable states only).

Change-Id: Ida8dd5b2ade1d039de363a88e4964c130cc17046
Reviewed-on: http://review.couchbase.org/22011
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
d2eb7d5
Filipe Manana fdmanana MB-7030 Avoid unnecessary group pid lookup
When list of partitions to mark as indexable or unindexable is
empty, just be a complete no-op, that is, don't look up the
design document in the cache (or disk), and don't compute its
signature to get a group pid.

Change-Id: If219aba61846ec6139e19055cbab0e1ad8f51764
Reviewed-on: http://review.couchbase.org/22012
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
1a5f629
Oct 29, 2012
Filipe Manana fdmanana MB-6957 Retry file operations on Windows
Due to the fact that some operations provided by the
Erlang 'file' module open files without all the share
flags, and that there may be external processes opening
files without those flags as well (Windows services,
antivirus software, etc), it makes some concurrent
operations against the same file fail with a Windows
share violation error, which Erlang file driver maps
to posix error 'eacces'.

When this happens, just retry the failed operations for
a limited period of time, after which we give up.

Change-Id: Iaecc6d520169d8b84bfcb354066e42b533c435cd
Reviewed-on: http://review.couchbase.org/22042
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
e6bcd6e
Oct 31, 2012
Filipe Manana fdmanana MB-7039 Avoid crash if replica index file is missing
When opening a previously created and configured index, if
for some reason the replica index file is missing (or doesn't
have an header, due to a crash), we reach a situation where
opening the index becomes impossible, unless we explicitly
delete the main index file.

The solution here is to configure the replica index, if it's
missing, when opening the main view group.

Haven't observed this happening in practice, but it's a possibility, therefore good to address for extra reliability.

Change-Id: I42fe0276770a4469a83a94e5e0f88bbbd5370d99
Reviewed-on: http://review.couchbase.org/22038
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
be312ff
Filipe Manana fdmanana MB-6107 Remove no longer needed code
If the set of replicas on transfer is modified, the compactor
is restarted, so there's no point of checking if the compactor's
group has an up to date set of replicas on transfer.

Change-Id: Ifd97c5ba79ee2ae6699939bd739378cd455152aa
Reviewed-on: http://review.couchbase.org/22034
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
c148470
Filipe Manana fdmanana MB-6107 Remove no longer needed and wrong logic
No need to modify the group snapshot bitmasks. The desired
set of indexable partitions must match the group's active
bitmask only.

Change-Id: I7dce56632742127e25b986f3cb30f80163f5ab8e
Reviewed-on: http://review.couchbase.org/22035
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
29ea390
Filipe Manana fdmanana MB-6107 Remove unnecessary tuple construction
Leaf nodes are always of type kv_node.

Change-Id: I5b3cfab90a3cb596e80aba0958b14acc2c2fd418
Reviewed-on: http://review.couchbase.org/22036
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
5d5d2dd
Filipe Manana fdmanana MB-6107 Remove couch_file:sync/1 call after index compaction
Not really worth the cost, and after a crash/node restart,
ns_server always attempts to correctly connfigure indexes
(as of MB-6310).

Change-Id: I13b39a70ea343641027bfe96c6ebffe5ed5888a5
Reviewed-on: http://review.couchbase.org/22037
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
47a991b
Damien Katz Damienkatz MB-6945: Increase size of rev seq from 32 bits to 48 bits
Also, bump file version so we don't open old files with 32 bit format.

Change-Id: Ifecaa99e6e7f6fd543d2ba0c88f5add0301508fd
d53f34c