Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

We’re showing branches in this repository, but you can also compare across forks.

base fork: basho/riak_core
base: cet-bg-mgr-proto
...
head fork: basho/riak_core
compare: cet-bg-mgr-vnode-lock-rebased
Commits on Jul 31, 2013
Scott Lystig Fritchie slfritchie Add #riak_core_fold_req_v2 record, part 1 of 2
Move definition of ?FOLD_REQ to v2.

Add functions to riak_core_util.erl to create v2 versions of the
record: arity 1 (upgrade #riak_core_fold_req_v1 record to the v2
record), arity 2 (args = the two record elements of the v1 record)
and arity 4 (args = the four record elements of the v2 record).

Disable request forwarding in riak_core_vnode:forward_or_vnode_command()
if the request explicitly contains not-forwardable annotation.
4e26f99
Commits on Aug 08, 2013
Jordan West jrwest Cluster Metadata: The Basics
This commit implements the basic moving parts for riak_core's new k/v
store for cluster-wide information. Cluster Metadata is meant to
replace current uses for metadata stored in `riak_core_ring`.

The interface for interacting with the store is defined in
`riak_core_metadata`. Data is read and written locally and is fully
replicated to all nodes eventually. Data is stored on disk using dets
and is managed by `riak_core_metadata_manager` (the ground work has
also been laid to keep a copy of data in-memory w/ ets for use on
reads). Data is addressed by a given prefix (a two-tuple of any erlang
terms) and a key. Each stored prefix/key has an associated logical
clock (a dvvset -- more below). The representation of the object and
its clock are abstracted via `riak_core_metadata_object`.
`riak_core_broadcast` implements a new broadcast/gossip protocol for
disseminating updates to other nodes. The module is implemented to be
a general broadcast protocol that could be used by other
subsystems. `riak_core_broadcast_handler` is a new behaviour that
defines the API a module must implement to use
`riak_core_broadcast`. `riak_core_metadata_manager` implements this
behaviour. `riak_core_broadcast` is based off the plumtree protocol
(http://www.gsd.inesc-id.pt/~jleitao/pdf/srds07-leitao.pdf) with some
modifications to make it more partition tolerant and fit with
riak_core's membership model. The implementation uses the sender-based
tree approach mentioned but not analyzed in the paper. The current
implementation does not fully guaruntee at-least once delivery (it
will be fully implemented in a future commit).

`riak_core_broadcast` and `riak_core_metadata_manager` run per node
and are started after the ring manager.

This commit introduces dvvset.erl from
https://github.com/ricardobcl/riak_core/blob/dvvset/src/dvvset.erl. This
may not be the actual version of Dynamic Version Vectors
used. `riak_core_metadata_object` is meant to abstract the object
representation and logical clock so a switch to a new implementation
of dvvset or one just using vector clocks can be made easily.
650a00a
Commits on Aug 12, 2013
Jordan West jrwest Improve Cluster Metadata Storage
* move to using a dets file per prefix/subprefix
* store data in ets table per prefix/subprefix. read from ets
* restricts prefix/subprefix to binaries
31dc209
Jordan West jrwest Basic Iterators for Cluster Metadata
Allows iteration and folding over all keys stored under a
prefix/subprefix
bd8218f
Commits on Aug 13, 2013
Jordan West jrwest Further Cluster Metadata Storage Improvements
* MD5 prefixes for use in filenames. Raw binaries are unsafe and
  difficult to parse.
* Store metadata in in a manifest file. Folding over files in a directory
  to determine existing data is unsafe, error prone, and stupid.
* Re-allow prefixes to contain atoms. To prevent collision between binaries
  and atoms a filename trailer is appended.
* Sync all dets writes to disk so we don't easily lose data on emulator crash"".
92b8a6a
Jordan West jrwest Fix Cluster Metadata Conflict Resolution
* pass proper arity function to dvvset:reconcile/2
* timestamp resolved value
1c47f8d
Jordan West jrwest Add Optional Conflict Resolution to Metadata Iterators
* When resolution is used, iterators are "single-use".
The iterator functions aren't pure in this case. If no resolution option
is provided then this resriction does not apply
* Resolution can be done via function or lww like get/3
2036a9b
Jordan West jrwest Allow Conflict Resolution on Metadata Write
extend riak_core_metadata_object:modify/4 to take a
resolution function (it differs from resolve/2, see edoc).
44495bb
Jordan West jrwest Add Conflict Resolution in Metadata Folds
using the existing support in iterators
c891275
Jordan West jrwest Add Logical Deletion to Cluster Metadata
* similar to ring metadata
* write tombstone values on delete
* handle tombstones on read
* resolvers and modifiers must be tombstone-aware. see typespecs.
* iterators/folds can optionally return the tombstone value or can provide
a default value to return or function to apply when they are encountered
7f3ed73
Jordan West jrwest Iterate only a Subset of Cluster Metadata Keys
If keys are tuples then a match-spec like pattern
can be passed as an option to iterator/2 and fold/4 to
filter which keys the iterator visits.
10f8403
Commits on Aug 14, 2013
Jordan West jrwest Teach riak_core Bucket Types
Like buckets, we must teach core this because a few subsystems
use them. Bucket Types provide a method for grouping buckets
logically (see basho/riak#362).
b74d7d5
Commits on Aug 15, 2013
Sean Cribbs seancribbs Support both new and old-style crypto MD5 functions. 5e76eb2
Commits on Aug 17, 2013
Evan Vigil-McClanahan evanmcc make core aware of possible null returns 6cfa2cb
Commits on Aug 19, 2013
Engel A. Sanchez engelsanchez Add protection against folsom stat errors
Folsom may sometimes return an error tuple if something goes wrong (see
folsom_ets.erl), but our code was only catching exceptions. So the error
would end up being used as a valid value and crash the riak_kv_stat
process later. This fixes that problem and gives us better protection
from folsom funkiness.
586df15
Evan Vigil-McClanahan evanmcc Merge pull request #359 from basho/pevm-drop-bad-data
Corruption filtering changes for core.
84eb7b1
Commits on Aug 20, 2013
Engel A. Sanchez engelsanchez Merge pull request #356 from basho/eas-folsom-stat-error-protection
Add protection against folsom stat errors
41add84
Commits on Aug 23, 2013
Jared Morrow jaredmorrow Roll webmachine dep to 1.10.4 5d6b5b9
Jared Morrow jaredmorrow Roll riak_core version 1.4.2 3352fa2
Commits on Aug 27, 2013
Andrew Thompson Vagabond Isolate app.config from non-default bucket type properties, implement…
… reset

Non-default bucket-types are unaffected by changes to app.config and
bucket types can have their properties reset to the hardcoded defaults.
81024be
Commits on Aug 30, 2013
Scott Lystig Fritchie slfritchie Add riak_core_capability handling for fold requests ef7c3ad
Commits on Sep 03, 2013
Engel A. Sanchez engelsanchez Merge branch 'jrw-cluster-meta' into eas-bucket-type-merged a41a7e8
Engel A. Sanchez engelsanchez Merge branch 'adt-bucket-type-defaults-and-reset' into eas-bucket-typ…
…e-merged
40d3220
Commits on Sep 04, 2013
Engel A. Sanchez engelsanchez Fix meck related unit test problems
Meck now needs a couple of flags in meck:new for some of our tests:
* non_strict so we can create a fresh non-existent module
* no_link so it doesn't link to the processes created with a meck
* module. Bad when we kill things on purpose.

Trying non_strict meck flag in riak_core_stat_cache.
The build failed with an undefined_module error setting up this text. I
can not get that to happen locally, but this flag has fixed similar
problems in other tests.
3ea67a8
Engel A. Sanchez engelsanchez Merge pull request #367 from basho/eas-fix-meck-funkiness
Fix meck related unit test problems
9a6074a
Commits on Sep 05, 2013
Engel A. Sanchez engelsanchez Merge 1.4.2 changes to develop
Conflicts:
	rebar.config
05e0cec
Ryan Zezeski rzezeski Use raw ring when setting bucket props
Fix the ring trans function used by `set_bucket` so that it uses the
passed-in, raw ring.  This is important because otherwise things like
bucket prop fixups which are performed on "my ring" can leak into the
raw ring casuing issues such as a Riak Searh pre-commit hook that
exists even when the search property is set to false.
a9a676a
Engel A. Sanchez engelsanchez Merge pull request #368 from basho/eas-merge-1.4.2
Merge 1.4.2 to develop
c2f06d6
Ryan Zezeski rzezeski Merge pull request #369 from basho/bugfix/set-bucket-develop
Use raw ring when setting bucket props (develop branch)
47283e7
Commits on Sep 07, 2013
Engel A. Sanchez engelsanchez Merge branch 'develop' into eas-bucket-type-merged
Conflicts:
	src/riak_core_bucket.erl
b557151
Jordan West jrwest move ring update logic into broadcast process
this avoids several unnecessary messsages and makes things cleaner
as pointed out in review.

In addition, change all_broadcast_members/1 to use riak_core_ring:all_members/1
since they are equivalent.
744c6ce
Jordan West jrwest add configurable timeouts to calls in riak_core_broadcast
the calls are primarly used for debugging (primarly in a live system)
so making the timeouts configurable seemed like the best idea. The original
functions default to 'infinity'.
82942f2
Jordan West jrwest fix typespec for broadcast_handler:graft/1 ee0d41f
Jordan West jrwest change all metadata manager calls to use infinite timeout c8cb75f
Jordan West jrwest Merge pull request #363 from basho/jrw-cluster-meta
Cluster Metadata (Part 1/2)
926a185
Engel A. Sanchez engelsanchez Merge bucket type integration branch to develop a5bec27
Commits on Sep 11, 2013
Sean Cribbs seancribbs Move webmachine to riak_api. 220cd86
Sean Cribbs seancribbs Remove mostly-unused riak_core_config. ae619be
Sean Cribbs seancribbs Copy in mochiglobal as riak_core_mochiglobal. 13f894a
Sean Cribbs seancribbs Move json_pp to riak_kv. e7633e0
Sean Cribbs seancribbs Remove default_bucket_props/0 that is unnecssary with bucket-types. 97fb69b
Sean Cribbs seancribbs Merge pull request #381 from basho/refactor/sdc-wm-move
Remove webmachine dependency
36e6f90
Commits on Sep 12, 2013
Steve Vinoski vinoski update folsom dependency to 0.7.4p3
Pick up version 0.7.4p3 of folsom due to upstream change in our meck
version.
38b73c7
Steve Vinoski vinoski use protobuffs 0.8.1p2 due to change in required meck version d334008
Scott Lystig Fritchie slfritchie Add #riak_core_fold_req_v2 record, part 1 of 2
Move definition of ?FOLD_REQ to v2.

Add functions to riak_core_util.erl to create v2 versions of the
record: arity 1 (upgrade #riak_core_fold_req_v1 record to the v2
record), arity 2 (args = the two record elements of the v1 record)
and arity 4 (args = the four record elements of the v2 record).

Disable request forwarding in riak_core_vnode:forward_or_vnode_command()
if the request explicitly contains not-forwardable annotation.
daf13ca
Chris T buddhisthead Merged #354 from slf-vnode-ops-opts c7aaf70
Chris T buddhisthead Merge branch 'develop' of https://github.com/basho/riak_core into dev…
…elop
2276faf
Chris T buddhisthead Fix merge conflict; why didn't I see this before, you ask? a2315c1
Commits on Sep 13, 2013
Scott Lystig Fritchie slfritchie Fix boolean() guard hack, missing from merge @ c7aaf70 8e220a3
Scott Lystig Fritchie slfritchie Merge branch 'slf-fix-354-merge-bug' into develop 4ae876b
Scott Lystig Fritchie slfritchie Add missing commit from PR 354
This patch adds a commit that was missing from the merge of PR 354,
commit ef7c3ad.
047a128
Commits on Sep 15, 2013
Jordan West jrwest Hash Trees for use in Cluster Metadata
hashtree_tree implements a data structure for efficiently determining
the differences between keys in two different stores. Unlike hashtree,
which this module is based on, keys can be logically grouped such that
differences found include missing groups as well as which groups the
differing or missing keys belong to.

This commit copies hashtree.erl from riak_kv (it will be removed in
that repo). In addition, it adds a (temporary) dependency on eleveldb,
which is used by hashtree.erl. hashtree_tree is intended to be used
ephermerally (although there is no reason it cannot be persistent) but
hashtree must first be extended to have pluggable backends.
114cd4a
Jordan West jrwest Add Prefix Iterators to Metadata Manager
These iterators are necessary to build hashtrees and
repair divergent or missing full-prefixes
6c2ecdc
Jordan West jrwest Cluster Metadata: Track Local Changes w/ hashtree_tree
Use hashtree_tree to store hashes for data in
riak_core_metadata_manager.  The tree is managed by
riak_core_metadata_hashtree. Hashes are inserted into the tree,
synchronously (for now) by metadata manager in the write path. In
addition, metadata_hashtree builds the tree with hashes for data
already stored on disk (ensuring not to trample new updates in the
process). Once the tree is built, it is periodically updated. Minus
the periodic update, the metadata_hashtree process is akin to
riak_kv_index_hashtree in Riak's AAE subsystem.

The API to access hashes is exposed via riak_core_metadata:prefix_hash/1.
de351bc
Jordan West jrwest Add Remote Iterators to Metadata Manager
Iterating from a node other than the one running the metadata manager
was not possible, previously -- ets continuations cannot cross
nodes. This is intedended for use in repair after hashtree exchanges.

In addition, iterator_close/1 has been added. This is necessary to cleanup
remote iterators. For local iterators it is a no-op.

This API is not exposed via riak_core_metadata since the iterface is
primarly meant to be local.
62a4d69
Jordan West jrwest Cluster Metadata Exchanges
Use metadata hashtrees between nodes to determine differences or
missing keys and (full-)prefixes. Repair using metadata manager's
merge function (an additional version has been added to merge remotely).
The metadadata_exchange_fsm is the managing process.

NOTE: currently exchanges are not "plugged in" anywhere. They are not
started automatically. This commit simply implements the determination
of differences and the repair of those differences.
e4acace
Jordan West jrwest add metadata hashtree modules to .app d6d8592
Jordan West jrwest Add General Exchanges to riak_core_broadcast
to ensure at-once delivery riak_core_broadcast must periodically
trigger full exchanges between two nodes. Since riak_core_broadcast is
meant to be general and can be used by multiple sub-systems
simultaneously, the definition of "exchange" is losesly
defined. Broadcast, now, simply takes a list of modules that will
broadcast messages (this list is not, yet?, checked on broadcast) and
periodically asks them to exchange with a peer.  An exchange may
involve database shipping or exchanges and repair using hashtrees --
as is done for riak_core_metadata_manager in this commit.
d957587
Jordan West jrwest copy in hashtree docs & eqc
these were originally in riak_kv. since hashtree.erl is now
in riak_core these should be here too
97c6b77
Jordan West jrwest Make Broadcast/Cluster Meta Exchanges more robust
* better process failure handling
* monitor exchanges from broadcast process
* don't link exchange fsm
ec747df
Jordan West jrwest cleanup metadata hashtrees on open
in the case that riak crashes, cluster metadata's hashtrees
may not be properly cleaned up. because we only want them for the lifetime
of a running node, we destroy the trees on open and then re-open them. Not
exactly ideal, but quick and easy.
a3fa1c6
Jordan West jrwest whitespace cleanup in cluster metadata code af1b4cf
Jordan West jrwest add simple logging to metadata exchange fsm 63e550b
Jordan West jrwest More Metadata Iterator Improvements
* Use safe_fixtable/2 for iterators that do first/next traversal of
ets tables
* Open iterators inside the calling process. this allows iterators to
take advantage of monitoring built into ets for processes using
safe_fixtable/2 and possibly select/match_object depending on
the implementation of ets. In addition, this prevents serialization through
the metadata manager.
* Closing iterators is now mandatory for all iterator types. when an iterator,
using select/match_object has been closed that has not been fully iterated,
closing the iterator will complete the iteration. This is a definisive measure
since its is unclear if this is necessary from the ets documentation.
* Processes using remote iterators are monitored. On death, their iterators are
closed.
0b59e4c
Jordan West jrwest Fix Some Metadata Exchange Bugs
* properly merge local object with remote node
* handle missing remote object on reconcile
9885dc5
Commits on Sep 16, 2013
Jordan West jrwest add module documentation for metadata hashtree mods 030d968
Jordan West jrwest update hashtree app env calls
try to get application env from riak_kv and if it doesn't exist
fallback to riak_core and then the default values. this is to ensure
upgrades do not lose settings specified in riak_kv.
ded140d
Commits on Sep 17, 2013
Chris T buddhisthead Expose the resolution of capabilities for general use in protocol neg…
…otiations
24e45d1
Chris T buddhisthead Clean up the doc lines. c4bfd01
Commits on Sep 20, 2013
Andrew Thompson Vagabond Initial work on user/source/grant/revoke management 40ed6d4
Andrew Thompson Vagabond More work on security 6caee15
Andrew Thompson Vagabond Some initial work on GRANT/REVOKE 55d2437
Andrew Thompson Vagabond Fix a bug with wildcards b5240df
Andrew Thompson Vagabond Implement REVOKE and fix some bugs 1a0463b
Andrew Thompson Vagabond Support for certificate authentication d2176b1
Andrew Thompson Vagabond Implement initial security context renewal based on a global epoch af064ce
Jared Morrow jaredmorrow Add pbkdf2 as a dep e31fbf0
Jared Morrow jaredmorrow Initial pass at generating a pbkdf2 pass f0fde04
Jared Morrow jaredmorrow Fix hmac algorithm 5c0d79c
Jared Morrow jaredmorrow Move password auth into its own module 130b804
Andrew Thompson Vagabond Add ability to register to a proplist 759253d
Andrew Thompson Vagabond Support dynamically registered auth backends 8c95105
Jared Morrow jaredmorrow Updates to pw_auth and riak_core_security modules ce320ff
Jared Morrow jaredmorrow Remove unneeded case statement 7e9088e
Jared Morrow jaredmorrow Logging the plaintext password would be bad ad0c976
Jared Morrow jaredmorrow Remove extra newline 9ff4146
Andrew Thompson Vagabond WIP rebase on top of cluster metadata 63fa8ea
Andrew Thompson Vagabond More WIP cluster metadata work 6c03f2b
Andrew Thompson Vagabond Finish porting to cluster metadata 6b4e9a8
Andrew Thompson Vagabond More work on security 54e921b
Andrew Thompson Vagabond Cleanup and refactor 3571941
Andrew Thompson Vagabond Add an is_enabled function to check security status 70353d3
Andrew Thompson Vagabond Merge pull request #386 from basho/adt-security
Add security to Riak
55c1058
Commits on Sep 22, 2013
Jordan West jrwest hashtree_tree: better cleanup before creating node
extends hashtree:destroy to take a path and pass it directly to
eleveldb
9f27816
Jordan West jrwest prevent leakage of hsahtree_tree snaphots
delete the ets data if its still valid, catch the error otherwise
27292c3
Jordan West jrwest change metadata manager/hashtree to use infinity timeouts b5efcc5
Jordan West jrwest link hashtree update proc to metadata_hashtree
otherwise the caller is not notified of its crash and waits
infinitely
fae3173
Jordan West jrwest add comments to metadata iterator functions 1e7fd12
Jordan West jrwest use sha for metadata object hash instead of phash2 2416eed
Jordan West jrwest cleanup metadata exchange logging 0c28ac2
Jordan West jrwest dialyzer, y u no support opaque polymorphic types e5b3a41
Jordan West jrwest Merge branch 'jrw-cm-hashtree-rebased' into develop
Conflicts:
	rebar.config
288bf49
Jordan West jrwest update hashtree_eqc. get it passing on eqc 1.29.1
* get rid of make_treevars since the {call, erlang,'*', _} was the
cause of unknown precondition failures and the same generation can be
done without it
* remove initial_state function and just pass the rest of the
initial state to commands/2
* use pretty_commands/4 so failures are easier to track down
e187f92
Commits on Sep 23, 2013
Engel A. Sanchez engelsanchez Fix rebar.config to use {branch, "name"}
The rebar.config file currently is using "master" by itself, which does
not make rebar actually update that branch on update-deps.
7c6740e
Engel A. Sanchez engelsanchez Merge pull request #394 from basho/bugfix/use-branch-tg-in-rebar-config
Fix rebar.config to use {branch, "name"}
6fa06e6
Jordan West jrwest fix unfortunate variable naming in hashtree_tree
addresses issue #391
a158ef9
Jordan West jrwest dont trap_exit in metadata hashtree
addresses issue #389
e49c5e7
Jordan West jrwest add ability to list and cancel broadcast exchanges
addresses issue #390
005a9c7
Jordan West jrwest double eunit timeout on hashtree_eqc 647e4a6
Jordan West jrwest Merge pull request #393 from basho/jrw-hashtree-eqc-fix
update hashtree_eqc. get it passing on eqc 1.29.1
97e75ac
Commits on Sep 24, 2013
Jordan West jrwest dialyzer fixes for cluster metadata
addresses issue #392

* fix some typespecs in dvvset, riak_core_metadata and metadata manager
* change some opaque types to regular ones
* don't reuse 'remote_iterator' as message to manager process since its
a record too
* fix typo in exchange_fsm that didn't harm anything

This commit does not fix the issues found by dialyzer in
riak_core_metadata_exchange_fsm. Dialyzer isn't able to figure
out we are doing the "right thing" with the iterators here.
bf86a39
Commits on Sep 26, 2013
Chris T buddhisthead Merge pull request #387 from basho/cet-expose-caps-resolver
Expose the resolution of capabilities for general use in protocol negotiations
679efc7
Commits on Sep 27, 2013
Scott Lystig Fritchie slfritchie Fixes #189 -- include socket peer info in lager messages 05dd3ab
Scott Lystig Fritchie slfritchie Patch everything from PR #306 except riak_core_handoff_manager.erl (w…
…hich was already fixed)
429d2d6
Scott Lystig Fritchie slfritchie Merge branch 'gh306-edoc-fixes' into develop 892cc49
Commits on Sep 30, 2013
Jordan West jrwest Provide Proper Creation Semantics for Bucket Types
Some properties need to be "create-only". To supoprt this we
need to ensure that nodes only see no type or a single type. This commit
implements this invariant. See riak_core_bucket_type module documentation
for more details.

In addition to changing how bucket types are created and modified this
commit changes the interface to riak_core bucket validators. The
validate function is now passed three additional arguments: an atom
indicating if the properties are being created or updated (create is
only for bucket types), the bucket type and bucket (or undefined bucket if
just updating the type), and the previous properties being
modified. This allows validators to provide more details validations
such as preventing which properties can be set only at creation time.
ec873e5
Jordan West jrwest use max function when resolving bucket properties b4c1a2b
Commits on Oct 01, 2013
Bryan Fink beerriot allow Context to be undefined in merge/2,3
This fixes the dialyzer errors about breaking the function's contract in
riak_core_metadata_exchange_fsm. That transitively fixes the non-local
return errors in the same module.

The Context is ignored in riak_core_metadata_manager:merge/2,3, so it
could instead be defined as any(), but adding undefined seemed the least
intrusive.
e6fff5b
Jordan West jrwest Merge pull request #397 from basho/jrw-cm-dialyzer-fixes
dialyzer fixes for cluster metadata
9c2a4c8
Bryan Fink beerriot export index type from chash
riak_pipe wants to use index
40a0cf1
Bryan Fink beerriot Merge PR #406 from branch 'refactor/bwf/pipe-dialyzer2' into develop 6afbfe5
Commits on Oct 03, 2013
MatthewVon mark AAE vnodes as internal databases for leveldb flexcache accounting 9c3bb91
Commits on Oct 04, 2013
Jordan West jrwest export vsn of riak_core_ring:future_index that doesn't take ring eb967e7
Scott Lystig Fritchie slfritchie Refactor make_fold_req(), add make_newest_fold_req(), add EUnit tests
Partially addresses basho/riak#407 ... a patch
to riak_kv is also required.
21cff50
Scott Lystig Fritchie slfritchie Sort AAE differences before acting upon them via read-repair
Apply patch to hashtree.erl that was developed for
basho/riak_kv#646.
e7a4f9e
Scott Lystig Fritchie slfritchie Use inet_parse:ntoa/1 for peer addresses 7f5d0a5
Scott Lystig Fritchie slfritchie Use 'infinity' or really long timeouts instead of default 5 seconds 824508a
Scott Lystig Fritchie slfritchie Merge branch 'gh267-gen_server-call-review' into develop e1a5f31
Bryan Fink beerriot fold unnecessary make_fold_req2/4 into make_fold_req/4
also renamed make_fold_req3 to make_fold_reqv, since the 3 didn't make
sense any more, but it's still nice to have the name slightly different
than the exported functions

also fix a bit of doc
5b4d1af
Jordan West jrwest Merge pull request #410 from basho/jrw-future-index-5-export
export vsn of riak_core_ring:future_index that doesn't take ring
03ae5a8
Jordan West jrwest Merge pull request #396 from basho/jrw-metadata-hashtree-updates
address some metadata hashtree issues
2f62635
Jordan West jrwest Merge pull request #402 from basho/jrw-bucket-type-invariant
Proper Bucket Type Creation
dc44ff1
Engel A. Sanchez engelsanchez Fix eunit test failures
* Scott's accidental cast to infinity (and beyond!!)
* core_vnode_eqc was failing due to meck unload module on process death
behavior, removed with no_link flag
45807ba
Engel A. Sanchez engelsanchez Merge pull request #413 from basho/eas-fix-eunit-tests
Fix eunit test failures
276d649
Evan Vigil-McClanahan evanmcc At this point in riak's history disterl is our primary bottleneck.
Something about the way that it is framing messages is causing its
tcp connection to size its send and recv buffers much smaller than
is optimal.

This change adds code to automatically set dist port sizes on nodeup,
as well as code to manually set all disterl connections to specific
sizes, meant to be used from the console for testing and tuning.
706aaf4
Engel A. Sanchez engelsanchez Merge pull request #400 from basho/gh189-handoff-sender-info
Fixes #189 -- include socket peer info in lager messages
dc63be6
Commits on Oct 05, 2013
Engel A. Sanchez engelsanchez Fix use of removed hashtree:compare/3 1240324
Engel A. Sanchez engelsanchez Merge pull request #411 from basho/slf-aae-sort-difflist-hashtree-moved
Sort AAE differences before acting upon them via read-repair
062c208
MatthewVon mark AAE vnodes as internal databases for leveldb flexcache accounting b375e10
Matthew V Merge branch 'mv-flexcache' of github.com:basho/riak_core into mv-fle…
…xcache
ef702de
Commits on Oct 07, 2013
Scott Lystig Fritchie slfritchie Small refactorings:
* Store the send & recv buffer sizes in #state so that nodeup events
that arrive after a set_dist_buf_sizes() call will use the specified
sizes rather than OTP app environment var defaults.

* Create get_riak_env_vars/0 helper function and export it for debugging use
After refactoring further, it's only used once internally, so if YMMV,
I'm fine with inlining this function's code and removing the export.

* Add explicit buffer size args to set_port_buffers().
afa502a
Scott Lystig Fritchie slfritchie Variable renaming to match inet:setopts() option names 4a71fc0
Scott Lystig Fritchie slfritchie Merge remote-tracking branch 'origin/slf-vnode-ops-opts' into develop
Conflicts:
	src/riak_core_util.erl (fixed -SLF)
31bb456
Evan Vigil-McClanahan evanmcc - remove redundant supervisor. ec5934d
Commits on Oct 09, 2013
Steve Vinoski vinoski move ebin/riak_core.app to src/riak_core.app.src 4fca5b3
Joe DeVivo refactored riak_core bits of the schema in here, with tests 544ae58
Joe DeVivo minor changes based on PR#417 dff2f47
Joe DeVivo joedevivo Merge pull request #417 from basho/jd-cuttlefish
refactored riak_core bits of the schema in here, with tests
2ec217c
Steve Vinoski vinoski Merge pull request #416 from basho/move-to-app-src
move ebin/riak_core.app to src/riak_core.app.src
7ebe208
Commits on Oct 12, 2013
Evan Vigil-McClanahan evanmcc allow feature to be disabled 2852030
Susan Potter mbbx6spp Update otp_release list in .travis.yml
Changes:
* Removed R14* Erlang version targets from `.travis.yml`
* Added R15B* and R16B Erlang version targets to `.travis.yml`

Background:
Until this change Travis CI was building against Erlang R14 which is not
supported for Riak (KV) according to [Basho documentation]
(http://docs.basho.com/riak/latest/ops/building/installing/erlang/),
which states, *"Riak requires Erlang R15B01"*. These consistently failed
since the ring resizing change was merged.

Since `.travis.yml` was updated last (a year ago) R15B0{2-3} and R16B
were released. R16B01 and R16B02 to not appear to be offered as choices
in Travis CI yet ([according to Travis CI's documentation]
(http://about.travis-ci.org/docs/user/languages/erlang/)).
52d4a5c
Commits on Oct 14, 2013
Matthew Von-Maszewski Merge pull request #408 from basho/mv-flexcache
mark AAE vnodes as internal databases for leveldb flexcache accounting
36437b6
Joe DeVivo Default bucket properties added to riak_core 4d0274d
Jordan West jrwest fill in missing props in riak_core_bucket_type:defaults/0 4af7e9a
Joe DeVivo turns sibilings on by default 3de2ca2
Joe DeVivo removed anti_entropy from core schema. for now... b19ad3e
Joe DeVivo joedevivo Merge pull request #424 from basho/jd-cuttlefish
moved default bucket props from riak_kv
9ac2211
Joe DeVivo moved default bucket props to {level, advanced} 965cb4e
Jordan West jrwest allow_mult false -> true for bucket types 02bbb5e
Commits on Oct 16, 2013
Jared Morrow jaredmorrow Standardize on a rebar.config dep format to reduce conflicts dd63f9c
Jared Morrow jaredmorrow Update rebar to 2.1.0-166 to pick up update-deps fix 4acc290
Jared Morrow jaredmorrow Remove cuttlefish dep since riak_sysmon brings it in d17e1e1
Commits on Oct 17, 2013
Christopher Meiklejohn cmeiklejohn Fix bad type specifications.
max_nb_const should be widened to include 'undefined', considering how
all callers into this function only ever pass undefined at this point.

Also, define a new 'target' type which is used, because currently, the
type is declared as string which is incorrect and triggering dialyzer
type warnings anywhere this type is referenced.
09d68a9
Sean Cribbs seancribbs Remove protobuffs dependency from riak_core.
The riak_core.proto was only used in riak_kv's handoff, and has no
references elsewhere in riak_core. It is appropriately moving to
riak_kv.

/cc @Vagabond @evanmcc @jrwest
441abd6
Sean Cribbs seancribbs Merge pull request #382 from basho/refactor/sdc-rm-protobuffs
Remove protobuffs dependency from riak_core.
be61de3
Jordan West jrwest do not serialize through gen_server on metadata get
addresses #409
09060be
Andrew Thompson Vagabond Make load_certs function exported and more useful (works on a single …
…file, too)
ccd1a1d
Jordan West jrwest Merge pull request #423 from basho/jrw-bucket-type-defaults
fill in missing props in riak_core_bucket_type:defaults/0
de70fef
Commits on Oct 18, 2013
Jordan West jrwest Merge pull request #431 from basho/jrw-cm-get-direct
do not serialize through gen_server on metadata get
f6a5492
Andrew Thompson Vagabond Merge pull request #432 from basho/adt-pb-crl
Make load_certs function exported and more useful (works on a single file, too)
4dd84bf
Commits on Oct 19, 2013
Evan Vigil-McClanahan evanmcc merge develop, add env var docs. a3ed175
Evan Vigil-McClanahan evanmcc clean up sup code 17f7e8b
Evan Vigil-McClanahan evanmcc Merge pull request #414 from basho/feature/pevm-dist-buf-size
Increase disterl buffer sizes.
e294b98
Commits on Oct 21, 2013
Andrew Thompson Vagabond Rework table formatting to be better and output better info
Basicially tables now auto-size much better and handle multiline fields
much better, without breaking the table. The original table.erl was
something Andrew Stone had lying around, so it wasn't particularly
polished.
b6df348
Joe DeVivo joedevivo Merge pull request #425 from basho/jd-cuttlefish
GET ON THIS LEVEL
64a992c
Commits on Oct 22, 2013
Joseph Blomstedt jtuple Switch to .app.src, add xref options to rebar.config 966e53e
Joseph Blomstedt jtuple Integrate riak_core with riak_ensemble c757b3c
Joseph Blomstedt jtuple Fix/extend vnode handoff/forwarding behavior
Fix issue where handoff_cancelled was not called on vnode modules
if handoff failed to start due to concurrency limit.

Add new optional set_vnode_forwarding callback that vnode modules
can implement to track riak_core's implicit forwarding behavior
that occurs between a primary handoff finishing and actual ownership
being changed.

All of the above changes are necessary for riak_kv_vnode to properly
support strongly consistent operations during handoff and ownership
transitions.
5c812ff
Evan Vigil-McClanahan evanmcc add a simple, point to point capability check for batch message
support. if not, fall back to per-object messages.
9d24748
Evan Vigil-McClanahan evanmcc Merge pull request #436 from basho/pevm-handoff-back-compat-2.0
2.0 version of handoff backwards compatability
f19e415
Commits on Oct 23, 2013
Andrew Thompson Vagabond Merge pull request #435 from basho/adt-table-formatting
Rework table formatting to be better and output better info
53a1bb8
Joseph Blomstedt jtuple Merge branch 'jdb-strong-1.4.2' into jdb-strong-2.0-wip
Conflicts:
	rebar.config
	src/riak_core.app.src
	src/riak_core_bucket.erl
	src/riak_core_claimant.erl
	src/riak_core_sup.erl
	src/riak_core_vnode.erl
b76d686
Commits on Oct 24, 2013
Joseph Blomstedt jtuple Disable ensembles subsystem by default 98b9ad0
Commits on Oct 26, 2013
Joseph Blomstedt jtuple Switch riak_ensemble to jdb-wip-2.0 for tech preview 56bcc16
Joseph Blomstedt jtuple Merge pull request #441 from basho/jdb-strong-2.0 19aeaba
Commits on Nov 04, 2013
Dave Parfitt initial add of riak_core_bucket_types.hrl fcfae67
Commits on Nov 05, 2013
Dave Parfitt Merge pull request #445 from basho/refactor/bucket_types_include
initial add of riak_core_bucket_types.hrl
6d53f2e
Dave Parfitt cleanup conn_mgr/service_mgr eunit output f329685
Dave Parfitt Merge pull request #446 from basho/refactor/noisy_tests
cleanup conn_mgr/service_mgr eunit output
140a330
Commits on Nov 06, 2013
Andreas Hasselberg anha0825 Removed webmachine from applications that should be started. bb46dc7
Sean Cribbs seancribbs Merge pull request #447 from anha0825/develop
Removed webmachine from applications that should be started.
68c7bdf
Commits on Nov 12, 2013
Jordan West jrwest share segment store accross all nodes in hashtree_tree
this ensures that we don't create an unbounded number of segment
stores (leveldb databases), instead only creating one.
4c21b3a
Jordan West jrwest Merge pull request #433 from basho/jrw-cm-hashtree-shared-seg-store
share segment store accross all nodes in hashtree_tree
18aabc4
Commits on Nov 18, 2013
Steve Vinoski vinoski add missing apps to src/riak_kv.app.src
The basho_stats, eleveldb, pbkdf2 and poolboy apps are used in riak_core
but were not listed as application dependencies in the
src/riak_core.app.src file.
8e7e758
Christopher Meiklejohn cmeiklejohn Merge pull request #453 from basho/vinoski/app-file-additions
add missing apps to src/riak_kv.app.src
4250543
Commits on Nov 20, 2013
Heinz N. Gies Licenser Added handoff concurrency setting to schema. a4d2b51
Jared Morrow jaredmorrow Update lager dep to 2.0.1 f48c60d
Jordan West jrwest riak_core_metadata:to_list/1,2
these are convenience functions for debugging and testing, although
probably not so useful for production code
020b25a
Jordan West jrwest close metadata fold iterators upon termination
as of 0b59e4c, iterators must be
closed when they are no longer used.

addresses #450
33d6dd7
Jordan West jrwest option to prevent writes of resolved values on metadata read
a new option, {allow_put, boolean()} can be passed to get/3,
iterator/2, fold/4, or to_list/2. the default is to perform the write.

addresses #440
95b7398
Commits on Nov 21, 2013
Sean Cribbs seancribbs Merge pull request #454 from project-fifo/schema-handoff_concurrency
Added handoff concurrency setting to schema.
bf0048f
Jordan West jrwest increase broadcast's initial eager set size for larger clusters
increasing the fanout reduces the chance of requiring exchange to deliver
messages to all alive nodes during a broadcast. This patch decreases the
likelyhood by 20 to 30 percent, on average.

The chosen fanout is determined by the discrete function
`round(log(NumNodes)+1)`, where `log(x)` is the natural log and `NumNodes`
is the number of nodes in the cluster.  For practical cluster sizes,
this curve has a maximum fanout of about six peers, dependning on what
is considered the upper limit of practical cluster sizes.

addresses #438
36c8348
Jordan West jrwest enable read/write_concurrency on cluster metadata ets tables
addresses #443
99677aa
Commits on Nov 27, 2013
Scott Lystig Fritchie slfritchie Fix riak_core_tracer: call order matters with R16B02 60fc0b6
Commits on Dec 03, 2013
Sean Cribbs seancribbs Added initial vclock eqc_statem. 0dfdbe2
Sean Cribbs seancribbs eunit-ify the vclock_qc ce51f06
Sean Cribbs seancribbs Generate the model along with the concrete in gen_vclock(). 46feefb
Sean Cribbs seancribbs Make property operate on many vclocks. 8c411c2
Sean Cribbs seancribbs Add vclock:dominates/2 with property. 108326a
Sean Cribbs seancribbs Remove unused generators, clarify the collect, add an eunit timeout a…
…nd more commands.
edaec97
Sean Cribbs seancribbs Update dominates/2 to use equal/2, QC property is broken. 2bd548b
Sean Cribbs seancribbs Adjust EQC model to handle timestamps. 1933464
Commits on Dec 04, 2013
Russell Brown russelldb Add convenience funtions for running vclock eqc from console 97d3877
Sean Cribbs seancribbs Merge pull request #462 from basho/sdc-vclock-eqc
Add EQC statem property for vclock. [rebased]
b4f4d97
Commits on Dec 09, 2013
Christopher Meiklejohn cmeiklejohn Provide a synchronous registration and unregistration of services.
Some parts of repl require being able to disable a service and block
until it's disabled.  Provide this API.
25c4943
Christopher Meiklejohn cmeiklejohn Merge pull request #471 from basho/feature/csm/sync-service-mgr-develop
Provide a synchronous registration and unregistration of services.
4dc9459
Commits on Dec 11, 2013
Joe DeVivo New cuttlefish api functions 5bfe02f
Joe DeVivo joedevivo Merge pull request #478 from basho/jd-cuttlefish-api
New cuttlefish api functions
0c1d73a
Commits on Dec 12, 2013
Scott Lystig Fritchie slfritchie Merge pull request #421 from mbbx6spp/clean-up-.travis.yml
Update otp_release list in .travis.yml
a725d33
Christopher Meiklejohn cmeiklejohn Merge pull request #430 from basho/bugfix/csm/bad-type-specifications
Fix bad type specifications.
ca73d1a
Christopher Meiklejohn cmeiklejohn Extract riak_core_connection and tests.
Conflicts:
	ebin/riak_core.app
	src/riak_core_connection.erl
	test/riak_core_connection_mgr_tests.erl
	test/riak_core_connection_tests.erl
	test/riak_core_service_mgr_tests.erl
54fffde
Christopher Meiklejohn cmeiklejohn Remove connection manager and service manager.
Conflicts:
	ebin/riak_core.app
	src/riak_core_connection_mgr.erl
	src/riak_core_service_mgr.erl
83b3da4
Christopher Meiklejohn cmeiklejohn Remove ranch dependency. 85e2f22
Christopher Meiklejohn cmeiklejohn Merge pull request #480 from basho/features/csm/connection-manager
Port 1.4 changes to the 2.0 branch.
d7139c1
Commits on Dec 19, 2013
Jordan West jrwest minor cleanup in riak_core_metadata & riak_core_broadcast
* fix some whitespace
* only allow valid boolean values for allow_put get option
d166a40
Jordan West jrwest Merge pull request #457 from basho/jrw-cm-improvements
Yet Another Round of Cluster Metadata Improvements (YAROCMI #1)
00e9474
Jordan West jrwest added btypes_eqc to test bucket types creation invariants
the test does not yet cover crashing and chaning the claimant or waiting for
type properties to propogate to other nodes (for that we will probably need/use riak_test).
However, it does reproduce #442
3ac551f
Jordan West jrwest validate riak_core properties that were added as part of by bucket types
addresses #442
0d03e46
Jordan West jrwest Merge pull request #460 from basho/jrw-btype-prop-validation
Validate Bucket Type Properties Required by Core
ad8be15
Commits on Dec 20, 2013
Chris T buddhisthead Add a background lock and task manager module to riak core.
* The goal is to allow riak sub-systems to coordinate use of shared resources,
* e.g. protect from concurrent vnode folds on the same partition.
* Locks and tokens have a settable maximum concurrency limit.
* "Taken" locks and tokens are tracked in an ETS table.
* max_concurrency is returned when the set limits are reached.
* Processes that take locks are monitored.
* Locks are released when the taking processes terminate.
* Tokens are refreshed at a specified periodic rate.
* Token processes are not monitored because tokens never "release".
* A table manager is introduced to add persistence across process crashes,
* and to allow proper table transfer to occur without losing the table.
* An EQC test exercises the majority of the API. see test/bg_manager_eqc.erl
* See the original PR for background manager here: #364
779273b
Chris T buddhisthead Concurrency API should return 'undefined' for limits when the ETS tab…
…le is unavailable * Allows callers to try again later
f29014b
Chris T buddhisthead Clean up query API
* Remove history API, use unregistered to signal unavailability of bg-mgr, fix set_token_rate bug.
* Prune the query API down to a consistent set of commands and adjust tests.
* Change query API to use named ETS table directly; is not threaded through the gen_server.
4eead17
Chris T buddhisthead Re-implement handoff integration.
* rebase on latest background manager.
* Add an optional callback entry point to the kv vnode: handoff_started/2
* core_vnode will check for the optional function and call if present
* returning max_concurrency will cause the handoff to be aborted and retried.
23f9464
Chris T buddhisthead Address remaining issues in PR
* Remove lock defs file; it was moved to riak_kv_vnode.hrl since all these locks are KV-specific
* The optional call to handoff_started/2 allows vnodes one last chance to abort the handoff process. the function is passed the source vnode's partition number and node name (this node) because the callback does not have access to the full vnode state at this time. In addition the worker pid is passed so the vnode may use that information in its decision to cancel the handoff or not (e.g. get a lock on behalf of the process)
* Move code that calls handooff_started/2 into separate function and put the call inside the try/catch.
ae6b3f8
Chris T buddhisthead Move commonly used configuration switch reader function into core. 66a4f65
Chris T buddhisthead Address final PR comments. c770a72