Update to latest riak_ensemble integrity approach #1002
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
riak_ensemble now supports an integrated integrity checking mechanism, and therefore no longer requires nor supports backend based trust and syncing logic. This commit removes all such logic from riak_kv.
The previous approach for consistent AAE was designed to function as the backend syncing mechanism for K/V ensembles. Now that riak_ensemble no longer requires backend syncing, this approach is no longer necessary nor is it correct. This commit completely removes the previous approach, reducing consistent AAE to simple read repair now that riak_ensemble has a read repair mechanism. This not only simplifies things, but this approach also properly works with the new integrated integrity/validation logic in riak_ensemble.
This commit adds the new synctree_path callback to the K/V ensemble backend. This callback is used to override how riak_ensemble maps peer synctrees to on-disk LevelDB instances. This commit implements the callback using a M:N mapping, where all peers that correspond to a single vnode map to a common LevelDB instance. This makes the number of synctree LevelDB instances scale with the size of the ring, rather than the number of distinct preflists. This is the same approach used for AAE in hashtree.erl. Requiring fewer instances makes better use of file handles, write buffers, and internal LevelDB caches.
Update riak_kv_ensemble_console to no longer talk about trust or AAE, as backend based trust and AAE syncing has been completely removed. The console output now references concepts relevant to the new synctree approach directly integrated into riak_ensemble. Remove the trust setting from riak_kv.schema since the setting no longer exists. Providing schema mappings for the new relevant settings is left for a future commit.
|
Concerning the builder failure, this pull-request will fail until the sibling |
| @@ -386,17 +399,11 @@ print_variations() -> | |||
| ensemble_overview(Details2), | |||
| io:format("~n~n"), | |||
|
|
|||
| %% Enabled, enough nodes, but AAE not enabled | |||
| io:format("(*** AAE disabled when required ***)~n~n"), | |||
| Details3 = Details2#details{nodes=['dev1@127.0.0.1', | |||
There was a problem hiding this comment.
Poor Details3 got lost in the shuffle.
There was a problem hiding this comment.
Aye, easier to just kill it then renumber all the other lines. Keeping diff minimal/etc.
andrewjstone
added a commit
to basho/riak_test
that referenced
this pull request
Jul 11, 2014
Include {error, <<"failed">>} as allowed failure so that test passes
with changes for basho/riak_ensemble#37 and basho/riak_kv#1002
andrewjstone
added a commit
to basho/riak_test
that referenced
this pull request
Jul 11, 2014
Allow {error, <<"failed">>} as an error response in ensemble_sync. Fixes
the test with basho/riak_ensemble#37 and basho/riak_kv#1002
|
👍 1519686 |
borshop
added a commit
that referenced
this pull request
Jul 11, 2014
Update to latest riak_ensemble integrity approach Reviewed-by: andrewjstone
borshop
added a commit
that referenced
this pull request
Jul 12, 2014
Update to latest riak_ensemble integrity approach Reviewed-by: andrewjstone
borshop
added a commit
that referenced
this pull request
Jul 12, 2014
Update to latest riak_ensemble integrity approach Reviewed-by: andrewjstone
borshop
added a commit
that referenced
this pull request
Jul 12, 2014
Update to latest riak_ensemble integrity approach Reviewed-by: andrewjstone
The riak_kv.consistency_trust setting has been removed, therefore this commit updates riak_kv_schema_tests to no longer test for this setting.
borshop
added a commit
that referenced
this pull request
Jul 12, 2014
Update to latest riak_ensemble integrity approach Reviewed-by: andrewjstone
|
@borshop merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the sibling pull-request for basho/riak_ensemble#37. That pull-request completely changes how
riak_ensembleperforms data validation and integrity checking, completely removing the need for backend based trust/syncing.This pull-request updates
riak_kvto work with this new approach. For the most part, that means deleting a bunch of code that is no longer necessary. As well as updating theensemble_statuscommand to no longer reference AAE or trust -- since neither of those concepts apply anymore.For more details, look at the individual commits.
_Note:_ a discussion with product management needs to happen concerning the new output of
ensemble_statusas well as how we should expose the new integrity checking settings toriak.conf. The old discussion abouttrustno longer applies since that setting has been removed. I think the approach as implemented in this pull-request is good enough for 2.0 RC But, we need to have that discussion / change things before 2.0 final.