Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

1.3 to master #519

merged 22 commits into from Apr 8, 2013


None yet
4 participants

russelldb commented Mar 26, 2013

Just to verify that the merge dealt with the conflicts correctly.

engelsanchez and others added some commits Mar 18, 2013

@engelsanchez engelsanchez Add bulk vnode start function to allow parallelism 4092945
@engelsanchez engelsanchez Merge pull request #511 from basho/eas-add-kv-vnode-parallel-init
riak_kv 1.3.1 using parallel vnode init
@engelsanchez engelsanchez support for handling legacy sext encoding of 2i keys
This commit is a squashed version of the branch:

The version of sext used by riak 1.3 and prior did not
preserve the sort order of bigints when serializing erlang
terms resulting in 2i queries that returned missing or
invalid results (riak_kv#499). The updated version of sext fixes the issue
with an incompatible change to the encoding of bigints. Special
handling of this incompatability is necessary to provide a path
for indexes to be fixed (as they are written to disk).

A utility function for reformatting the on-disk indexes (per-node)
is provided. This allows indexes to be upgraded on a running node.
The utility folds over all indexes, per partition, and rewrites any
invalid index (those containing bigints encoded using the old version
of sext). In order to safely migrate indexes, until the fold that
rewrites the indexes in the new format completes, all riak_object
puts/deletes must delete the legacy index format in addition to the new format

Upon successful completion or if the partition is empty a flag is
written to the leveldb backend to ensure that future writes/deletes
to not inccur the additional cost of deleting the legacy index format

add support for downgrading back to legacy index format

Fix eleveldb fold_objects to start at 1st object

Also fixed special bucket option so it starts at the beginning of the
given bucket. But that doesn't appear to be used anywhere...

Add key namespace for special leveldb markers

My argument is that this may make it less messy down the road if others
start adding other special markers in leveldb partitions. They now have
a separate namespace, like objects and index keys do.

add console support for reformatting indices

Fix 2i bigint eq query during transition

This causes a double scan while in the transition from the legacy
(broken for big int) sext encoding for 2i keys. Once starting at the old
key value, later at the new one. Once the upgrade or
downgrade is finished, it should return to a normal scan.

add support for multi-backend index reformatting

the multibackend supports indexing on buckets assigned to sub-backends
that support indexing. LevelDB sub-backends must be reformatted.
@engelsanchez engelsanchez Merge pull request #516 from basho/jrw-eas-2i-fix-keys-backport-squashed
support for handling legacy sext encoding of 2i keys
@jtuple jtuple Change AAE to use incremental crypto:sha calculations
The crypto library is implemented as a NIF, and long running NIFs can
negatively impact the work balancing algorithm of the Erlang
scheduler. Using normal crypto:sha(object) is unbounded in duration as
the size of an object is unbounded. This commit therefore changes AAE
to use an incremental SHA calculation (sha_init/sha_update/sha_final)
with a set chunk size (defaults to 4096; configurable app var
@jtuple jtuple Merge pull request #514 from basho/jdb-aae-incremental-sha-1.3 98aba59
@russelldb russelldb Re-use the stat calc funs in riak_core
Use the cached riak_pipe stats for legacy stats

When a broken stat is detected, register it.

Fix bug where stats endpoints were calculating _all_ riak_kv stats

Since adding many more stats, and most of the infrastructure for
ad hoc querying of stats, the stat calculation code for the
(not yet legacy) endpoints was calculating all stats for riak_kv.

As there are about (ring_size * vnode stats) + (fsm stages * fsm stats)
more stats now, this calculation, understandbly, took a long time.

This patch instead only calculates the minimum subset of stats
needed to support the (not yet legacy) stats endpoints.
@russelldb russelldb Since stats now get repaired when an update fails, log as `warning` 5d3a080
@russelldb russelldb Merge pull request #517 from basho/kv508-stats-warn
Since stats now get repaired when an update fails, log as `warning`
Jared Morrow Roll riak_kv version 1.3.1 eb24ff9
@russelldb russelldb Merge branch '1.3'
Bryan Fink spell badarg correctly
Spelled incorrectly, the catch clause fails to prevent the process from
blowing up on error. Spelled correctly, the user receives an error
message describing what the problem is.
Bryan Fink Merge pull request #522 from branch 'bwf-badard' into 1.3 8e7318d
@engelsanchez engelsanchez Fix perf problems and bug in 2i reformat
Add batch size parameter to control how many keys
are fixed at a time.
Add backpressure to query of bad idx entries to avoid
having the reformat process flooded with msgs.
Fixed problem with list keys and code triggering a second
scan during 2i reformat transition.
@engelsanchez engelsanchez Make multi backend work with batched 2i reformats
This is WIP. Testing is not conclusive on whether it works, but it runs
to completion now.
@engelsanchez engelsanchez Fix 2i reformat batch for multibackend 3e1ec43
@engelsanchez engelsanchez Make reformat-indexes print default parameters 1bfaeae
@jrwest jrwest move querying of fixed index status to seperate backend function
makes top-level fixed_indexes status on multi-backend,
which messes with stats, unncessary allowing it to be removed

engelsanchez commented on b50d1df Apr 1, 2013

The code looks good to me. I tested index reformatting with a small data set in a mixed bitcask/memory/2 leveldb setup (from 1.3.0 -> 1.3.1). Ship it...
👍 💃


engelsanchez replied Apr 1, 2013

And to be clear: the logs are not spammed by bad stat messages anymore with this applied

engelsanchez and others added some commits Apr 1, 2013


engelsanchez commented Apr 5, 2013

I just merged the latest 1.3 into this and I'm reviewing the diff to merge into master.

@engelsanchez engelsanchez added a commit that referenced this pull request Apr 8, 2013

@engelsanchez engelsanchez Merge pull request #519 from basho/1.3_to_master
1.3 to master

@engelsanchez engelsanchez merged commit 19f19a9 into master Apr 8, 2013

1 check failed

default The Travis build failed

@engelsanchez engelsanchez deleted the 1.3_to_master branch Apr 8, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment