Permalink
Branch: develop
Commits on Jul 16, 2014
  1. Check AAE throttle only when AAE is enabled

    rzezeski committed Jul 16, 2014
    Move the AAE throttle check to the actual tick function to prevent it
    from running when AAE is disabled.  Otherwise, users see AAE throttle
    messages when AAE is not enabled and that is just plain confusing.
Commits on Apr 22, 2014
  1. Fix some AAE dialyzer errors

    rzezeski committed Apr 22, 2014
    This commit fixes all the dialyzer errors for `riak_kv_entropy_info`
    as well as an error for `riak_kv_exchange_fsm`.  The main change was
    to fix the spec for `#index_info.repaired` which is actually a
    `#simple_stat{}` not a `pos_integer()`. However, even after that
    dialyzer was not happy with the explicit default values of `undefined`
    without corresponding entries in the type definition. This seemed like
    a waste to me given that Erlang will implicitly default records values
    to `undefined` and insert the appropriate type spec. This is discussed
    in the following reference manual:
    
    http://www.erlang.org/doc/reference_manual/typespec.html#typeinrecords
    
    Here is the full list of dialyzer errors which have been fixed:
    
    ```
    riak_kv_entropy_info.erl:78: Function exchange_complete/4 has no local
    return
    riak_kv_entropy_info.erl:83: Function exchange_complete/5 has no local
    return
    riak_kv_entropy_info.erl:84: The call
    riak_kv_entropy_info:update_index_info({_,_},{'exchange_complete',_,_,_})
    will never return since it differs in the 2nd argument fr
    riak_kv_entropy_info.erl:161: Record construction
    repaired::pos_integer() and last_exchange::{non_neg_inte
    riak_kv_entropy_info.erl:194: The call
    riak_kv_entropy_info:update_simple_stat(Repaired::'undefined' |
    pos_integer(),pos_integer()) will never return since it differs in
    riak_kv_entropy_info.erl:208: Function compute_exchange_info/4 has no
    local return
    riak_kv_entropy_info.erl:218: The call
    riak_kv_entropy_info:stat_tuple(Repaired::pos_integer()) will never
    return since it differs in the 1st argument from the success t
    riak_kv_entropy_info.erl:225: Function update_simple_stat/2 has no local
    return
    riak_kv_entropy_info.erl:225: The pattern <Value, 'undefined'> can never
    match the type <'undefined' | pos_integer(),pos_integer()>
    riak_kv_entropy_info.erl:227: The pattern <Value, Stat = {'simple_stat',
    _, Min, Max, Cnt, Sum}> can never match the type <'undefined' |
    pos_integer(),pos_integer()>
    riak_kv_entropy_info.erl:234: Function stat_tuple/1 has no local return
    riak_kv_entropy_info.erl:234: The pattern 'undefined' can never match
    the type pos_integer()
    riak_kv_entropy_info.erl:236: The pattern {'simple_stat', Last, Min,
    Max, Cnt, Sum} can never match the type pos_integer()
    riak_kv_exchange_fsm.erl:379: Function exchange_complete/4 has no local
    return
    ```
Commits on Mar 27, 2014
  1. Merge pull request #879 from basho/feature/rz/yz-aae-status

    rzezeski committed Mar 27, 2014
    Export AAE format functions for reuse
Commits on Mar 25, 2014
  1. Export AAE format functions for reuse

    rzezeski committed Mar 25, 2014
    Export the internal AAE format functions so that they may be reused by
    Yokozuna.
Commits on Dec 26, 2013
  1. Fix logging call

    rzezeski committed Dec 26, 2013
Commits on Dec 24, 2013
  1. Merge pull request #782 from basho/bugfix/rz/dot-js-mr

    rzezeski committed Dec 24, 2013
    Don't encode object dot for JS map-reduce
Commits on Dec 23, 2013
  1. Don't encode object dot for JS map-reduce

    rzezeski committed Dec 23, 2013
    There is no need to encode the dot for JavaScript map-reduce. It is an
    internal thing and doesn't need to be exposed.
Commits on Nov 16, 2013
  1. Merge pull request #711 from basho/feature/rz/yz-handoff-hook

    rzezeski committed Nov 16, 2013
    Add hook to Yokozuna for handoff
Commits on Nov 11, 2013
  1. Add hook to Yokozuna for handoff

    rzezeski committed Oct 25, 2013
    Yokozuna needs to hook into KV handoff in order to behave in a
    predictable manner during ownership change. Yokozuna doesn't have
    vnodes. Instead it piggy-backs the KV vnode. When an object is written
    Yokozuna's index hook is called so that Solr may be updated. When
    ownership change happens this hook will be executed for writes
    arriving at the new node because they are writes like any
    other. However in order for successful indexing the indexes and bucket
    properties must arrive at the new node first.
    
    If the indexes don't arrive before the ownership handoff objects then
    those objects will fail to index and cause incorrect query results for
    the user. If the bucket properties don't arrive before then the index
    hook won't be invoked at all and index entries will again be
    missing. Currently Yokozuna uses some tricky code to detect a
    handoff-based write and do a busy wait for the index to appear. This
    is undesirable because it blocks the vnode. It also doesn't work for
    the new bucket properties based on cluster metadata since they are no
    longer stored in the ring.
    
    Rather than create more hacks this patch adds a handoff hook to
    Yokozuna. This way Yokozuna can make sure the indexes and properties
    are sync'd before KV is allowed to send its data. This removes the
    race on the new node and will allow removal of the busy-wait hack
    currently in the vnode write path.
Commits on Sep 25, 2013
  1. Merge pull request #664 from basho/feature/bwf/mr-switch

    rzezeski committed Sep 25, 2013
    Use yokozuna or riak_search for "search" MapReduce inputs
  2. Simply treat Bucket as Index

    rzezeski committed Sep 25, 2013
    Abondon the idea of having an implicit lookup from bucket to index
    when using Yokozuna to drive search input to map-reduce.
    
    This is a very long-winded commit message because the subject matter
    is confusing.  Scroll to end for TL;DR.
    
    Riak Search and Yokozuna can both be used as engines to produces
    results for a search input to a map-reduce job.  For example, in
    pre-Yokozuna days if a user specified the following HTTP JSON
    map-reduce job Riak Search would be used to run a query against the
    "bucket" foo.
    
    "inputs":{"bucket":"foo", "query":"bar"}
    
    Calling it bucket is really a misnomer because what is being searched
    is actually an _index_ of bucket foo.  However, since Riak Search
    forced a 1:1 mapping from bucket to index name there wasn't much of a
    difference.  The possibility of a M:1 bucket to index in the future
    was never considered.
    
    Fast forward to today.  There are two search systems in Riak now.
    Yokozuna will eventually replace Riak Search but for the time being
    there will be migrations from one to the other.  Furthermore, Search
    APIs have already been exposed that assumed the functionality of Riak
    Search, which is a small subset of Yokozuna.  Therefore, Yokozuna must
    work around these weird cases.
    
    In this case, passing the previous HTTP JSON map-reduce to Yokozua,
    the user might expect 1 of 2 behaviors.
    
    1. A bucket has an associated index.  Get the index from the mapping
    and run the query against that.
    
    2. Since the input should have been an index name to being with, and
    1:1 mapping was just a coincidence in Riak Search, then simply treat
    the 'bucket' input as the index name to search against.
    
    The first option might seem like the most obvious but what if the user
    decided to index multiple buckets under the same index?  The results
    would now include results from other buckets.  An implicit filter
    could be added to the query but we've started introducing more
    implicit (magical) behavior.  Furthermore, under the covers, it
    complicates the code.  For security purposes do we check for
    permission on the bucket, on both?  But even if that wasn't enough,
    things get EVEN STRANGER when you consider that you can use 2i as
    input to a map-reduce job with the following JSON.
    
    "inputs":{"bucket":"foo", "index":"field1_bin", "key":"bar"}
    
    Hey, I know, we could overload this "inputs" field even more and
    covert the following into a search request.
    
    "inputs":{"index":"foo", "query":"bar"}
    
    But if we chose option #1 from above we have a problem here.  The
    Yokozuna code needs to know if the incoming search request is using an
    index name or a bucket name.  If the later if needs to lookup the
    index to query.  But in order to do this the Riak KV code needs to
    create a new data structure to pass as the "Bucket" parameter so that
    the Yokozuna callback knows which is which.  Does this sound very
    confusing?  I hope so, because it is!
    
    So after a long talk with Bryan Fink we decided to stop trying to be
    cute and just face the fact that "bucket" was a poor name for the
    input structure.  Just pretend it is actually "index" and Yokozuna
    will treat it as such.
    
    TL;DR;
    
    If you are migrating from Riak Search then you should make sure to
    keep a 1:1 mapping for bucket to index if you don't want to change
    your map-reduce input.
    
    If you are starting fresh then just use the new JSON format feed
    map-reduce with Yokozuna.
    
    "inputs":{"index":"foo", "query":"field1:bar", "filter"field2:baz"}
    
    (Note that "filter" is optional)
Commits on Sep 24, 2013
  1. Fix permission for map-reduce/search

    rzezeski committed Sep 24, 2013
    The case statement was erroneously re-using the function parameter
    `_Query` as an ignored variable in two of the case clases.  The
    `_Query` function parameter and the `_Query` in the search tuple are
    two different things and thus will never match.
Commits on Sep 7, 2013
  1. Merge pull request #654 from basho/feature/rz/generic-aae-status

    rzezeski committed Sep 7, 2013
    Generic AAE Status
  2. Add default type for tree_built

    rzezeski committed Sep 7, 2013
    Use default type of `riak_kv` when calling `tree_built/2`.
Commits on Sep 6, 2013
  1. Merge pull request #653 from basho/feature/rz/yz-hook

    rzezeski committed Sep 6, 2013
    Add Yokozuna index hook
  2. Make sure AAE status is backwards compatible

    rzezeski committed Sep 6, 2013
    Add functions which provide a default `Type` so that the riak_kv calls
    can stay as they currently are.
  3. Fix AAE filtering

    rzezeski committed Sep 6, 2013
    Fix the filtering logic which removes AAE info for indices no longer
    owned by the local node.  The key is now `{Type, Index}`.
  4. Parametrize mod:fun to determine all exchanges

    rzezeski committed Jan 10, 2013
    The exchanges that need to take place in Yokozuna are not the same as
    those in KV.
  5. Export get_build_time

    rzezeski committed Jan 10, 2013
    This function can be reused by Yokozuna.
  6. Add type to entropy info

    rzezeski committed Jan 10, 2013
    Adding a type namespace to entropy info allows me to reuse it for
    Yokozuna AAE status as well as future uses of AAE.
Commits on Sep 5, 2013
  1. Add Yokozuna index hook

    rzezeski committed Sep 5, 2013
    Add a low-level hook in the vnode for the purposes of indexing objects
    in Yokozuna.  This hook provides a notification to Yokozuna _any time_
    an object's bytes are modified on disk.
Commits on Aug 9, 2013
  1. Merge pull request #608 from basho/sbv-fix-backend-tests

    rzezeski committed Aug 9, 2013
    fix riak_kv_backend standard tests
Commits on Aug 1, 2013
  1. Use riak_pipe 1.4.1

    rzezeski committed Aug 1, 2013
  2. Roll version 1.4.1

    rzezeski committed Aug 1, 2013
  3. Use 1.4.1 deps

    rzezeski committed Aug 1, 2013
Commits on Jul 12, 2013
  1. Merge pull request #603 from basho/rz-remove-authors

    rzezeski committed Jul 12, 2013
    Remove -author attributes
  2. Remove -author attributes

    rzezeski committed Jul 12, 2013
    They aren't kept current and the same information can be ascertained
    from git.
Commits on Nov 21, 2012
  1. Fix hashtree_eqc

    rzezeski committed Nov 21, 2012
  2. Fix build_tokens spec

    rzezeski committed Nov 21, 2012
Commits on Aug 24, 2012
  1. Rename gen_server2 to riak_core_gen_server

    rzezeski committed Aug 24, 2012
Commits on Jun 22, 2012
  1. Merge branch 'rz-fix-tests'

    rzezeski committed Jun 22, 2012
Commits on Jun 20, 2012
  1. Start folsom so stat system will work during tests

    rzezeski committed Jun 20, 2012
    Now that our underlying stats system is based on folsom we need to
    make sure the application is started and the stats are registered.
Commits on Jun 19, 2012
Commits on Jun 11, 2012
  1. Factor general filter code into Core

    rzezeski committed Jun 11, 2012
    Factor out the repair filter code shared by Search & KV into Core.