Skip to content
Commits on Jan 10, 2014
  1. @evanmcc
Commits on Jan 7, 2014
  1. @seancribbs

    Merge pull request #130 from liveforeverx/fix_expiry

    fix bitcask.schema for multi_backend
    seancribbs committed Jan 7, 2014
  2. @liveforeverx
Commits on Jan 3, 2014
  1. @evanmcc

    - start with scott's commit 0d3acb0

    - fix the issue introduced in 6c97173 more mininally
    - ifdef out env reading even though it doesn't matter
    evanmcc committed Jan 2, 2014
Commits on Dec 26, 2013
  1. @slfritchie

    Merge pull request #127 from basho/refactor/confbal/schema-renames

    Rename/refactor a bunch of cuttlefish settings.
    slfritchie committed Dec 26, 2013
Commits on Dec 15, 2013
  1. @slfritchie
  2. @slfritchie

    Correct usage of ?SHRINK()

    slfritchie committed Dec 15, 2013
Commits on Dec 14, 2013
  1. @seancribbs
Commits on Dec 13, 2013
  1. @seancribbs

    [confbal] Rename and redesign a bunch of cuttlefish settings, improve…

    … docs.
    
    * Setting name is removed from all docstrings. The name can be
      generated from the schema and will be displayed when using the
      "describe" cuttlefish command.
    * sync_strategy -> sync.strategy; sync_interval -> sync.interval
    * merge_window -> merge.policy -- There is no window if set to
      'always' or 'never', so "policy" seems a better name.
    * merge_window.start/end -> merge.window.start/end
    * frag_merge_trigger -> merge.triggers.fragmentation (more
      descriptive). The "is_percentage" validator was also added to
      constrain the values.
    * dead_bytes_merge_trigger -> merge.triggers.dead_bytes (similar to
      frag_merge_trigger).
    * frag_threshold -> thresholds.fragmentation (again adding
      "is_percentage" validator)
    * dead_bytes_threshold -> thresholds.dead_bytes
    * small_file_threshold -> thresholds.small_file
    * max_fold_age -> fold.max_age. Added the possible value of
      `unlimited` (replaces -1 magic value) as well as a duration in
      milliseconds. The input value is upscaled to microseconds, but we
      have no evidence of anyone setting this setting. Probably best to
      change it in advanced.config.
    * max_fold_puts -> fold.max_puts. Added value of `unlimited` which
      replaces the magic value of -1.
    * Added `off` to expiry, which disables the feature (instead of the -1
      magic value).
    * require_hint_crc -> hintfile_checksums. Changed settings to `strict`
      and `allow_missing` instead of true/false.
    * expiry_grace_time -> expiry.grace_time
    * multi_backend settings were converted to use @see tags instead of
      copying the docstrings.
    J4/CV/JD/SC committed with seancribbs Dec 13, 2013
  2. @slfritchie

    Fix (I hope) PULSE model nondeterminism

    I should have checked this list of side-effect functions
    much earlier, but ... EBRAINTOOSMALL.  I hope that these
    changes will help avoid the evil nondeterminism that I've
    seen with shrinking PULSE model counterexamples.
    slfritchie committed Dec 13, 2013
Commits on Dec 12, 2013
  1. @slfritchie

    Merge pull request #124 from basho/feature/pulse-makefile-target

    Add a Makefile target for pulse tests
    slfritchie committed Dec 11, 2013
  2. @slfritchie
  3. @slfritchie

    Merge pull request #123 from basho/refactor/fileops-fold-keys

    Refactor bitcask_fileops:fold_keys function
    slfritchie committed Dec 11, 2013
Commits on Dec 11, 2013
  1. @joedevivo

    Merge pull request #125 from basho/jd-cuttlefish-api

    Updated schema for new cuttlefish api
    joedevivo committed Dec 11, 2013
  2. Updated schema for new cuttlefish api

    Joe DeVivo committed Dec 11, 2013
Commits on Dec 10, 2013
  1. @kellymclaughlin
  2. @kellymclaughlin

    Add a Makefile target for pulse tests

    Add a Makefile target to run the pulse tests. The goal is to make it
    as simple as possible to run these tests in order to increase the
    likelihood they are run during the normal course of code reviews and
    not simply forgotten. The pulse test modules are moved up directly
    into the test directory to allow rebar to find them and the test
    module is renamed to bitcask_pulse for easier identification.  Also a
    new version of rebar that can properly execute the new Makefile target
    is added.
    kellymclaughlin committed Nov 21, 2013
Commits on Dec 2, 2013
  1. @slfritchie
  2. @slfritchie
Commits on Nov 27, 2013
  1. @slfritchie
Commits on Nov 26, 2013
  1. @slfritchie

    Work-around model limitation/bug in PULSE test, bitcask_eqc.erl

    In the C65 counterexample below, the {var,2} worker proc tries to
    get key #19 and allegedly fails.  The last step of the main proc is
    a put loop of keys #1 - #34.  The {var,2} worker is racing with
    both the main proc and also with the {var,7} worker that is
    doing a fold.  The fold happens first, which freezes the keydir
    when the next put happens.  Then the {var,2} worker does its get
    and finds (correctly) `not_found` ... but the model is buggy and
    can't predict the correct answer.
    
    I'm not good enough at the temporal logic to fix the model
    The Correct Way.  So I've patched it up so that any 'bad' thing
    detected by the main/1st proc is truly bad.  But if a 'bad' thing
    happens in a forked worker, then if we know that there's a fold
    happening at the same time, we assume that the model's predicted
    answer is bad and thus we *ignore* the 'bad' thing.
    
        191> C65 = binary_to_term(element(2,file:read_file("zzz.c65@develop@commit=a2d75.bin"))).
        [[{set,{var,2},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,8},{not_call,bitcask_eqc,bc_open,[false]}}]]}},
          {set,{var,4},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,2},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,3},
                            {not_call,bitcask_eqc,bc_close,[{not_var,2}]}},
                       {set,{not_var,9},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,10},
                            {not_call,bitcask_eqc,get,[{not_var,9},19]}}]]}},
          {set,{var,6},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,3},{not_call,bitcask_eqc,bc_open,[false]}}]]}},
          {set,{var,7},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,5},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,7},
                            {not_call,bitcask_eqc,fold,[{not_var,5}]}}]]}},
          {set,{var,12},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,3},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,8},
                            {not_call,bitcask_eqc,fold,[{not_var,3}]}}]]}},
          {set,{var,17},{call,bitcask_eqc,bc_open,[true]}},
          {set,{var,28},{call,bitcask_eqc,put,[{var,17},1,<<>>]}},
          {set,{var,29},{call,bitcask_eqc,needs_merge,[{var,17}]}},
          {set,{var,30},{call,bitcask_eqc,fork_merge,[{var,17}]}},
          {set,{var,32},
               {call,bitcask_eqc,puts,[{var,17},{1,34},<<>>]}}],
         {77863,46676,48146},
         [{events,[]}]]
    slfritchie committed Nov 26, 2013
Commits on Nov 25, 2013
  1. @slfritchie
  2. @slfritchie

    Fix model problem in bitcask_qc_expiry.erl

    Here's a failing test case:
    
        [{[{delete,<<1>>,<<>>},{put,<<0>>,<<>>}],10,1,5,5}]
    
    ... and the result:
    
                          {expected,false},
                          {value,
                              {true, [blah blah...]
    
    It's just too damn difficult to model needs_merge() when there's
    so little data written.  So, we use the ?IMPLIES to check that
    there are more than 2 'put' operations.
    slfritchie committed Nov 25, 2013
  3. @slfritchie

    Tentative fix for PULSE model: kill vs. stuck merge lock

    If the EQC PULSE model kills a subprocess that is in the middle
    of opening the bitcask and has obtained the merge lock but not
    yet released it when it is killed, then subsequent attempts to
    open the cask will fail because the merge lock cannot be detected
    as stale.
    
    Failing test case:
    
        [[{set,{var,5},{call,bitcask_eqc,bc_open,[true]}},
          {set,{var,8},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,2},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,3},
                            {not_call,bitcask_eqc,fold,[{not_var,2}]}}]]}},
          {set,{var,15},{call,bitcask_eqc,fork_merge,[{var,5}]}},
          {set,{var,17},{call,bitcask_eqc,kill,[{var,8}]}},
          {set,{var,19},{call,bitcask_eqc,bc_close,[{var,5}]}},
          {set,{var,22},{call,bitcask_eqc,bc_open,[true]}}],
         {22143,93849,96204}]
    
    Failing execution:
    
            [{11419,{call,<0.27681.87>,{open,true}}},
             {21034,{result,<0.27681.87>,#Ref<0.0.359.29314>}},
             {21076,{call,<0.27681.87>,fork}},
             {21095,{result,<0.27681.87>,<0.27834.87>}},
             {21204,{call,<0.27681.87>,{fork_merge,#Ref<0.0.359.29314>}}},
             {21396,{call,<0.27834.87>,{open,false}}},
             {21619,{result,<0.27681.87>,not_needed}},
             {21650,{call,<0.27681.87>,{kill,<0.27834.87>}}},
             {21682,{result,<0.27681.87>,true}},
             {21700,{call,<0.27681.87>,{close,#Ref<0.0.359.29314>}}},
             {21743,{result,<0.27681.87>,ok}},
             {21768,{call,<0.27681.87>,{open,true}}},
    
    This patch introduces a hack that is only used during the bitcask_eqc
    test, via a "magic" shared process dictionary key.  If that key
    exists, then if a `poll_for_merge_lock()` attempt fails due to a stale
    lock, we will allow the operation to pseudo-succeed and permit the
    test case to continue.
    slfritchie committed Nov 25, 2013
  4. @slfritchie

    Pass read-write mode to keydir_get_int: permit read-write mode to alw…

    …ays see latest mutations
    
    Work around problems like this:
    
        1> C30 = binary_to_term(element(2,file:read_file("zzz.c30@develop@commit=dacf8f0630.bin"))).
        [[{set,{var,3},
               {call,bitcask_eqc,fork,
                     [[{init,{state,undefined,false,false,[]}},
                       {set,{not_var,2},{not_call,bitcask_eqc,bc_open,[false]}},
                       {set,{not_var,7},
                            {not_call,bitcask_eqc,fold,[{not_var,2}]}}]]}},
          {set,{var,6},{call,bitcask_eqc,bc_open,[true]}},
          {set,{var,7},{call,bitcask_eqc,put,[{var,6},1,<<>>]}},
          {set,{var,10},{call,bitcask_eqc,get,[{var,6},1]}}],
         {90598,47026,49786},
         [{events,[]}]]
    
    This looks like a very simple test case.  It is.  Here's an interleaved
    execution that shows a model failure.
    
        2> eqc:check(eqc:testing_time(15*60, bitcask_eqc:prop_pulse(local,false)), C30).
        Starting Quviq QuickCheck version 1.30.2
           (compiled at {{2013,11,18},{14,34,14}})
        Licence for Basho reserved until {{2013,11,23},{10,19,48}}
        Failed!
        [{set,{var,3},
              {call,bitcask_eqc,fork,
                    [[{init,{state,undefined,false,false,[]}},
                      {set,{not_var,2},{not_call,bitcask_eqc,bc_open,[false]}},
                      {set,{not_var,7},{not_call,bitcask_eqc,fold,[{not_var,2}]}}]]}},
         {set,{var,6},{call,bitcask_eqc,bc_open,[true]}},
         {set,{var,7},{call,bitcask_eqc,put,[{var,6},1,<<>>]}},
         {set,{var,10},{call,bitcask_eqc,get,[{var,6},1]}}]
        {90598,47026,49786}
    
        State: {state,#Ref<0.0.0.520>,true,false,[<0.169.0>]}
        events: failed
        Events:
        [{0,23208,[]},
         {23208,23209,[{call,<0.161.0>,fork}]},
         {23209,23225,[]},
         {23225,23226,[{result,<0.161.0>,<0.169.0>}]},
         {23226,23351,[]},
         {23351,23352,[{call,<0.161.0>,{open,true}}]},
         {23352,28804,[]},
         {28804,28805,[{call,<0.169.0>,{open,false}}]},
         {28805,41666,[]},
         {41666,41667,[{result,<0.161.0>,#Ref<0.0.0.520>}]},
         {41667,41694,[]},
         {41694,41695,[{call,<0.161.0>,{put,#Ref<0.0.0.520>,1,<<>>}}]},
         {41695,48172,[]},
         {48172,48173,[{result,<0.169.0>,#Ref<0.0.0.545>}]},
         {48173,48209,[]},
         {48209,48210,[{call,<0.169.0>,{fold,#Ref<0.0.0.545>}}]},
         {48210,48413,[]},
         {48413,48414,[{result,<0.161.0>,ok}]},
         {48414,48445,[]},
         {48445,48446,[{call,<0.161.0>,{get,#Ref<0.0.0.520>,1}}]},
         {48446,48453,[]},
         {48453,48454,[{result,<0.161.0>,not_found}]},
         {48454,49098,[]},
         {49098,49099,[{result,<0.169.0>,[]}]},
         {49099,49123,[]},
         {49123,49124,[{call,<0.169.0>,{close,#Ref<0.0.0.545>}}]},
         {49124,49155,[]},
         {49155,49156,[{result,<0.169.0>,ok}]},
         {49156,infinity,[]}]
        Bad:
        [{0,48453,[]},{48453,infinity,[{bad,{get,1,[<<>>],not_found}}]}]
        false
    
    At time 48453, we got a `get` result where we expected to find
    a `<<>>` but instead got a `not_found`.  The `fold` operation is
    interleaved at just the proper timing to trigger a keydir freezing,
    which causes an unexpected result.
    
    The patches in this commit add a read/write flag to the `keydir_get_int`
    NIF function.  If the `keydir_get_int` being performed is by the one
    process that has the bitcask open for read/write mode, then we should
    ignore all snapshots and be able to get the latest-and-greatest mutation
    of the keydir.  The diff is enlarged/cluttered by small patches to fix
    up other test cases: to add the appropriate new argument where needed.
    slfritchie committed Nov 22, 2013
Commits on Nov 22, 2013
  1. @slfritchie
  2. @slfritchie
Commits on Nov 21, 2013
  1. @kellymclaughlin

    Refactor bitcask_fileops:fold_keys function

    Refactor bitcask_fileops:fold_keys to remove nested case statements
    and improve the clarity. Also change the has_valid_hintfile function
    to only check for a valid hintfile and not also check for the
    existence of a hintfile. This change has the side effect of preventing
    the logging of error messages about invalid hintfiles when the case is
    just that no hintfile exists.
    kellymclaughlin committed Nov 21, 2013
  2. @slfritchie
Commits on Nov 20, 2013
  1. @evanmcc

    - retry create lock acquistion with delay

    - remove double close in merge fold
    evanmcc committed Nov 19, 2013
  2. @evanmcc
Commits on Nov 19, 2013
  1. @evanmcc

    rework merge worker to do use an ad-hoc bounded queue rather than

    erlang's default queue implementation.
    
    also hibernate the process when a queue item is removed to expedite
    GCing and resizing the heap.
    evanmcc committed Nov 19, 2013
  2. @evanmcc

    Merge pull request #119 from Licenser/patch-1

    added erlang file header.
    evanmcc committed Nov 19, 2013
Commits on Nov 18, 2013
  1. @evanmcc

    Merge pull request #118 from basho/refactor/pevm-avoid-file-server

    avoid file server
    evanmcc committed Nov 18, 2013
  2. @evanmcc

    alter some specs to get dialyzer cleaner, but leave one in as it is

    a) not needed, and b) changes code.
    evanmcc committed Nov 18, 2013
Something went wrong with that request. Please try again.