Skip to content
This repository
branch: develop

Jul 11, 2014

  1. Scott Lystig Fritchie

    Make EUnit happy by skipping generic_qc_fsm.erl tests, due to faulter…

    …l dependency
    slfritchie authored
  2. Scott Lystig Fritchie

    Remove faulterl from rebar.config to avoid Basho builder boxen not ha…

    …ving XML2 dev package
    slfritchie authored

Jul 10, 2014

  1. Bishop Bors

    Merge pull request #179 from basho/feature/tools.mk-0.5.5

    Update tools.mk to v0.5.5
    
    Reviewed-by: reiddraper
    borshop authored
  2. Reid Draper

    Ignore log/ directory

    reiddraper authored
  3. Reid Draper

    Update tools.mk to v0.5.5

    This also disregards the bitcask-specific tools.mk change made in
    755cf48. Per @seancribbs advice, we
    simply add a prerequisite to the `test` target, in the form of
    `eunit_nif`. The `erlang` version runs because we globally set
    BITCASK_IO_MODE=erlang. Testing this with `make -n test`, we see:
    
        ./rebar compile
        ./rebar get-deps
        BITCASK_IO_MODE="nif" ./rebar skip_deps=true eunit
        ./rebar eunit skip_deps=true
    reiddraper authored

Jul 08, 2014

  1. Bishop Bors

    Merge pull request #175 from basho/bugfix/fold-open-delete-race

    Bugfix/fold open delete race
    
    Reviewed-by: slfritchie
    borshop authored
  2. Bishop Bors

    Merge pull request #170 from basho/bug/deferred-delete-Cd8

    Deferred delete bug (aka Cd8)
    
    Reviewed-by: engelsanchez
    borshop authored

Jul 07, 2014

  1. Bishop Bors

    Merge pull request #177 from basho/bugfix/epoch-check-when-pending

    Fix epoch comparison by find_keydir_entry() when keydir->pending != NULL
    
    Reviewed-by: engelsanchez
    borshop authored

Jul 05, 2014

  1. Scott Lystig Fritchie

    Fix epoch comparison by find_keydir_entry() when keydir->pending != NULL

    When the keydir is truly frozen, i.e., when keydir->pending != NULL,
    then the comparison of the desired epoch must be greater-than-or-equal.
    This fix matches the epoch comparison that is performed by the
    `while (s != NULL)` loop in `proxy_kd_entry_at_epoch()`.
    
    I originally found this bug using a merge of the
    `bug/deferred-delete-Cd8origin` and `bugfix/fold-open-delete-race`
    branches at commits ac71627 and c8c0acc, respectively.
    
        Cn4 = [[{set,{var,1},{call,bitcask_pulse,fork,[[{init,{state,undefined,false,false,[]}},{set,{not_var,9},{not_call,bitcask_pulse,bc_open,[false,{true,{102,47,431},15}]}},{set,{not_var,15},{not_call,bitcask_pulse,fold,[{not_var,9}]}},{set,{not_var,16},{not_call,bitcask_pulse,fold,[{not_var,9}]}}]]}},{set,{var,2},{call,bitcask_pulse,bc_open,[true,{true,{351,137,29},75}]}},{set,{var,4},{call,bitcask_pulse,put,[{var,2},1,<<51,170,22,66,176,75,118>>]}},{set,{var,5},{call,bitcask_pulse,bc_close,[{var,2}]}},{set,{var,6},{call,bitcask_pulse,fork,[[{init,{state,undefined,false,false,[]}},{set,{not_var,14},{not_call,bitcask_pulse,bc_open,[false,{true,{133,280,435},32}]}},{set,{not_var,26},{not_call,bitcask_pulse,fold,[{not_var,14}]}},{set,{not_var,27},{not_call,bitcask_pulse,fold,[{not_var,14}]}}]]}},{set,{var,7},{call,bitcask_pulse,bc_open,[true,{true,{430,377,266},86}]}},{set,{var,8},{call,bitcask_pulse,merge,[{var,7}]}},{set,{var,18},{call,bitcask_pulse,put,[{var,7},2,<<0,0,0,0,0,0,0,0,0>>]}},{set,{var,20},{call,bitcask_pulse,merge,[{var,7}]}},{set,{var,23},{call,bitcask_pulse,fold_keys,[{var,7}]}},{set,{var,30},{call,bitcask_pulse,merge,[{var,7}]}}],{61218,16404,17227},[{events,[]}]].
        [begin Now = {1404,524363,267720}, io:format("Now ~p ", [Now]), true = eqc:check(eqc:testing_time(24*3600, bitcask_pulse:prop_pulse()), [lists:nth(1, Cn4), Now, lists:nth(3, Cn4)]) end || _ <- lists:seq(1,5)].
        [begin Now = now(), io:format("Now ~p ", [Now]), true = eqc:check(eqc:testing_time(24*3600, bitcask_pulse:prop_pulse()), [lists:nth(1, Cn4), Now, lists:nth(3, Cn4)]) end || _ <- lists:seq(1,550)].
    
    ... which results in a fold by the 3rd pid (i.e. the 2nd forked pid)
    to find only 1 of the two keys: `<<"kk01">>` is wrongly skipped by the
    fold.
    
    A merge prior to the start of the bad fold takes place during another
    fold.  That merge copies the only key, `<<"kk01">>`, forward during
    epoch 9 ... but that mutation causes the keydir to freeze, so now
    keydir->pending != NULL.  When the 3rd pid performs its fold, it is
    also in epoch 9.  When folding over the new file, the bug uses a
    strictly greater than comparison, so the correct entry in the pending
    part of the keydir is skipped, and so fold's get() finds the
    old/prior-to-merge entry in the keydir ... and that old entry
    does not correspond to the the current fold file and so is skipped.
    The fold (correctly) never opens the old/prior-to-merge file, so the
    `<<"kk01">>` never appears in the fold's results.
    slfritchie authored
  2. Scott Lystig Fritchie

    Change 'the_biggest += 100' to 'the_biggest++'

    slfritchie authored

Jul 04, 2014

  1. Scott Lystig Fritchie

    Raise fork_results() timeout 1000x to avoid false timeout errors from…

    … commit ee05a89
    slfritchie authored

Jul 02, 2014

  1. Engel A. Sanchez

    Avoid creating file stat on deleted files

    Because now we have code appending tombstones to potentially deleted files,
    file stats entries were occasionally created on already deleted files.
    Now the update_fstats call has a flag to signal when it is OK to create
    a file stat entry if it doesn't exist already.
    engelsanchez authored
  2. Scott Lystig Fritchie

    Fix Dialyzer warning

    slfritchie authored
  3. Scott Lystig Fritchie

    Add interfere_with_pulse_test_only() to aid PULSE in finding issue #174

    …'s race
    slfritchie authored
  4. Engel A. Sanchez

    Avoid retrying to open missing files on folds

    The fix to the fold open/delete file race made folds fail for a long
    time if a file was deleted from the cask. This fix makes it so that we
    detect that and avoid trying to re-open a file that has gone truly
    missing again and again.
    engelsanchez authored
  5. Scott Lystig Fritchie

    Fix for Dialyzer complaints related to change in keydir_info() spec

    slfritchie authored

Jul 01, 2014

  1. Engel A. Sanchez

    Fix fold file open/delete race

    This fixes the race between a fold getting an epoch and opening the
    files in the Bitcask directory.  If between those two actions a file
    needed for that snapshot is deleted by a merge, the fold will return
    incorrect data.
    
    A fold will now fetch the list of files that it needs to open and the
    epoch to use to access a snapshot of data with one operation. It will
    then try to open the files, and if it fails try again with a new set of
    files and a new epoch.
    
    One question remains open: Now if someone deletes a file from under
    Bitcask, the fold will fail indefinitely. Before, it would eventually
    proceed and return whatever data is still left. Obviously we'll need to
    let it do something similar, falling back to returning partial data.
    engelsanchez authored
  2. Scott Lystig Fritchie

    Preserve biggest_file_id across close & re-open

    slfritchie authored
  3. Scott Lystig Fritchie

    Bugfix: calculate end of table while still holding lock

    slfritchie authored

Jun 23, 2014

  1. Bishop Bors

    Merge pull request #173 from basho/metafold-fwd-port

    forward port of fix for fold_file_loop
    
    Reviewed-by: slfritchie
    borshop authored

Jun 18, 2014

  1. Evan Vigil-McClanahan

    the refactoring in the prior commit caused the hintfile CRC check to

    be moved from one loop to another, this adds a test to make sure that
    the expected behavior is seen.
    evanmcc authored
  2. Evan Vigil-McClanahan

    The fold_file_loop function had a bug (a misoptimization, really) that

    caused it to fail to fold over files that were exact multiples of the
    chunk size the outside fold was using to iterate with.  In addition to
    the misoptimization, this also goes afainst POSIX semantics by
    treating short reads as an exceptional case, rather than something
    that can happen at any read.
    
    The fix makes each of the inner fold functions check all matches for
    potential end of input.  The outer fold never tries to guess, but
    reads until the inner fold says it is done, or eof is reached. I was
    worried that this would cause a performance problem, but the perf
    testing scripts show no or negligible impact.
    evanmcc authored

Jun 04, 2014

  1. Scott Lystig Fritchie

    EUnit regression test case for `Cd8` of PR 156

    See the `Cd8` PULSE counterexample in #156
    slfritchie authored
  2. Scott Lystig Fritchie

    Merge branch 'feature/partial-merge-tombstone-removal' into develop

    slfritchie authored

May 28, 2014

  1. Scott Lystig Fritchie

    Buildbot cannot be trusted with EUnit's default 5 second timeout

    slfritchie authored
  2. Scott Lystig Fritchie

    Avoid memory leaks by PULSE model checking during 1+ day-long runs

    slfritchie authored

May 27, 2014

  1. Scott Lystig Fritchie

    Fix race in PULSE counterexample Cg4 in PR 156.

    Please see the PR 156 comment
    #156 (comment)
    for details on the race condition fixed by this patch.
    
    Fix: for the interest of safety, if we're iterating at all, then
    an update will create a new sibling.
    slfritchie authored
  2. Scott Lystig Fritchie

    Non-determinism hack: use raw file I/O for make_merge_file()

    slfritchie authored
  3. Scott Lystig Fritchie

    Fix EUnit nondeterminism by using unique data dir for each test itera…

    …tion
    slfritchie authored
  4. Scott Lystig Fritchie

    Add test time fudget to PR156 regression test cases

    slfritchie authored
  5. Scott Lystig Fritchie

    Adjust EUnit timeout for max_merge_size_test()

    slfritchie authored

May 26, 2014

  1. Scott Lystig Fritchie

    Another round of trying to eliminate PULSE nondeterminism.

    * PULSE side-effect for all bitcask_nifs functions, !@#$!.  The
    function-by-function declaration is too specific and thus too brittle.
    Every time a new NIF is added or an existing NIF arity is changed,
    everyone (myself included) forgets to change rebar.config.script.
    Instead, we'll use a blanket wildcard statement, and for the few
    non-NIF functions in bitcask_nifs.erl, they'll get instrumented too.
    
    * Move make_merge_file() from bitcask_pulse.erl -> bitcask.erl,
    because bitcask_pulse has an explicit `pulse_no_side_effect`
    directive for file:'_' calls.
    slfritchie authored
  2. Scott Lystig Fritchie

    Add more EUnit fudge time to generic_qc_fsm:qc_test_()

    slfritchie authored
  3. Scott Lystig Fritchie

    Fix truncated_datafile_test(), close all EUnit test keydirs

    slfritchie authored

May 25, 2014

  1. Scott Lystig Fritchie

    Fix race condition in write_and_keydir_put()

    Yet another unconditional keydir update that returns to bite later.
    In a race with merge, if merge makes a keydir update between the
    keydir read of do_put() and the keydir update of
    write_and_keydir_put(), we must tell the keydir update that this
    update needs the OldFileId prerequisite check.
    
    * Add args to write_and_keydir_put() to pass in prerequisite info
    * Use bitcask_nifs:keydir_put/10.
    * Export bitcask_nifs:keydir_put/10.
    
    Passes this previously-failing case, with 5,000 iterations:
    
        Cqq3 = [[{set,{var,3},{call,bitcask_pulse,fork,[[{init,{state,undefined,false,false,[]}},{set,{not_var,1},{not_call,bitcask_pulse,incr_clock,[]}},{set,{not_var,2},{not_call,bitcask_pulse,incr_clock,[]}},{set,{not_var,3},{not_call,bitcask_pulse,incr_clock,[]}},{set,{not_var,4},{not_call,bitcask_pulse,bc_open,[false,{true,{76,128,158},62}]}},{set,{not_var,5},{not_call,bitcask_pulse,fold,[{not_var,4}]}},{set,{not_var,6},{not_call,bitcask_pulse,incr_clock,[]}},{set,{not_var,7},{not_call,bitcask_pulse,fold,[{not_var,4}]}},{set,{not_var,8},{not_call,bitcask_pulse,gets,[{not_var,4},{3,11}]}},{set,{not_var,9},{not_call,bitcask_pulse,gets,[{not_var,4},{7,50}]}},{set,{not_var,11},{not_call,bitcask_pulse,incr_clock,[]}}]]}},{set,{var,4},{call,bitcask_pulse,bc_open,[true,{true,{255,63,413},100}]}},{set,{var,8},{call,bitcask_pulse,puts,[{var,4},{1,30},<<168,239,137>>]}},{set,{var,9},{call,bitcask_pulse,fold,[{var,4}]}},{set,{var,12},{call,bitcask_pulse,delete,[{var,4},7]}},{set,{var,13},{call,bitcask_pulse,puts,[{var,4},{1,13},<<7,103,94,199,217,211,64,106>>]}},{set,{var,14},{call,bitcask_pulse,puts,[{var,4},{18,33},<<141,252,203,157>>]}},{set,{var,15},{call,bitcask_pulse,puts,[{var,4},{42,47},<<209,111,152,239,176,33,14,200>>]}},{set,{var,19},{call,bitcask_pulse,fork_merge,[{var,4}]}},{set,{var,21},{call,bitcask_pulse,puts,[{var,4},{3,20},<<228,240,196,12>>]}},{set,{var,22},{call,bitcask_pulse,puts,[{var,4},{47,48},<<206,42,5,22>>]}},{set,{var,23},{call,bitcask_pulse,delete,[{var,4},7]}},{set,{var,27},{call,bitcask_pulse,sync,[{var,4}]}},{set,{var,28},{call,bitcask_pulse,puts,[{var,4},{31,38},<<71,173,38,2,161,59,41>>]}},{set,{var,29},{call,bitcask_pulse,incr_clock,[]}},{set,{var,30},{call,bitcask_pulse,bc_close,[{var,4}]}},{set,{var,31},{call,bitcask_pulse,incr_clock,[]}},{set,{var,32},{call,bitcask_pulse,bc_open,[true,{true,{218,167,488},84}]}},{set,{var,33},{call,bitcask_pulse,delete,[{var,32},28]}},{set,{var,34},{call,bitcask_pulse,fork_merge,[{var,32}]}},{set,{var,35},{call,bitcask_pulse,delete,[{var,32},7]}},{set,{var,37},{call,bitcask_pulse,delete,[{var,32},23]}},{set,{var,38},{call,bitcask_pulse,put,[{var,32},44,<<227,215,111,250,91,38>>]}},{set,{var,39},{call,bitcask_pulse,delete,[{var,32},29]}},{set,{var,41},{call,bitcask_pulse,sync,[{var,32}]}},{set,{var,9999},{call,bitcask_pulse,get,[{var,32},7]}},{set,{var,42},{call,bitcask_pulse,bc_close,[{var,32}]}},{set,{var,43},{call,bitcask_pulse,incr_clock,[]}}],{22840,31851,8451},[{0,[]},{1,[]},{errors,[]},{events,[]}]].
        [begin Now = now(), io:format("Now ~w ", [Now]), true = eqc:check(bitcask_pulse:prop_pulse(), [lists:nth(1, Cqq3), Now, lists:nth(3, Cqq3)]) end || _ <- lists:seq(1,5000)].
    slfritchie authored
Something went wrong with that request. Please try again.