Remove the possibility for merge and open to race. #150

evanmcc · 2014-03-21T15:47:42Z

As described in #95, if a vnode dies and takes down your bitcask instance, it's possible that a merge will get there first, causing repeated open failures that can take the whole node down. This change alters how open and merge work, disallowing merges if the bitcask instances is not currently opened by a pid in the same VM.

Additionally, as described in #149, initialization can hang on startup if the deletion queue is being blocked. Since that polling was intended to fix an issue on the merge init path (which no longer exists), we can remove it. The removal requires making some other pathways more robust.

This PR also contains some test cleanup.

slfritchie · 2014-03-24T07:09:14Z

-1 as-is, sorry. I'm seeing nondeterministic failures of the bitcask_qc_fsm:prop_bitcask() test: the failing cases appear to have zero failures while shrinking, which means that the original case == shrunk case, and attempts to get it to fail again don't work, e.g. [true = eqc:check(eqc:testing_time(5*60, bitcask_qc_fsm:prop_bitcask()), C1e) || _ <- lists:seq(1,500)]. won't fail. :-(

One counterexample:

....Failed! After 1835 tests.
[{set,{var,1},{call,bitcask_qc_fsm,set_keys,[[<<"É">>,<<"ÂxQ">>]]}},
 {set,{var,2},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,3},{call,bitcask,delete,[{var,2},<<"É">>]}},
 {set,{var,4},{call,bitcask,get,[{var,2},<<"ÂxQ">>]}},
 {set,{var,5},{call,bitcask,get,[{var,2},<<"k">>]}},
 {set,{var,6},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,7},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,8},{call,bitcask,delete,[{var,2},<<"k">>]}},
 {set,{var,9},{call,bitcask,get,[{var,2},<<"k">>]}},
 {set,{var,10},{call,bitcask,delete,[{var,2},<<"k">>]}},
 {set,{var,11},{call,bitcask,get,[{var,2},<<"ÂxQ">>]}},
 {set,{var,12},{call,bitcask,put,[{var,2},<<"ÂxQ">>,<<"Ùµ">>]}},
 {set,{var,13},{call,bitcask,close,[{var,2}]}},
 {set,{var,14},{call,bitcask_qc_fsm,truncate_hint,[4,2]}},
 {set,{var,15},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,16},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}}]
{exception,{'EXIT',{{badmatch,{error,einval}},
                    [{bitcask_qc_fsm,truncate_hint,2,
                                     [{file,"test/bitcask_qc_fsm.erl"},
                                      {line,195}]},
                     {eqc_statem,run_commands,2,
                                 [{file,"../src/eqc_statem.erl"},{line,737}]},
                     {eqc_fsm,run_commands,2,
                              [{file,"../src/eqc_fsm.erl"},{line,658}]},
                     {bitcask_qc_fsm,'-prop_bitcask/0-fun-0-',1,
                                     [{file,"test/bitcask_qc_fsm.erl"},
                                      {line,136}]}]}}} /= ok

Another, with a more common postcondition failure:

Failed! After 104 tests.
[{set,{var,1},
      {call,bitcask_qc_fsm,set_keys,
            [[<<15,110,21,184,133,7>>,
              <<212,170,216,23,19,99,208,49,145,186,84,204>>,
              <<20,250,144>>,
              <<137,166,109,136,140,80>>,
              <<176,175,189,19,136,143,88,113,202>>,
              <<138,112>>,
              <<77,91,242,18,223,147,118,165,24,113,91>>,
              <<22,89>>,
              <<116,56,59,128,93,176,127>>]]}},
 {set,{var,2},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,3},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,4},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,5},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,6},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,7},{call,bitcask,delete,[{var,5},<<15,110,21,184,133,7>>]}},
 {set,{var,8},{call,bitcask,get,[{var,5},<<22,89>>]}},
 {set,{var,9},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,10},
      {call,bitcask,delete,
            [{var,5},<<77,91,242,18,223,147,118,165,24,113,91>>]}},
 {set,{var,11},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,12},
      {call,bitcask,put,
            [{var,5},
             <<176,175,189,19,136,143,88,113,202>>,
             <<62,141,93,78,169,41>>]}},
 {set,{var,13},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,14},
      {call,bitcask,delete,
            [{var,5},<<77,91,242,18,223,147,118,165,24,113,91>>]}},
 {set,{var,15},{call,bitcask,put,[{var,5},<<138,112>>,<<">R">>]}},
 {set,{var,16},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,17},{call,bitcask,close,[{var,5}]}},
 {set,{var,18},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,19},{call,bitcask,delete,[{var,18},<<22,89>>]}},
 {set,{var,20},{call,bitcask,delete,[{var,18},<<116,56,59,128,93,176,127>>]}},
 {set,{var,21},
      {call,bitcask,delete,
            [{var,18},<<212,170,216,23,19,99,208,49,145,186,84,204>>]}},
 {set,{var,22},
      {call,bitcask,get,[{var,18},<<176,175,189,19,136,143,88,113,202>>]}},
 {set,{var,23},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}},
 {set,{var,24},{call,bitcask,delete,[{var,18},<<138,112>>]}},
 {set,{var,25},{call,bitcask,put,[{var,18},<<"k">>,<<"êæi">>]}},
 {set,{var,26},{call,bitcask,close,[{var,18}]}},
 {set,{var,27},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,28},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,29},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,30},{call,bitcask,get,[{var,29},<<"k">>]}},
 {set,{var,31},{call,bitcask,get,[{var,29},<<22,89>>]}},
 {set,{var,32},
      {call,bitcask,delete,[{var,29},<<176,175,189,19,136,143,88,113,202>>]}},
 {set,{var,33},{call,bitcask,close,[{var,29}]}},
 {set,{var,34},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,35},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.712",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,36},{call,bitcask,get,[{var,35},<<"k">>]}},
 {set,{var,37},{call,bitcask,delete,[{var,35},<<"k">>]}},
 {set,{var,38},
      {call,bitcask,put,
            [{var,35},
             <<212,170,216,23,19,99,208,49,145,186,84,204>>,
             <<"Cè<Ý'\"">>]}},
 {set,{var,39},{call,bitcask,merge,["/tmp/bitcask.qc.712"]}}]
{postcondition,{expected,<<62,141,93,78,169,41>>,got,not_found}} /= ok

slfritchie · 2014-03-24T08:27:52Z

Hrm. The 2nd non-deterministic case that I describe above doesn't appear to happen on develop as of commit f40fb00 ... if it does, it is happening with veryvery low probability. I can't get it to happen once on develop in several CPU hours of testing, whereas it's very rare to run 10 minutes on pevm-avoid-merge-open-race at commit c34dd5c without an error.

evanmcc · 2014-03-24T18:09:56Z

There are two ways to go with this, both seem to fix the issue. Still doing a basic PULSE run on 1, but it passed 450 seconds of the fsm test without throwing either counter.

Move the close_all line to after the setuid list comprehension.
Just delete the files, don't bother with the setuid stuff, get rid of deferred deletion entirely.

Opinions? My preference, due to it being simpler, is to do 2, but I can see an argument for 1 being that it's a smaller change and we're very close to release (also it's easier to backport).

evanmcc · 2014-03-25T00:31:41Z

https://github.com/basho/bitcask/tree/pevm-avoid-merge-open-race-opt1
I have a local branch for option 2, but it still has two issues that I'd like to look into before I push it (deleted a number of tests without thinking, need to port them else where, and investigate the source of some worrying log messages.). If time becomes an issue and # 1 works, then we can go with that.

slfritchie · 2014-03-25T03:03:22Z

Hi, Evan. pevm-avoid-merge-open-race-opt1 hasn't fixed the race, sorry.

Every failure that I see has a common pattern:

Put K with value V1
Merge
Close
Open
Get K -> old value V0

slfritchie · 2014-03-25T03:04:03Z

For example:

...Failed! After 208 tests.
[{set,{var,1},
      {call,bitcask_qc_fsm,set_keys,
            [[<<106,118,94,249,112,75,22>>,
              <<0,28,130>>,
              <<195,132,98,191>>,
              <<"??w8N">>]]}},
 {set,{var,2},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.71246",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,3},{call,bitcask,delete,[{var,2},<<"??w8N">>]}},
 {set,{var,4},{call,bitcask,merge,["/tmp/bitcask.qc.71246"]}},
 {set,{var,5},{call,bitcask,put,[{var,2},<<"k">>,<<"[">>]}},
 {set,{var,6},{call,bitcask,delete,[{var,2},<<0,28,130>>]}},
 {set,{var,7},{call,bitcask,put,[{var,2},<<195,132,98,191>>,<<"?p:">>]}},
 {set,{var,8},{call,bitcask,get,[{var,2},<<"??w8N">>]}},
 {set,{var,9},{call,bitcask,merge,["/tmp/bitcask.qc.71246"]}},
 {set,{var,10},{call,bitcask,put,[{var,2},<<"k">>,<<"?">>]}},
 {set,{var,11},{call,bitcask,close,[{var,2}]}},
 {set,{var,12},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,13},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,14},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,15},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,16},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.71246",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,17},{call,bitcask,close,[{var,16}]}},
 {set,{var,18},{call,bitcask_qc_fsm,create_stale_lock,[]}},
 {set,{var,19},
      {call,bitcask,open,
            ["/tmp/bitcask.qc.71246",
             [read_write,{open_timeout,1},{sync_strategy,none}]]}},
 {set,{var,20},{call,bitcask,get,[{var,19},<<195,132,98,191>>]}},
 {set,{var,21},{call,bitcask,get,[{var,19},<<106,118,94,249,112,75,22>>]}},
 {set,{var,22},{call,bitcask,merge,["/tmp/bitcask.qc.71246"]}},
 {set,{var,23},{call,bitcask,delete,[{var,19},<<0,28,130>>]}},
 {set,{var,24},{call,bitcask,get,[{var,19},<<"k">>]}},
 {set,{var,25},{call,bitcask,merge,["/tmp/bitcask.qc.71246"]}},
 {set,{var,26},{call,bitcask,put,[{var,19},<<"k">>,<<"?">>]}},
 {set,{var,27},{call,bitcask,delete,[{var,19},<<0,28,130>>]}},
 {set,{var,28},{call,bitcask,put,[{var,19},<<"??w8N">>,<<133,132,6,67,64>>]}},
 {set,{var,29},{call,bitcask,delete,[{var,19},<<195,132,98,191>>]}},
 {set,{var,30},{call,bitcask,delete,[{var,19},<<106,118,94,249,112,75,22>>]}},
 {set,{var,31},{call,bitcask,merge,["/tmp/bitcask.qc.71246"]}}]
{postcondition,{expected,<<"?p:">>,got,not_found}} /= ok
*failed*

evanmcc · 2014-03-25T04:35:00Z

Sorry about the multiple tries here, my machine hasn't done a great job of hitting these.

evanmcc · 2014-03-26T00:34:57Z

Spent a lot of time today trying to get this to happen reliably, but haven't seen it once.

I pushed an exploratory commit to the opt1 branch tweaking the locking behavior. Scott, if you have the time, could you run it and see if it turns the counter from a model check failure to a badmatch on the lock? Also, any repro tips would be appreciated. I tried running it on a ramdisk, running several testers in parallel to slow the disk down, but nothing seems to help, even on the original code.

slfritchie · 2014-03-26T06:00:20Z

FWIW, bitcask_qc_fsm's create_stale_lock command is not required to cause an error in bitcask_qc_fsm:prop_bitcask().

To avoid tearing my SSD to shreds, I've started using a RAM disk for QuickCheck testing. My OS X steps are borrowed from http://bogner.sh/2012/12/os-x-create-a-ram-disk-the-easy-way/

Looking at commits going forward in time:

commit 6a549bf: no failure in 50 minutes of CPU time
commit f3374d2: {postcondition,{expected,<<122,76,234,0,206>>,got,not_found}} /= ok inside of a few minutes
commit c34dd5c: fails, same reason
commit 50d2ab4: fails, same reason
commit 804495e: fails, same reason

If things are working correctly as of 6a549bf, is it worth declaring victory there? (Assuming that yet-to-be-finished long-term QuickCheck & PULSE testing at that commit doesn't find another problem.)

evanmcc · 2014-03-26T16:53:08Z

commit 6a549bf fixes #95, but leaves #149 hanging. I am OK with
resetting to there and merging, but it leaves work to be done. If we
can work around this, it's potentially a much simpler solution than
partitioning the delete queue to avoid fate-sharing. I'd like a bit
more time to poke at it before we go to the more complicated solution.

slfritchie · 2014-03-27T06:54:15Z

FWIW, I see commit 6a549bf passing the bitcask_pulse:prop_pulse() test after 8 days of CPU time.

evanmcc · 2014-03-27T18:33:12Z

PULSE testing doesn't close and reopen fast enough to hit this issue, I don't think.

I've found the issue and will likely rebase this branch Real Soon Now. The issue is that list_data_files can get an old write file from a write lock file whose unlike hasn't quite completed. Luckily, since the semantics of merging have changed, there's a checkpoint at the start of the initial file scan where we are able to say: "no locks should exist here, ever". So the fix is to strongly enforce that.

I'd really like to get my remove-merge-delete branch in (opt 2 above), but there are some additional issues with it, and I am out of time. I am going to clean up the opt one branch and make that the review candidate today, and I'll revisit the cleanup later.

evanmcc · 2014-03-27T23:17:37Z

Other than the PULSE comment, ignore what I said in the last thing. There's strong evidence (I shouldn't have been so definite last time, but I am pretty sure here), that this is a test-design problem. The issue seems to be that the delete worker isn't getting reset at the start of the test, so it can still be carrying delete orders from several test ago when it finally triggers. Here's some debugging prints showing the issue:

purge ldf: [] undefined undefined
ikskf ldf: [] undefined undefined
fresh  
put <<56,23,162,252,57,130,19,150,241,85>>, ret ok
deleting ["/tmp/bitcask.qc.1312/1.bitcask.data"]
put <<125,8,233,152,31,153,2,29,136,3,225,45,47>>, ret ok
put <<62,6,20>>, ret ok
deleting ["/tmp/bitcask.qc.1312/1.bitcask.data"]
merge deleting ["/tmp/bitcask.qc.1312/3.bitcask.data"]
deleting ["/tmp/bitcask.qc.1312/2.bitcask.data",
          "/tmp/bitcask.qc.1312/4.bitcask.data"]
ldf: [] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
put <<62,6,20>>, ret ok
deleting ["/tmp/bitcask.qc.1312/5.bitcask.data",
          "/tmp/bitcask.qc.1312/6.bitcask.data"]
merge ldf: [] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
deleting ["/tmp/bitcask.qc.1312/8.bitcask.data"]
merge deleting ["/tmp/bitcask.qc.1312/9.bitcask.data"]
ldf: [] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
deleting ["/tmp/bitcask.qc.1312/7.bitcask.data",
          "/tmp/bitcask.qc.1312/10.bitcask.data"]
getting <<"k">>:  nf 
merge deleting ["/tmp/bitcask.qc.1312/1.bitcask.data"]
deleting ["/tmp/bitcask.qc.1312/2.bitcask.data"]
ldf: [] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
deleting ["/tmp/bitcask.qc.1312/1.bitcask.data"]
deleting ["/tmp/bitcask.qc.1312/2.bitcask.data"]
put <<"k">>, ret ok
getting <<36,60,144,163,199,106,155,61,220>>:  nf 
put <<125,8,233,152,31,153,2,29,136,3,225,45,47>>, ret ok
getting <<"k">>:  getting <<"k">>:  merge ldf: [] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
purge ldf: [] undefined undefined
ikskf ldf: [] undefined undefined
fresh
put <<"k">>, ret ok
getting <<36,60,144,163,199,106,155,61,220>>:  nf 
merge ldf: [{1,"/tmp/bitcask.qc.1312/1.bitcask.data"}] "/tmp/bitcask.qc.1312/1.bitcask.data" undefined
getting <<41,64,176,20,229,18,197,12,110,36,242,4,102>>:  nf 
getting <<56,23,162,252,57,130,19,150,241,85>>:  nf 
{expected,<<56,23,162,252,57,130,19,150,241,85>>,
          <<197,165,94,159,122,46,144,48,211,243,175,251,88,254>>,
          got,not_found}

As you can see, there are a number of deletes spread throughout the various opens and merges. A delete queued in this execution would be prefixed with FF, and there are none. You can see that we're seeing deletes of file 7 and 10 even though we've only seen a few puts.

Seeing fresh more than once is also a really bad sign.

Anyway, I am feeling non-confident about anything, but I think that I might actually be on to it this time. I am going to let things churn for a couple of hours on this branch, then rebase if everything is still looking good.

The branch that I am currently testing is: poll-race-try2

opens always work.

- avoid fate-sharing by not caring about the delete queue. - clean up an unused variable warning in the maybe_new nif.

evanmcc · 2014-03-28T03:58:59Z

rebased to current develop, fixed a few more test errors, should be good to go now.

pre-rebase it did 3 hours of qc_fsm and 4 hours of pulse. Restarting PULSE now.

slfritchie · 2014-03-31T03:17:17Z

My testing over the weekend found another race. Quickly, this exact pattern match needs to be more flexible: https://github.com/basho/bitcask/blob/pevm-avoid-merge-open-race/src/bitcask.erl#L1349 ... the failure case isn't 100% deterministic, so I'm going to try to hunt for where the nondeterminism is creeping in.

Ce5 = [[{set,{var,3},{call,bitcask_pulse,bc_open,[true]}},
  {set,{var,5},
       {call,bitcask_pulse,put,[{var,3},39,<<0,0,0,0>>]}},
  {set,{var,8},
       {call,bitcask_pulse,fork,
             [[{init,{state,undefined,false,false,[]}},
               {set,{not_var,18},{not_call,bitcask_pulse,incr_clock,[]}},
               {set,{not_var,19},{not_call,bitcask_pulse,bc_open,[false]}},
               {set,{not_var,21},
                    {not_call,bitcask_pulse,sync,[{not_var,19}]}},
               {set,{not_var,22},
                    {not_call,bitcask_pulse,fold,[{not_var,19}]}},
               {set,{not_var,23},
                    {not_call,bitcask_pulse,fold,[{not_var,19}]}},
               {set,{not_var,25},
                    {not_call,bitcask_pulse,bc_close,[{not_var,19}]}},
               {set,{not_var,26},{not_call,bitcask_pulse,incr_clock,[]}},
               {set,{not_var,27},{not_call,bitcask_pulse,incr_clock,[]}},
               {set,{not_var,28},
                    {not_call,bitcask_pulse,bc_open,[false]}}]]}},
  {set,{var,16},{call,bitcask_pulse,delete,[{var,3},39]}},
  {set,{var,24},{call,bitcask_pulse,bc_close,[{var,3}]}},
  {set,{var,28},{call,bitcask_pulse,bc_open,[true]}},
  {set,{var,29},{call,bitcask_pulse,merge,[{var,28}]}},
  {set,{var,30},{call,bitcask_pulse,bc_close,[{var,28}]}},
  {set,{var,34},{call,bitcask_pulse,incr_clock,[]}},
  {set,{var,35},{call,bitcask_pulse,incr_clock,[]}},
  {set,{var,36},{call,bitcask_pulse,bc_open,[true]}},
  {set,{var,40},{call,bitcask_pulse,merge,[{var,36}]}},
  {set,{var,62},{call,bitcask_pulse,fold,[{var,36}]}},
  {set,{var,63},{call,bitcask_pulse,delete,[{var,36},22]}}],
 {23848,45862,63967},
 [{0,[]}]]

[true = eqc:check(eqc:testing_time(24*3600, bitcask_pulse:prop_pulse()), Ce5) || _ <- lists:seq(1, 100)].

slfritchie · 2014-03-31T06:39:01Z

Hrm, no that pattern match doesn't need flexibility. But that case writes a key, deletes it, merges it away, close & reopen ... and the reopen discovers that all data files are setuid. So the keydir has never seen a call to increment_file_id() and thus the keydir's view of the world is wrong. I have a patch testing right now.....

…minism in PULSE test

…etuid files

slfritchie · 2014-04-02T09:13:32Z

Commit f085660 has passed 8 CPU days without failing, yay.

evanmcc · 2014-04-08T19:16:01Z

ping? has there been code review or just testing? do we need someone else
to look at it?

slfritchie · 2014-04-09T03:34:42Z

@evanmcc I'm waiting for review of my 5 commits from last week.

evanmcc · 2014-04-09T06:32:20Z

They look good to me.

slfritchie · 2014-04-09T07:48:37Z

+1 f085660

Remove the possibility for merge and open to race. Reviewed-by: slfritchie

slfritchie · 2014-04-09T08:04:50Z

+1 f085660

slfritchie · 2014-04-09T08:05:30Z

Hrm, I hope I haven't confused borshop with an extra approval.

slfritchie · 2014-04-09T08:06:33Z

@borshop merge

slfritchie added the Bug label Mar 24, 2014

slfritchie added this to the 2.0-RC milestone Mar 24, 2014

evanmcc added 3 commits March 27, 2014 23:30

make it so that a merge will not implicitly open a cask, so that

b6e729e

opens always work.

- change test designs to no longer throw spurious errors.

43bb080

- avoid fate-sharing by not caring about the delete queue. - clean up an unused variable warning in the maybe_new nif.

get dialyzer clean.

b432c59

slfritchie added 5 commits March 31, 2014 15:50

Update NIFs in rebar.config.script to try to eliminate some non-deter…

45259f5

…minism in PULSE test

Avoid compiler complaints about missing #state record types

750d6db

Fix compilation bug in slow_folder()

abd21e2

Bugfix: only delete merge input files that have been successfully merged

956cec0

Bugfix: Keydir init needs to inform keydir about largest tstamp for s…

f085660

…etuid files

borshop added a commit that referenced this pull request Apr 9, 2014

Merge pull request #150 from basho/pevm-avoid-merge-open-race

e670764

Remove the possibility for merge and open to race. Reviewed-by: slfritchie

borshop merged commit f085660 into develop Apr 9, 2014

borshop deleted the pevm-avoid-merge-open-race branch April 9, 2014 08:08

slfritchie restored the pevm-avoid-merge-open-race branch April 9, 2014 08:17

This was referenced Apr 16, 2014

bitcask:init_keydir(): Avoid unnecessary wait by bitcask:poll_deferred_delete_queue_empty/0 #149

Closed

Investigate possible startup race between open and merge. #95

Closed

seancribbs deleted the pevm-avoid-merge-open-race branch April 1, 2015 22:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the possibility for merge and open to race. #150

Remove the possibility for merge and open to race. #150

evanmcc commented Mar 21, 2014

slfritchie commented Mar 24, 2014

slfritchie commented Mar 24, 2014

evanmcc commented Mar 24, 2014

evanmcc commented Mar 25, 2014

slfritchie commented Mar 25, 2014

slfritchie commented Mar 25, 2014

evanmcc commented Mar 25, 2014

evanmcc commented Mar 26, 2014

slfritchie commented Mar 26, 2014

evanmcc commented Mar 26, 2014

slfritchie commented Mar 27, 2014

evanmcc commented Mar 27, 2014

evanmcc commented Mar 27, 2014

evanmcc commented Mar 28, 2014

slfritchie commented Mar 31, 2014

slfritchie commented Mar 31, 2014

slfritchie commented Apr 2, 2014

evanmcc commented Apr 8, 2014

slfritchie commented Apr 9, 2014

evanmcc commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014

Remove the possibility for merge and open to race. #150

Remove the possibility for merge and open to race. #150

Conversation

evanmcc commented Mar 21, 2014

slfritchie commented Mar 24, 2014

slfritchie commented Mar 24, 2014

evanmcc commented Mar 24, 2014

evanmcc commented Mar 25, 2014

slfritchie commented Mar 25, 2014

slfritchie commented Mar 25, 2014

evanmcc commented Mar 25, 2014

evanmcc commented Mar 26, 2014

slfritchie commented Mar 26, 2014

evanmcc commented Mar 26, 2014

slfritchie commented Mar 27, 2014

evanmcc commented Mar 27, 2014

evanmcc commented Mar 27, 2014

evanmcc commented Mar 28, 2014

slfritchie commented Mar 31, 2014

slfritchie commented Mar 31, 2014

slfritchie commented Apr 2, 2014

evanmcc commented Apr 8, 2014

slfritchie commented Apr 9, 2014

evanmcc commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014

slfritchie commented Apr 9, 2014