SIGSEGV in leveldb::MemTableIterator::key

[slfritchie]

Fault injection parameters:

• EIO on pread() 25% of the time
• EIO on stat() 25% of the time

Based upon develop branch as of fc3f09e

Steps to reproduce on OS X:

git checkout 74b272f861b73d49044f2046ebebd0538a87af78
make
rebar skip_deps=true -v eunit suites=XX
make compile_scenarios
./deps/faulterl/ebin/make_intercept_c.escript trigger_commonpaths yo "perl -npi -e 's/bc_fi_enabled = 1/bc_fi_enabled = 0/'"
env `./deps/faulterl/ebin/example_environment.sh yo` erl -pz .eunit deps/*/ebin

Then run in the Erlang shell:

[begin Rand = random:uniform(1 bsl 27), faulterl_nif:poke("bc_fi_random_seed", 0, <<Rand:32>>, false), faulterl_nif:poke("bc_fi_random_reseed", 0, <<1:8>>, false), Rand end || true].
[eunit:test(iterators, [verbose]) || _ <- lists:seq(1,99999)].

It will likely take several hundred iterations to hit a SIGSEGV crash. If you don't get one, try the last Erlang command again.

GDB says:

(gdb) where
#0  leveldb::MemTableIterator::key (this=0x7fe673101340) at db/memtable.cc:61
#1  0x0000000021bd506e in leveldb::(anonymous namespace)::DBIter::key (this=<value temporarily unavailable, due to optimizations>) at db/db_iter.cc:67
#2  0x0000000021bc77f3 in eleveldb::MoveTask::operator() (this=0x7fe671c20fd0) at c_src/workitems.cc:298
#3  0x0000000021bc6873 in eleveldb::eleveldb_thread_pool::notify_caller (work_item=@0x7fe671c20fd0) at c_src/threading.cc:309
#4  0x0000000021bc6818 in eleveldb::WorkTask::resubmit () at /Users/fritchie/b/src/eleveldb/c_src/workitems.h:373
#5  0x0000000021bc6818 in eleveldb::eleveldb_write_thread_worker (args=0x7fe6732022b0) at c_src/threading.cc:374
#6  0x00000000160255b5 in erl_drv_thread_wrapper (vdtid=0x1ff03ed0) at beam/erl_drv_thread.c:117
#7  0x00000000161c29c4 in thr_wrapper (vtwd=0xb0eca988) at pthread/ethread.c:106
#8  0x00000000167e9772 in _pthread_start ()
#9  0x00000000167d61a1 in thread_start ()
(gdb) list
56        virtual void Seek(const Slice& k) { iter_.Seek(EncodeKey(&tmp_, k)); }
57        virtual void SeekToFirst() { iter_.SeekToFirst(); }
58        virtual void SeekToLast() { iter_.SeekToLast(); }
59        virtual void Next() { iter_.Next(); }
60        virtual void Prev() { iter_.Prev(); }
61        virtual Slice key() const { return GetLengthPrefixedSlice(iter_.key()); }
62        virtual Slice value() const {
63          Slice key_slice = GetLengthPrefixedSlice(iter_.key());
64          return GetLengthPrefixedSlice(key_slice.data() + key_slice.size());
65        }
(gdb) p iter_
$2 = {
  list_ = 0x7fe673100e68, 
  node_ = 0x0
}

[matthewvon]

retry with this: #104

[slfritchie]

Still crashes, just takes longer ... but that test isn't deterministic, so 104's fix may have zero impact on how long it takes to crashes with SIGSEGV, sorry.

[slfritchie]

Hi, sorry, please ignore my comment from 4 minutes ago.

[slfritchie]

As far as I can tell, both the crash of this ticket and the bad iterator of #102 are resolved by #104 and basho/leveldb#125