Segmentation fault

[cffan]

Hi, I got a segmentation fault when running "lmplz -o 3 < text > arpa" on a corpus. Stack trace is pasted below. I've got lmplz running fine on several other corpora. The only thing special about this corpus is it contains a lot of duplicated sentences, don't know if this could cause the segmentation fault.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffca054700 (LWP 9216)]
0x00000000004856ca in lm::builder::NGram::IsMarked (this=0x7fffca053c20) at ./lm/builder/ngram.hh:77
77 return Value().count >> (sizeof(Value().count) * 8 - 1);
(gdb) bt
#0 0x00000000004856ca in lm::builder::NGram::IsMarked (this=0x7fffca053c20)

at ./lm/builder/ngram.hh:77

#1 0x000000000048e12a in lm::builder::NGram::CutoffCount (this=0x7fffca053c20)

at ./lm/builder/ngram.hh:93

#2 0x000000000048afa6 in lm::builder::(anonymous namespace)::PruneNGramStream::operator++ (

this=0x7fffca053c20) at /home/cfan/tools/kenlm/lm/builder/initial_probabilities.cc:74

#3 0x000000000048bb40 in lm::builder::(anonymous namespace)::MergeRight::Run (this=0x95bc78,

primary=...) at /home/cfan/tools/kenlm/lm/builder/initial_probabilities.cc:238

#4 0x000000000048df48 in util::stream::Thread::operator()<util::stream::ChainPosition, lm::builder::{anonymous}::MergeRight>(const util::stream::ChainPosition &, lm::builder::(anonymous namespace)::MergeRight &) (this=0x928170, position=..., worker=...) at ./util/stream/chain.hh:77
#5 0x000000000048ddf1 in boost::_bi::list2boost::_bi::value<util::stream::ChainPosition, boost::_bi::valuelm::builder::{anonymous}::MergeRight >::operator()boost::reference_wrapper<util::stream::Thread, boost::_bi::list0>(boost::_bi::type, boost::reference_wrapperutil::stream::Thread &, boost::_bi::list0 &, int) (this=0x95bc40, f=..., a=...) at /usr/include/boost/bind/bind.hpp:313
#6 0x000000000048dccf in boost::_bi::bind_t<void, boost::reference_wrapperutil::stream::Thread, boost::_bi::list2boost::_bi::value<util::stream::ChainPosition, boost::_bi::valuelm::builder::{anonymous}::MergeRight > >::operator()(void) (this=0x95bc38)

at /usr/include/boost/bind/bind_template.hpp:20

#7 0x000000000048dc34 in boost::detail::thread_data<boost::_bi::bind_t<void, boost::reference_wrapperutil::stream::Thread, boost::_bi::list2boost::_bi::value<util::stream::ChainPosition, boost::_bi::valuelm::builder::{anonymous}::MergeRight > > >::run(void) (this=0x95bab0)

at /usr/include/boost/thread/detail/thread.hpp:61

[kpu]

Hmmm I've heard rumors of a problem with duplicated corpora. Is there a chance I can get a copy of the corpus? Right now it looks like classic buffer overrun but it's unclear whose fault that is.

[cffan]

I've email you the link for corpus. Thanks!

[kpu]

Paging @emjotde probably the same segfault that @phikoehn saw.

[cffan]

Thanks!