Program terminated with signal 4, Illegal instruction.

[linux1fans]

Please answer these questions before submitting your issue. Thanks!

1. What version of Rust are you using (rustc --version)?
   bin/tikv-server --version
   TiKV
   Release Version: 2.0.0-rc.3
   Git Commit Hash: 4c3b6e5
   Git Commit Branch: master
   UTC Build Time: 2018-03-29 04:16:58
   Rust Version: 1.25.0-nightly (73ac5d6a8 2018-01-11)

2. What operating system and processor architecture are you using?

Red Hat Enterprise Linux Server release 7.2(Maipo)
processor : 47
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping : 1
microcode : 0xb00001f
cpu MHz : 2487.117
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 13
cpu cores : 12
apicid : 59
initial apicid : 59
fpu : yes
fpu_exception : yes
cpuid level : 20
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc
bogomips : 4404.41
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.

./sysbench --test=lua-tests/db/parallel_prepare.lua --db-driver=mysql --mysql-host=192.168.20.250 --mysql-port=4000 --mysql-user=root --mysql-password= --mysql-db=sbtest --oltp-tables-count=64 --oltp-table-size=50000000 --rand-init=on --num-threads=8 run

4. What did you expect to see?
   load successful

5. What did you see instead?
   core was generated by 'bin/tikv-server -C conf/tikv6.toml'
   Program terminated with signal 4, Illegal instruction.
   #0 alloc_system::platform::$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::oom::h8217d9dced0e3490 () at liballoc_system/lib.rs:210
   210 liballoc_system/lib.rs: No such file or directory.
   warning: Missing auto-load scripts referenced in section .debug_gdb_scripts
   of file /opt/tidb/bin/tikv-server
   Use `info auto-load python [REGEXP]' to list them.
   Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 libgcc-4.8.5-16.el7.x86_64
   (gdb) bt
   #0 alloc_system::platform::$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::oom::h8217d9dced0e3490 () at liballoc_system/lib.rs:210
   proto: Don't use short term #1 0x00007f309d02b830 in _$LT$alloc_system..System$u20$as$u20$alloc..allocator..Alloc$GT$::oom::h48f3fcc4861161c5 () at liballoc_system/lib.rs:81
   proto: Rename apply to append #2 0x00007f309d02b684 in __rde_oom () at liballoc_jemalloc/lib.rs:126
   *: add Cargo.toml #3 0x00007f309d55f1a7 in oom (self=, err=...) at /checkout/src/liballoc/heap.rs:98
   *: add skeleton files and log library #4 reserve<u8,alloc::heap::Heap> (used_cap=, needed_extra_cap=, self=) at /checkout/src/liballoc/raw_vec.rs:537
   *: Rename raft.proto to raftpb.proto #5 reserve (self=, additional=) at /checkout/src/liballoc/vec.rs:464
   *: move main.rs to src/bin/server.rs #6 spec_extend (self=, iterator=...) at /checkout/src/liballoc/vec.rs:1903
   *: update format #7 extend<u8,&[u8]> (self=, iter=...) at /checkout/src/liballoc/vec.rs:2054
   raft: Add storage trait skeleton #8 protobuf::stream::CodedOutputStream::write_raw_bytes::h1b286f77d27c389d (self=0x7f306edfb9a0, bytes=...) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/stream.rs:959
   storage: local storage engine #9 0x00007f309d2aeb91 in write_bytes_no_tag (self=0x7f306edfb9a0, bytes=...) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/stream.rs:1200
   raft: Introduce error #10 write_bytes (field_number=1, bytes=..., self=) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/stream.rs:1214
   update error #11 _$LT$kvproto..coprocessor..Response$u20$as$u20$protobuf..core..Message$GT$::write_to_with_cached_sizes::h3f899dc8d6107d19 (self=0x7f306edfc270, os=0x7f306edfb9a0)
   at /home/jenkins/.cargo/git/checkouts/kvproto-e694950bee90bc86/340ba9f/src/coprocessor.rs:944
   raft: add base codec. #12 0x00007f309dc88ccf in protobuf::core::Message::write_to::hef7392f6735f9be6 (self=0x7f306edfc270, os=0x7f306edfb9a0) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/core.rs:54
   raft: un-export protobuf error. #13 0x00007f309dca7b7a in {{closure}}kvproto::coprocessor::Response (os=0xfffffffffffffffb) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/core.rs:98
   raft: Implement more methods #14 _$LT$$RF$$u27$a$u20$mut$u20$alloc..vec..Vec$LT$u8$GT$$u20$as$u20$protobuf..stream..WithCodedOutputStream$GT$::with_coded_output_stream::he6198cf10200a923 (self=, cb=...)
   at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/stream.rs:752
   add nocapture for test #15 0x00007f309daaa439 in write_to_veckvproto::coprocessor::Response (self=, v=0x7f309e25f085 <str.1>) at /home/jenkins/.cargo/git/checkouts/rust-protobuf-c5cebfff504ea45b/51f50fb/protobuf/src/core.rs:98
   storage: add mvcc mod #16 grpcio::codec::pb_codec::ser::hcff4287942a81c65 (t=, buf=0x7f309e25f085 <str.1>) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/codec.rs:43
   refactor codec #17 0x00007f309db6cd31 in {{closure}}kvproto::coprocessor::Response (t=0x7f306edfc270) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/call/server.rs:289
   raft: Finish log_unstable model #18 map<&kvproto::coprocessor::Response,alloc::vec::Vec,closure> (self=..., f=...) at /checkout/src/libcore/option.rs:404
   mvcc: implement scan #19 _$LT$grpcio..call..server..UnarySink$LT$T$GT$$GT$::complete::ha0c4bb419b68c930 (self=..., status=..., t=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/call/server.rs:287
   raft: Clean up tests #20 0x00007f309dc0d0d7 in successkvproto::coprocessor::Response (self=..., t=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/call/server.rs:279
   *: resolve warnings #21 {{closure}}tikv::server::transport::ServerRaftStoreRouter (resp=...) at /home/jenkins/workspace/build_tikv_master/go/src/github.com/pingcap/tikv/src/server/service/kv.rs:827
   update proto #22 {{closure}}<futures::future::map_err::MapErr<futures::sync::oneshot::Receiverkvproto::coprocessor::Response, fn(futures::sync::oneshot::Canceled) -> tikv::server::errors::Error>,futures::future::map_err::MapErr<grpcio::call::server::UnarySinkResult, fn(grpcio::error::Error) -> tikv::server::errors::Error>,closure> (e=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:34
   storage: refactor, make 'mvcc' an extend Trait for 'Engine' #23 map<kvproto::coprocessor::Response,tikv::server::errors::Error,core::result::Result<(), futures::future::map_err::MapErr<grpcio::call::server::UnarySinkResult, fn(grpcio::error::Error) -> tikv::server::errors::Error>>,closure> (
   self=..., op=...) at /checkout/src/libcore/result.rs:468
   storage: add tests #24 {{closure}}<futures::future::map_err::MapErr<futures::sync::oneshot::Receiverkvproto::coprocessor::Response, fn(futures::sync::oneshot::Canceled) -> tikv::server::errors::Error>,futures::future::map_err::MapErr<grpcio::call::server::UnarySinkResult, fn(grpcio::error::Error) -> tikv::server::errors::Error>,closure> (result=..., f=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:33
   print backtrace for panic test. #25 _$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::h912fec5162952872 (self=0x7f2fc75892c0, f=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/chain.rs:39
   raft: Add raft-log implemention #26 0x00007f309db54a66 in poll<futures::future::map_err::MapErr<futures::sync::oneshot::Receiverkvproto::coprocessor::Response, fn(futures::sync::oneshot::Canceled) -> tikv::server::errors::Error>,futures::future::map_err::MapErr<grpcio::call::server::UnarySinkResult, fn(grpcio::error::Error) -> tikv::server::errors::Error>,closure> (self=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/and_then.rs:32
   refactor and simplify it. #27 _$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h2c68b71904f0b5f7 (self=0x7f2fc75892c0)
   at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map.rs:30
   proto: update raft server proto. #28 0x00007f309dd23cf0 in _$LT$futures..future..map_err..MapErr$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h0dacdfa43ec12f9e (self=0x7f2fc75892c0)
   at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/map_err.rs:30
   add base raftserver support.  #29 0x00007f309d0dd387 in poll (self=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/future/mod.rs:113
   add rpc server #30 {{closure}}<alloc::boxed::Box,alloc::arc::Arcgrpcio::async::executor::SpawnNotify> (f=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:288
   proto: skip proto source file. #31 {{closure}}<alloc::boxed::Box,closure,core::result::Result<futures::poll::Async<()>, ()>> () at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:362
   support tick/timer #32 futures::task_impl::std::set::h6baf7a0cb92d85f6 (task=0x7f306edfe530, f=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/std/mod.rs:78
   refactor raftserver msg #33 0x00007f309d0dfae0 in enter<alloc::boxed::Box,closure,core::result::Result<futures::poll::Async<()>, ()>> (unpark=New, self=, f=...)
   at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:362
   raft: port partial tests from upstream #34 poll_future_notify<alloc::boxed::Box,alloc::arc::Arcgrpcio::async::executor::SpawnNotify> (notify=, id=0, self=)
   at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.17/src/task_impl/mod.rs:288
   *: Treat files that generated by protocol buffer as binary #35 grpcio::async::executor::poll::ha52ca6dca1ceaaf6 (notify=..., woken=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/async/executor.rs:158
   *: ignore target files #36 0x00007f309d0df5f3 in grpcio::async::executor::SpawnNotify::resolve::he41c12b01d7a0fee (self=..., success=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/async/executor.rs:124
   raftserver: retry send when notify queue full. #37 0x00007f309d0e4483 in grpcio::async::CallTag::resolve::hc1ea0e1e3508e55c (self=..., cq=0x0, success=) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/async/mod.rs:193
   raft: Add implementation of progress. #38 0x00007f309d0da099 in grpcio::env::poll_queue::h6a51670a700be043 (cq=...) at /home/jenkins/.cargo/registry/src/github.com-1ecc6299db9ec823/grpcio-0.2.0/src/env.rs:39
   raftserver: use front_mut instead of pop first. #39 0x00007f309d0e25ae in __rust_maybe_catch_panic () at /checkout/src/libpanic_unwind/lib.rs:102
   porting non panic-required tests from upstream #40 try<(),std::panic::AssertUnwindSafe> (f=...) at /checkout/src/libstd/panicking.rs:458
   raftserver: simplify and cleanup #41 catch_unwind<std::panic::AssertUnwindSafe,()> (f=...) at /checkout/src/libstd/panic.rs:365
   support sending data to remote peer.  #42 {{closure}}<closure,()> () at /checkout/src/libstd/thread/mod.rs:405
   raftserver: return handler for simplifying test and later use. #43 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h304172f3a97bae8d (self=0x7f309b7f4240, args=) at /checkout/src/liballoc/boxed.rs:817
   port all remaining raft_log_tests #44 0x00007f309d5fd718 in call_once<(),()> () at /checkout/src/liballoc/boxed.rs:827
   cleanup raftserver #45 std::sys_common::thread::start_thread::h1943b0e3d7d32532 () at libstd/sys_common/thread.rs:24
   cleanup register #46 0x00007f309d604839 in std::sys::unix::thread::Thread::new::thread_start::h5fee8a1b1de2d71e () at libstd/sys/unix/thread.rs:90
   prepare for raft porting #47 0x00007f309c5bce25 in start_thread () from /lib64/libpthread.so.0
   add handle_quit interface.  #48 0x00007f309c0d434d in clone () from /lib64/libc.so.6

[siddontang]

Did TiKV crash after running for a long time?

[linux1fans]

大概半小时左右，一部分tikv-server 出现上述问题

[siddontang]

seem that your memory is not enough. What is the Mem on your machine? How many TiKVs do you run in one machine?

[linux1fans]

free -g
total used free shared buff/cache available
Mem: 251 15 71 0 165 202
Swap: 7 0 7

一台机器6个tikv，cache配置如下:
[defaultcf]
block-cache-size = "6GB"
[writecf]
block-cache-size = "2GB"
[raftdb.defaultcf]
block-cache-size = "1GB"

[Hoverbear]

Hi, I'm closing this due to inactivity. Please reopen it if it is still an issue!