Coredump when using latest nightly rustc

[BusyJay]

Hi, recently we upgrade our rustc compiler to the latest nightly version, but the compiled binary core dumps quickly under stress tests. A few stacks can be found in tikv/tikv#1144. But when we downgrade rustc to rustc 1.12.0-nightly (b30eff7ba 2016-08-05), the binary works just fine.

The stacks look weird, because the segment fault happens in liballoc, but we don't manage memory by ourselves. We are guessing that there might be some problems in the versions later than rustc 1.12.0-nightly (b30eff7ba 2016-08-05). Could you please help us check it out? Thanks!

[alexcrichton]

Thanks for the report @BusyJay! Is there also a set of steps to reproduce the crash you're seeing as well locally? Also, is this on Linux?

[BusyJay]

Yes, it's on Linux. It needs a few steps to reproduce the crash, we will provide it via a demo project later.

[siddontang]

Hi @alexcrichton

The steps to reproduce the crash may be not easy:

Download official binaries

wget http://download.pingcap.org/tidb-latest-linux-amd64.tar.gz
tar -xzf tidb-latest-linux-amd64.tar.gz
cd tidb-latest-linux-amd64

You can use tidb-server and pd-server in bin directory.

Clone TiKV and build

git clone https://github.com/pingcap/tikv.git tikv 
cd tikv
make static_release 
// If you have already installed RocksDB (ver > 4.12), you can use `make` directly.

The tikv-server is installed in bin directory.

Run PD

./bin/pd-server --cluster-id=1 --data-dir=./var/pd

pd-server will listen 2379 and 2380 ports.

Run TiKV

./bin/tikv-server -I 1 -S raftkv --pd 127.0.0.1:2379 -s ./var/tikv

tikv-server will listen 20160 port.

Run TiDB

./bin/tidb-server --store=tikv --path="127.0.0.1:2379?cluster=1" 

tidb-server will listen 4000 port.

Run Test

Unzip bank test bank.zip and run

./bank 

Then wait a long time......

If you have any problem, please tell me.

[eddyb]

(Shot in the dark) Does this reproduce on rustc 1.12.0-nightly (b30eff7ba 2016-08-05) when built with the environment variable RUSTFLAGS set to -Zorbit?

[brson]

@siddontang Is there any way this crash can be reduced to a smaller test case? It's going to be quite tricky to track down otherwise. Do you have instructions for reproducing the whole thing from source, without a binary download? If parts of it are not open source we may be able to arrange to debug it privately, but in any case a smaller test case would help immensely.

[TimNN]

@eddyb: Doesn't that nightly already default to orbit? As far as I can tell the switch happened between nightly-2016-08-02 and nightly-2016-08-03.

[eddyb]

@TimNN Ah you are indeed correct. Well, that's one down, 99 potential causes to go.

[eddyb]

Everyone involved in this thread: if you haven't already, check out http://rr-project.org/.
Reverse debugging might be the only affordable way to nail this bug down.

[siddontang]

Hi @brson

We tried to reproduce this coredump with a simple test, but failed. 😭
You can't use tikv alone, must with PD and TiDB.

PD and TiDB are both written with Go, and are all open source under Apache-2, so you can use the pd-server and tidb-server binaries freely.

If you want to build yourself, you must install go 1.6+ first (https://golang.org/doc/install), then:

git clone https://github.com/pingcap/tidb.git $GOPATH/src/github.com/pingcap/tidb
cd $GOPATH/src/github.com/pingcap/tidb
make 
# the tidb-server is installed in $GOPATH/src/github.com/pingcap/tidb/bin directory 

git clone https://github.com/pingcap/pd.git $GOPATH/src/github.com/pingcap/pd
cd $GOPATH/src/github.com/pingcap/pd
make
# the pd-server is installed in $GOPATH/src/github.com/pingcap/pd/bin directory 

[pnkfelix]

@siddontang what is the source to the bank program? The zip file only has an executable.

[pnkfelix]

@siddontang I cannot reproduce the scenario; when I try to run the bank program, I get the following output:

https://gist.github.com/pnkfelix/4c87f20badee2c5110c23005984830cd

and the tidb-server terminates with the output:

2016/10/28 18:55:27 server.go:167: [error] accept error accept tcp [::]:4000: accept4: too many open files
2016/10/28 18:55:28 main.go:145: [error] accept tcp [::]:4000: accept4: too many open files

(The other two services keep running...)

I second @eddyb 's suggestion of trying to use rr to look into this on your end.

[siddontang]

Hi @pnkfelix
Sorry that I forgot give you the bank source, it is here https://gist.github.com/siddontang/2fb9cd4cf736199b0b017f7263c41ab4

In the bank case, we will create at lease concurrency(default is 10000) connections.
Seem that you should use ulimit -n 10240 to ensure we can open so many connections :-)

Of course, you can use another concurrency in bank like ./bank --concurrency 2000, this will only use 2000 connections.