Floating point exception (core dumped)

[slava-vishnyakov]

Cool database, guys! (And I believe this is the first time I say this in my life, like "cool database!" 👍 )

But it crashed within 15 minutes of playing with it :(

Well, basically I've created a table (tv_shows), added a second server, sharded table into two tables. I've started inserting as much data as possible, then I Ctrl+C'ed the second server. Declared it dead from dashboard. Then I started the second server again. It said to stop the "dead" server. I stopped it. And got this:

info: Server ready
info: Connected to server "Riker" de049dca-22f3-4363-8fd0-e470f5e84606
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"00000000-0000-0000-0000-000000000000":1},"ack_expectations":{"00000000-0000-0000-0000-000000000000":1}}}}
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"shards":["[\"\",\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\"]","[\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\",null]"],"primary_pinnings":{"[\"\",\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\"]":null,"[\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\",null]":null},"secondary_pinnings":{"[\"\",\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\"]":[],"[\"S8362cf0e-0472-4e83-b42e-8b1c15c17891\",null]":[]}}}}
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"00000000-0000-0000-0000-000000000000":0},"ack_expectations":{"00000000-0000-0000-0000-000000000000":1}}}}
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"00000000-0000-0000-0000-000000000000":0},"ack_expectations":{"00000000-0000-0000-0000-000000000000":1}}}}
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"shards":["[\"\",null]"],"primary_pinnings":{"[\"\",null]":null},"secondary_pinnings":{"[\"\",null]":[]}}}}
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"00000000-0000-0000-0000-000000000000":1},"ack_expectations":{"00000000-0000-0000-0000-000000000000":1}}}}
info: Disconnected from server "Riker" de049dca-22f3-4363-8fd0-e470f5e84606
info: Deleting /machines/de049dca-22f3-4363-8fd0-e470f5e84606
error: Namespace 2cc90e8a-83af-4280-b5c6-066942ad270c has unsatisfiable goals
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"shards":["[\"\",null]"],"primary_pinnings":{"[\"\",null]":null},"secondary_pinnings":{"[\"\",null]":[]}}}}
error: Namespace 2cc90e8a-83af-4280-b5c6-066942ad270c has unsatisfiable goals
info: Applying data {"datacenters":{"new":{"name":"Main"}}}
error: Namespace 2cc90e8a-83af-4280-b5c6-066942ad270c has unsatisfiable goals
info: Applying data {"machines":{"cf153bff-5cfa-4a11-b992-d3af4b24dd84":{"datacenter_uuid":"827da5e3-f416-42c5-aa86-ecb13ee8be0b"}}}
error: Namespace 2cc90e8a-83af-4280-b5c6-066942ad270c has unsatisfiable goals
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"827da5e3-f416-42c5-aa86-ecb13ee8be0b":0},"ack_expectations":{"827da5e3-f416-42c5-aa86-ecb13ee8be0b":0}}}}
error: Namespace 2cc90e8a-83af-4280-b5c6-066942ad270c has unsatisfiable goals
info: Applying data {"rdb_namespaces":{"2cc90e8a-83af-4280-b5c6-066942ad270c":{"replica_affinities":{"00000000-0000-0000-0000-000000000000":0},"ack_expectations":{"00000000-0000-0000-0000-000000000000":1}}}}

Here I start the second machine and slightly later Ctrl+C'ed it.

info: Connected to server <ghost machine> de049dca-22f3-4363-8fd0-e470f5e84606
info: Disconnected from server <ghost machine> de049dca-22f3-4363-8fd0-e470f5e84606
[3850] worker: Couldn't read job function: end-of-file received
[3850] worker: Failed to accept job, quitting.
[3856] worker: Couldn't read job function: end-of-file received
[3856] worker: Failed to accept job, quitting.
Floating point exception (core dumped)

Still, RethinkDB is awesome!

[slava-vishnyakov]

[truncated... Moved to Issue #15 ..]

Also, is it normal that I get only 200 writes per seconds (this is VirtualBox on regular HDD, not SSD)? Even in bulk mode?

[coffeemug]

@slava-vishnyakov -- we've seen the end-of-file bug before, but it was quite rare and we didn't pin it down. It's definitely a different issue, could you open a new one? (Also, thanks for reporting these, it's immensely helpful!)

As far as 200 writes per second -- see limitations in https://github.com/rethinkdb/rethinkdb/blob/rethinkdb-next/NOTES -- currently we flush on every write with almost no batching, so we get really poor hdd write performance. This should be resolved in the coming release.

[Tryneus]

I should probably mention that the extproc end-of-file error is most likely just the result of a rethinkdb process shutting down improperly. The real error is elsewhere.

[jdoliner]

End of file messages is really just java script being overly chatty about
the fact that its exiting. I think this is actually a dupe of 15 but let's
confirm that.

On Saturday, November 10, 2012, Tryneus wrote:

I should probably mention that the extproc end-of-file error is most
likely just the result of a rethinkdb process shutting down improperly. The
real error is elsewhere.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/13#issuecomment-10254127.

[mlucy]

@slava-vishnyakov: Do you happen to have a core dump from any of these crashes?

[slava-vishnyakov]

@mlucy: Sadly, no, tried to recraete it today with core dumps enabled, but not luck.

[slava-vishnyakov]

@mlucy: Sent to your email

[coffeemug]

@slava-vishnyakov -- thanks, we'll look at it tomorrow. Really appreciate your patience!

[mlucy]

@slava-vishnyakov -- Do you remember what version of RethinkDB you were running? In particular, did you download it with apt-get or did you build it from source? If the former, what ubuntu distro are you on, or if the latter, do you know the commit hash?

(We had a small problem where we weren't properly generating the RethinkDB version string that we put in the binary, so we're having some trouble matching core dumps with symbol files.)

Thanks!

[slava-vishnyakov]

@mlucy It was the apt-get version, released on the day you got to Hacker News.

$ rethinkdb --version
rethinkdb 

:)

[mlucy]

@slava-vishnyakov -- Thanks! (And thanks for providing so much info).

As far as I can tell the core files don't match any of the symbols we have on launchpad. I talked to @frank-trampe and he said that the very first versions we released on launch day didn't have associated symbol packages. So there's really no good way forward on this unless it crops up again with a version that we have symbols for (i.e. anything not released on the first day).

[coffeemug]

@slava-vishnyakov -- thanks so much for the bug report. I'm going to close it as jammed because unfortunately we can't proceed on it unless it's reproduced on latest rethink version. It's totally our fault that the early build didn't have debugging symbols -- unfortunately we didn't line up all the pins for the release.

We're also working on #41 to make the report process much more smooth and streamlined.