Specify that encryptfs doesn't work in the error message

[presidentbeef]

Installed per directions, and ran rethinkdb:

$ rethinkdb 
info: Creating directory rethinkdb_data
info: Creating a default database for your convenience. (This is because you ran 'rethinkdb' without 'create', 'serve', or '--join', and the directory 'rethinkdb_data' did not already exist.)
error: Error in arch/runtime/thread_pool.cc at line 323:
error: Segmentation fault from reading the address (nil).
error: Backtrace:
error: Sat Nov 10 12:38:29 2012

       1: rethinkdb() [0x8d0a82]
       2: rethinkdb() [0x8aceb4]
       3: rethinkdb() [0x483284]
       4: +0xfcb0 at 0x7fe53d8b9cb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
       5: rethinkdb() [0x48804a]
       6: rethinkdb() [0x68cf41]
       7: rethinkdb() [0x692541]
       8: rethinkdb() [0x6b3d37]
       9: rethinkdb() [0x6b52be]
       10: rethinkdb() [0x4813fb]
       11: rethinkdb() [0x481488]
       12: rethinkdb() [0x47f30e]
error: Exiting.
[13082] worker: Couldn't read job function: end-of-file received
[13080] worker: Couldn't read job function: end-of-file received
[13084] worker: Couldn't read job function: end-of-file received
[13082] worker: Failed to accept job, quitting.
[13080] worker: Failed to accept job, quitting.
[13084] worker: Failed to accept job, quitting.
Segmentation fault

This is on Ubuntu 12.04 LTS (GNU/Linux 2.6.32-316-ec2 x86_64)

[jdoliner]

Hmm I think we're going to probably need to get a core dump here to do
anything. Any chime in with the easiest way to get a version of rethinkdb
with symbols? Also what sort of file system is this running on.

On Saturday, November 10, 2012, Justin wrote:

Installed per directions, and ran rethinkdb:

$ rethinkdb
info: Creating directory rethinkdb_data
info: Creating a default database for your convenience. (This is because you ran 'rethinkdb' without 'create', 'serve', or '--join', and the directory 'rethinkdb_data' did not already exist.)
error: Error in arch/runtime/thread_pool.cc at line 323:
error: Segmentation fault from reading the address (nil).
error: Backtrace:
error: Sat Nov 10 12:38:29 2012

   1: rethinkdb() [0x8d0a82]
   2: rethinkdb() [0x8aceb4]
   3: rethinkdb() [0x483284]
   4: +0xfcb0 at 0x7fe53d8b9cb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
   5: rethinkdb() [0x48804a]
   6: rethinkdb() [0x68cf41]
   7: rethinkdb() [0x692541]
   8: rethinkdb() [0x6b3d37]
   9: rethinkdb() [0x6b52be]
   10: rethinkdb() [0x4813fb]
   11: rethinkdb() [0x481488]
   12: rethinkdb() [0x47f30e]

error: Exiting.
[13082] worker: Couldn't read job function: end-of-file received
[13080] worker: Couldn't read job function: end-of-file received
[13084] worker: Couldn't read job function: end-of-file received
[13082] worker: Failed to accept job, quitting.
[13080] worker: Failed to accept job, quitting.
[13084] worker: Failed to accept job, quitting.
Segmentation fault

This is on Ubuntu 12.04 LTS (GNU/Linux 2.6.32-316-ec2 x86_64)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/18.

[presidentbeef]

It's on ext3.

[mlucy]

I think apt-get install rethinkdb-dbg should install the debugging symbols for use with gdb and an unstripped binary at /usr/lib/debug/usr/bin/rethinkdb.

A backtrace + core file from that unstripped binary would make debugging much easier. You can run ulimit -c unlimited to turn core file generation on if it isn't already.

[samstokes]

Could one cause of this be if the user's home directory is ecryptfs (which is available as a default option on Ubuntu)? I got a similar error on Ubuntu 11.10 (it printed different output - below - so not sure if it's actually the same error, but it had the same ultimate symptom that the initial rethinkdb command failed. I tried again specifying -d /var/data/sam/rethinkdb_data (/var is ext4) and it seems to have worked.

The error I got:

$ rethinkdb create
info: Our machine ID: <blah>
error: Inaccessible database file: "rethinkdb_data/metadata": Invalid argument
       Some possible reasons:
       - the database file couldn't be created or opened for reading and writing
       - the database file is located on a filesystem that doesn't support O_DIRECT open flag (e.g. in case when the filesystem is working in journaled mode)
       - user which was used to start the database is not an owner of the file
Crashing while already crashed. Printing error message to stderr.
Segmentation fault from reading the address (nil).Trace/breakpoint trap

[coffeemug]

Yep, rethink doesn't work on encrypted volumes (there isn't anything we can do about it -- you can't run with direct io on encrypt fs). We'd still appreciate a core file for the original bug so we can exit with a meaningful error message.

[samstokes]

I tried installing rethinkdb-dbg, but /usr/lib/debug/usr/bin/rethinkdb wasn't executable, and after I chmod +xed it, I get:

$ /usr/lib/debug/usr/bin/rethinkdb
bash: /usr/lib/debug/usr/bin/rethinkdb: cannot execute binary file

[frank-trampe]

Sorry about that. The unstripped binaries in Debian dbg packages are not exectutable. (We do not know why.) If you run the main binary in gdb, you ought to get a meaningful backtrace if symbols are installed.

[Tryneus]

So, the file at /usr/lib/debug/usr/bin/rethinkdb is a symbol file, not an executable. You can specify it in gdb using -s /usr/lib/debug/usr/bin/rethinkdb on the command-line, or symbol /usr/lib/debug/usr/bin/rethinkdb inside gdb, if it doesn't load it at startup. I'm not sure what tells gdb where to look for the symbol file automatically.

It's pretty important that this symbol file is the result of the same build of rethinkdb, or none of the symbols will make sense. I don't think we have a symbol file for the version that the original stack trace in this issue came from, so if you could produce one with the latest released version, that would help.

[presidentbeef]

Here's a core dump: http://dl.dropbox.com/u/12108749/core.28866

I have no idea if the symbols match, I just used the provided symbol file above.

[Tryneus]

So, the symbols I'm getting back in the backtrace don't make much sense, but the one at the top of the stack could very well have segfaulted:

#0  0x000000000048804a in get_ips () at arch/io/network.cc:900

where

    int addr_res = getifaddrs(&if_addrs);
    guarantee_err(addr_res == 0, "getifaddrs failed, could not determine local ip addresses");

    for (ifaddrs *p = if_addrs; p != NULL; p = p->ifa_next) {
>>>     if (p->ifa_addr->sa_family == AF_INET) {
            if (!(p->ifa_flags & IFF_LOOPBACK)) {

We were assuming that getifaddrs() would not return a structure with ifa_addr = NULL, but according to the documentation, that's not always true.

I fixed this in commit f872a0b, and it should be published soon.

[Tryneus]

Ok, version 1.2.5 is up, and should have the fix I just mentioned. Let me know if this is still reproducible.

[coffeemug]

Ok, the only thing that remains to be done here is to clarify the error message to include encrypted file systems. If other bugs pop up, we'll open a new issue. Assigning to self.

[presidentbeef]

Um...not sure why the title of this issue was changed? I never said I was using an encrypted file system.

I am seeing a different problem now, so I guess this is fixed.

[samstokes]

Sorry, I may have confused the issue by posting mine here (which was caused by ecryptfs).

[coffeemug]

Yes, I think multiple posts confused it a little bit. @presidentbeef -- the issue you've been experiencing has been fixed. Try running sudo apt-get update, sudo apt-get upgrade rethinkdb. If you encounter it again, please open another issue.

[srh]

As for making the error message mentioning encrypted file systems, we are waiting on @wmrowan to finish a code review.

[coffeemug]

This is fixed in next.