Guarantee failed: [size > 0 && _val[0] == resource_parts_sep_char[0]] resource path must start with a '/' Edit

[neumino]

On a fresh ubuntu 12.04 install (not on a vm). I got a crash

error: Error in http/http.cc at line 37:
error: Guarantee failed: [size > 0 && _val[0] == resource_parts_sep_char[0]] resource path must start with a '/'
error: Backtrace:
error: Sun Nov 11 22:57:10 2012

       1: rethinkdb() [0x515d92]
       2: rethinkdb() [0x512f54]
       3: rethinkdb() [0x5e7e98]
       4: rethinkdb() [0x5ec0fc]
       5: rethinkdb() [0x5ee213]
       6: rethinkdb() [0x5ef732]
       7: rethinkdb() [0x5fa2d5]
       8: rethinkdb() [0x606f6e]
error: Exiting.
Crashing while already crashed. Printing error message to stderr.
Segmentation fault from reading the address (nil).Trace/breakpoint trap

I got tons of warnings before

warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)

rethinkdb --version returns "rethinkdb"...

I was built from ppa this morning, not from source.

[jdoliner]

This maybe shouldn't be a guarantee since it seems to not like the http
request its receiving. On the other hand we should confirm this. I'm not
looking at the code right now.

On Sunday, November 11, 2012, Michel wrote:

On a fresh ubuntu 12.04 install (not on a vm). I got a crash

error: Error in http/http.cc at line 37:
error: Guarantee failed: [size > 0 && _val[0] == resource_parts_sep_char[0]] resource path must start with a '/'
error: Backtrace:
error: Sun Nov 11 22:57:10 2012

   1: rethinkdb() [0x515d92]
   2: rethinkdb() [0x512f54]
   3: rethinkdb() [0x5e7e98]
   4: rethinkdb() [0x5ec0fc]
   5: rethinkdb() [0x5ee213]
   6: rethinkdb() [0x5ef732]
   7: rethinkdb() [0x5fa2d5]
   8: rethinkdb() [0x606f6e]

error: Exiting.
Crashing while already crashed. Printing error message to stderr.
Segmentation fault from reading the address (nil).Trace/breakpoint trap

I got tons of warnings before

warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)
warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)

rethinkdb --version returns "rethinkdb"...

I was built from ppa this morning, not from source.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40.

[mlucy]

My faith in software is restored! I have a sane backtrace (from 1.2.5 precise):

format_backtrace(bool) + 130 in section .text
report_fatal_error(char const*, int, char const*, ...) + 276 in section .text
http_req_t::resource_t::assign(char const*, unsigned long) + 56 in section .text
tcp_http_msg_parser_t::parse(linux_tcp_conn_t*, http_req_t*, signal_t*) + 796 in section .text
http_server_t::handle_conn(scoped_ptr_t<linux_tcp_conn_descriptor_t> const&, auto_drainer_t::lock_t) + 387 in section .text
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf2<void, http_server_t, scoped_ptr_t<linux_tcp_conn_descriptor_t> const&, auto_drainer_t::lock_t>, boost::_bi::list3<boost::_bi::value<http_server_t*>, boost::arg<1>, boost::_bi::value<auto_drainer_t::lock_t> > >, void, scoped_ptr_t<linux_tcp_conn_descriptor_t>&>::invoke(boost::detail::function::function_buffer&, scoped_ptr_t<linux_tcp_conn_descriptor_t>&) + 98 in section .text
linux_nonthrowing_tcp_listener_t::handle(int) + 53 in section .text
coro_t::run() + 46 in section .text

[mlucy]

So, I have a fix that does something smarter than failing a guarantee at http://newton:8080/ui#review:id=60 . Unfortunately I don't know how to track down whatever made the bad request. @neumino -- if I push the guarantee fix, do you think you could find the source of the request?

[neumino]

Euh, I'm not sure to understand what you are talking about. Are you refering to these warnings

warn: Error in collecting system stats (on demand): Could not open '/proc/net/dev': No such file or directory (errno = 2)

If yes, I just did a little more testing. The warnings appear only when I'm doing a request to /ajax/stats (I did request this page manually).
/proc/net is a symbolic link to /proc/self/net
/proc/self/net/dev does exist but has the following rights
-r--r--r-- 1 root root 0 Nov 13 09:42 dev

Also I get some "errors" in /ajax/stats like for example

"bytes_sent": {
"avg": "-",
"max": "-",
"min": "-",
"per_sec": "0.00000000"
}

A little more about my server

Linux ***** 3.2.13-grsec-xxxx-grs-ipv6-64 #1 SMP Thu Mar 29 09:48:59 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Changing the chmod didn't solve the problem (I added rwx for everyone - and I did restart the server).
Is it because of the symbolic link?

[mlucy]

Ah, I see, I was confused about the scope of the problem.

The problem I was addressing was the guarantee fail that resulted in a crash. We now fail to parse the tcp message if the path is malformed rather than failing a guarantee. The next question is "what was sending the malformed path?", but if you were point to /ajax/stats by hand maybe you just mistyped.

I'll work on the proc/net/dev stuff.

[neumino]

The web ui triggers the same warnings. If I don't have a web ui opened, I don't get any warnings.

[mlucy]

The guarantee failure and the warnings are two separate problems I think. If the web UI is causing the guarantee to fail, it's sending a malformed path and needs to be fixed; if the web UI is causing these /proc/net/dev warnings, that's something we need to fix on the server.

[neumino]

About the guarantee failure, I got a second crash two days ago, but this time I almost certain that I didn't have a web ui opened.

About the warnings, the web UI is causing the /proc/net/dev warnings.

[mlucy]

So, theoretically the call to open should go through symlinks just fine. I'm installing a 12.04 VM to see if I can reproduce this.

[mlucy]

So, here's the current state of this bug:

• The guarantee failure seems fixed.
• I can't reproduce the /proc/net/dev thing at all on an Ubuntu 12.04 VM. @neumino -- do you still get it with the latest version from launchpad? If so, could I come look at this on whatever machine you can reproduce it on tomorrow?
• The empty entries in /ajax/stat seems to be a problem with the perfmons. I've tried to debug it but they're more complicated than I was expecting and I can't really figure out what's supposed to be happening. For some reason the bytes_sent entry only seems to log the number of bytes sent inside of connectivity_cluster_t::send_message while the /ajax/stat request is outstanding. I.e. if I make several requests in rapid succession before the first one returns then I can get some numbers there, but otherwise it's always empty.

@jdoliner -- I seem to remember you knowing something about the perfmons. Is there any chance you'd be willing to take this bug, or alternately pair program it with me so that I learn how the perfmons work?

[neumino]

I still get the warnings about /proc/net/dev after an update.

I can give you access to my server tomorrow. It's probably going to be easier that way.

[mlucy]

Alright, I think I know the root of the /proc/net problem. It looks like you're running a kernel with grsecurity patches, which prevents non-root users from reading /proc/net by default. There's theoretically a configuration option CONFIG_GRKERNSEC_PROC_USER somewhere that controls this, but I didn't want to start messing around with your server config.

@srh - OS X doesn't have /proc at all, right? What are we planning to do for the port? Maybe we should just do that by default and solve this problem for people with hardened kernels.

That just leaves the perfmon stuff, which I'd like to talk to @jdoliner about. I'll try to wake up earlier tomorrow and do that.

[coffeemug]

What's the state of affairs here? It looks like the original bug has been fixed, in which case we should probably create a new one for /proc stuff (if that's still an issue).

[mlucy]

The guarantee failure is (I think) fixed. The /proc stuff will be fixed as a natural consequence of the OS X port. The weird perfmon stuff I need to talk to Joe about, but yeah, maybe it should be in a new issue.

[coffeemug]

@mlucy -- what's the state of affairs here? As far as I understand, we should close this and create a new perfmon-related issue (though I'm not sure what we can do about that if we don't have permissions, other than perhaps giving a better warning message and referring to gsecurity).

[jdoliner]

I think we should close this and make a new issue.

On Mon, Nov 19, 2012 at 2:20 PM, coffeemug notifications@github.com wrote:

@mlucy https://github.com/mlucy -- what's the state of affairs here? As
far as I understand, we should close this and create a new perfmon-related
issue (though I'm not sure what we can do about that if we don't have
permissions, other than perhaps giving a better warning message and
referring to gsecurity).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/40#issuecomment-10533842.

[mlucy]

No objections to closing and reopening. The perfmon problem is the:

"bytes_sent": {
"avg": "-",
"max": "-",
"min": "-",
"per_sec": "0.00000000"
}

stuff @neumino was talking about, which seems to happen for me even when we can read from /proc.

[coffeemug]

Ok! Could you close this one and open a new one?

[jdoliner]

@mlucy could you reference the commit that fixes this? Or a commit in which this is definitely fixed if no single commit exists?

[mlucy]

@jdoliner -- bcef3ee