Skip to content
This repository has been archived by the owner on Oct 12, 2022. It is now read-only.

Server crashes during benchmarking. #1

Closed
dpc opened this issue Jul 25, 2015 · 6 comments
Closed

Server crashes during benchmarking. #1

dpc opened this issue Jul 25, 2015 · 6 comments

Comments

@dpc
Copy link

dpc commented Jul 25, 2015

Hi,

I wanted to use your server as benchmark point for mioco. Unfortunately it does not survive the benchmark.

[futex:mio/mob] (master)% cargo run --release --bin mob-server
   Compiling slab v0.1.1
   Compiling libc v0.1.8
   Compiling bitflags v0.1.1
   Compiling bytes v0.2.10
   Compiling regex-syntax v0.2.1
   Compiling nix v0.3.9
   Compiling winapi v0.1.23
   Compiling log v0.3.1
   Compiling memchr v0.1.3
   Compiling clock_ticks v0.0.5
   Compiling aho-corasick v0.3.0
   Compiling mio v0.4.1
   Compiling regex v0.1.41
   Compiling env_logger v0.3.1
   Compiling mob v0.1.0 (file:///home/dpc/lab/rust/mio/mob)
     Running `target/release/mob-server`
ERROR:mob_server::connection: Failed to send buffer for Token(9), error: Connection reset by peer (os error 104)
ERROR:mob_server::connection: Failed to send buffer for Token(10), error: Connection reset by peer (os error 104)
ERROR:mob_server::connection: Failed to send buffer for Token(11), error: Connection reset by peer (os error 104)
ERROR:mob_server::connection: Failed to send buffer for Token(11), error: Connection reset by peer (os error 104)
thread '<main>' panicked at 'invalid index', ../src/libcore/option.rs:330
An unknown error occurred

To learn more, run the command again with --verbose.

The benchmark itself is: https://github.com/dpc/benchmark-echo , and as it works fine against mioco, libev, and node tcp server implementation, I guess it's OK. I start it with ./benchmark.sh 8000

@dpc dpc mentioned this issue Jul 26, 2015
8 tasks
@tryge
Copy link

tryge commented Aug 15, 2015

From your benchmark, I guess you expect an echo server. This, however is a multi-echo server i.e. if you write something on one connection it gets echoed to every existing connection.

Crashing is not nice, I guess mob should handle reset connections better, but your benchmark is not suited for this server, I think, unless I missed something :-)

@hjr3
Copy link
Owner

hjr3 commented Aug 15, 2015

@dpc thanks for the feedback. This is basically a use after free error. The read connection gets reset, removed from the slab and then another event for that token is sent from mio. I am removing the I have not been able to track it down, though I had to take a break from digging into it. I am working on it now though.

@dpc
Copy link
Author

dpc commented Aug 16, 2015

@tryge: Oh, I see. Thanks for pointing that out.

@hjr3: It might be one of:
tokio-rs/mio#184
tokio-rs/mio#219
or something similar.

@hjr3
Copy link
Owner

hjr3 commented Aug 16, 2015

It is most certainly tokio-rs/mio#219 . Thank you for the tip.

hjr3 added a commit that referenced this issue Aug 22, 2015
A single event loop tick may contain multiple events for a token. If the
connection is reset on the first event, subsequent events will still try
to access that token/connection during the event loop tick. The strategy
now is to mark a connection as reset and remove them only when the event
loop tick is finished.

See tokio-rs/mio#219 for more details.

Fixes #1
@hjr3 hjr3 closed this as completed in #2 Aug 22, 2015
@hjr3 hjr3 reopened this Aug 23, 2015
@hjr3
Copy link
Owner

hjr3 commented Aug 23, 2015

I thought this was fixed, but it seems like I am still receiving events after I reset a connection.

TRACE:mio::event_loop: event=IoEvent { kind: Readable | Writable | Hup, token: Token(2) }
DEBUG:mob_server::server: events = Readable | Writable | Hup
TRACE:mob_server::server: Hup event for Token(2)
DEBUG:mob_server::server: pending reset connection; token=Token(2)
TRACE:mio::timer: tick_to; now=166; tick=167
TRACE:mob_server::server: Handling end of tick
DEBUG:mob_server::server: reset connection; token=Token(2)
TRACE:mio::event_loop: event loop tick
TRACE:mio::event_loop: event=IoEvent { kind: Readable | Writable, token: Token(2) }
DEBUG:mob_server::server: events = Readable | Writable
TRACE:mob_server::server: Write event for Token(2)
thread '<main>' panicked at 'invalid index', ../src/libcore/option.rs:331

@hjr3
Copy link
Owner

hjr3 commented Oct 18, 2015

This is fixed. It was an underlying bug in the way mio worked with kqueue.

@hjr3 hjr3 closed this as completed Oct 18, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants