apachebench hangs on httpd #3

nyh · 2014-10-14T12:23:56Z

When running Apachebench (ab) on seastar's httpd application, it hangs immediately.

I think the problem is that httpd doesn't correctly implement the finer details of the HTTP protocol.

The concrete issue is that our server doesn't close the connection, as it should, when connected by an HTTP/1.0 client. RFC2616 states:

Clients and servers SHOULD NOT assume that a persistent connection is maintained for HTTP versions less than 1.1 unless it is explicitly signaled. See section 19.6.2 for more information on backward compatibility with HTTP/1.0 clients.

As can be seen with "ab -v4", ApacheBench does use HTTP 1.0.

There seem to be other problems in our correct support of the HTTP protocol. E.g, I noticed the connection hangs if there is no HTTP/... on the GET line. This is wrong - HTTP/0.9 should be assumed, and definitely the request should not hang.

The text was updated successfully, but these errors were encountered:

asias · 2014-10-22T06:07:29Z

Change the response from HTTP/1.0 to HTTP1.1. ab still hangs.

asias@hj:osv$ ab -v4 http://192.168.66.123:10000/
This is ApacheBench, Version 2.3 <$Revision: 1554214 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.66.123 (be patient)...INFO: GET header ==
---
GET / HTTP/1.0
Host: 192.168.66.123:10000
User-Agent: ApacheBench/2.3
Accept: */*


---
LOG: header received:
HTTP/1.0 200 OK
Content-Length: 87
Content-Type: text/html

<html><head><title>this is the future</title></head><body><p>Future!!</p></body></html>
LOG: Response code = 200
apr_pollset_poll: The timeout specified has expired (70007)

asias · 2014-10-27T06:25:43Z

@nyh I did not see the issue any more. Can you please confirm so we can close this issue?

nyh · 2014-10-27T07:30:49Z

I tried this on the default (POSIX) stack, ab still hangs...

Moreover, HTTP 0.9 requests (i.e., just "GET /", without the HTTP/1.0 part) now crash. The response is returned correctly, but close() crashes. To reproduce this, try echo GET / | nc 127.0.0.1 10000. The stack trace I see in gdb:

#0  0x0000000000415f6f in close (this=0x6000001c4830) at ./core/reactor.hh:558
#1  close (this=0x6000001c4830) at ./core/reactor.hh:607
#2  http_server::connection::generate_response (this=0x6000001c4800, 
    req=std::unique_ptr<http_request> containing 0x60000005cf80)
    at apps/httpd/httpd.cc:166
#3  0x0000000000416379 in http_server::connection::read()::{lambda()#1}::operator()() const (__closure=0x7fffffffd010) at apps/httpd/httpd.cc:77
#4  0x000000000041664f in _ZZN7promiseIIEE8scheduleIZN6futureIIEE4thenIZN11http_server10connection4readEvEUlvE_EENSt9result_ofIFT_vEE4typeEOS9_NSt9enable_ifIXsr9is_futureIISC_EE5valueEPvE4typeEEUlRT_E_EEvSD_EN15task_with_state3runEv ()
    at ./core/apply.hh:36
#5  0x000000000044ac88 in reactor::run (this=0x7ffff7d1f798)
    at core/reactor.cc:334
#6  0x000000000040839b in run<main(int, char**)::<lambda()> > (
    func=<optimized out>, av=<optimized out>, ac=<optimized out>, 
    this=<optimized out>) at ./core/app-template.hh:62
#7  main (ac=-10416, av=0x7fffffffd770) at apps/httpd/httpd.cc:193

asias · 2014-10-27T10:38:30Z

Thanks for trying. I will take a look.

nyh · 2014-10-27T13:26:10Z

I just verified that ab does not hang with the native stack, just (apparently) with the Posix stack.
The HTTP 0.9 crash ("GET" without a version number) happens also for the native stack.

nyh · 2014-10-29T07:56:35Z

Unfortunately, with the Posix stack, things still don't work - when I run on the latest master:

build/release/apps/httpd/httpd    # default is Posix stack
ab http://127.0.0.1:10000/     # default is just one request

I get a crash:

#0  reactor::get_epoll_future(pollable_fd_state&, promise<> pollable_fd_state::*, int) (this=0x7ffff7b137a0, pfd=..., pr=pr@entry=&pollable_fd_state::pollin, 
    event=event@entry=1) at core/reactor.cc:70
#1  0x000000000044f6bf in readable (fd=..., this=<optimized out>)
    at core/reactor.cc:89
#2  reactor::read_some (this=<optimized out>, fd=..., buffer=<optimized out>, 
    len=<optimized out>) at core/reactor.hh:674
#3  0x000000000045f50e in read_some (size=8192, buffer=0x600000405000 "", 
    this=<optimized out>) at ./core/reactor.hh:844
#4  net::posix_data_source_impl::get (this=0x600000074460)
    at net/posix-stack.cc:78
#5  0x00000000004142b7 in get (this=<optimized out>) at ./core/reactor.hh:537
#6  input_stream<char>::consume<http_request_parser> (
    this=this@entry=0x6000001c4808, consumer=...) at ./core/reactor.hh:783
#7  0x0000000000416a50 in http_server::connection::read (
    this=this@entry=0x6000001c4800) at apps/httpd/httpd.cc:87
#8  0x0000000000416456 in http_server::connection::read()::{lambda()#1}::operator()() const (__closure=<optimized out>) at apps/httpd/httpd.cc:85
#9  0x000000000041670f in _ZZN7promiseIIEE8scheduleIZN6futureIIEE4thenIZN11http_server10connection4readEvEUlvE_EENSt9result_ofIFT_vEE4typeEOS9_NSt9enable_ifIXsr9is_futureIISC_EE5valueEPvE4typeEEUlRT_E_EEvSD_EN15task_with_state3runEv ()
    at ./core/apply.hh:36
#10 0x000000000044a608 in reactor::run (this=this@entry=0x7ffff7b137a0)
    at core/reactor.cc:334
#11 0x000000000040840b in run<main(int, char**)::<lambda()> > (
    func=<optimized out>, av=<optimized out>, ac=<optimized out>, 
    this=0x7fffffffd8f0) at ./core/app-template.hh:62
#12 main (ac=<optimized out>, av=<optimized out>) at apps/httpd/httpd.cc:193

asias · 2014-10-29T08:04:18Z

@nyh I can reproduce it here too.

On Wed, Oct 29, 2014 at 3:56 PM, nyh notifications@github.com wrote:

Unfortunately, with the Posix stack, things still don't work - when I run
on the latest master:

build/release/apps/httpd/httpd # default is Posix stack
ab http://127.0.0.1:10000/ # default is just one request

I get a crash:

#0 reactor::get_epoll_future(pollable_fd_state&, promise<> pollable_fd_state::_, int) (this=0x7ffff7b137a0, pfd=..., pr=pr@entry=&pollable_fd_state::pollin,
event=event@entry=1) at core/reactor.cc:70
#1 0x000000000044f6bf in readable (fd=..., this=)
at core/reactor.cc:89
#2 reactor::read_some (this=, fd=..., buffer=,
len=) at core/reactor.hh:674
#3 0x000000000045f50e in read_some (size=8192, buffer=0x600000405000 "",
this=) at ./core/reactor.hh:844
#4 net::posix_data_source_impl::get (this=0x600000074460)
at net/posix-stack.cc:78
#5 0x00000000004142b7 in get (this=) at ./core/reactor.hh:537
#6 input_stream::consume<http_request_parser> (
this=this@entry=0x6000001c4808, consumer=...) at ./core/reactor.hh:783
#7 0x0000000000416a50 in http_server::connection::read (
this=this@entry=0x6000001c4800) at apps/httpd/httpd.cc:87
#8 0x0000000000416456 in http_server::connection::read()::{lambda()#1}::operator()() const (__closure=) at apps/httpd/httpd.cc:85
#9 0x000000000041670f in ZZN7promiseIIEE8scheduleIZN6futureIIEE4thenIZN11http_server10connection4readEvEUlvE_EENSt9result_ofIFT_vEE4typeEOS9_NSt9enable_ifIXsr9is_futureIISC_EE5valueEPvE4typeEEUlRT_E_EEvSD_EN15task_with_state3runEv ()
at ./core/apply.hh:36
#10 0x000000000044a608 in reactor::run (this=this@entry=0x7ffff7b137a0)
at core/reactor.cc:334
#11 0x000000000040840b in run<main(int, char*)::<lambda()> > (
func=, av=, ac=,
this=0x7fffffffd8f0) at ./core/app-template.hh:62
#12 main (ac=, av=) at apps/httpd/httpd.cc:193

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

Asias

asias · 2014-11-02T07:36:02Z

@nyh

With

commit b0b36c2
Author: Avi Kivity avi@cloudius-systems.com
Date: Thu Oct 30 14:09:47 2014 +0200

httpd: simplify connection termination

I see no problems now with both native and posix stack.

On Wed, Oct 29, 2014 at 4:03 PM, Asias He asias@cloudius-systems.com
wrote:

@nyh I can reproduce it here too.

On Wed, Oct 29, 2014 at 3:56 PM, nyh notifications@github.com wrote:

Unfortunately, with the Posix stack, things still don't work - when I run
on the latest master:

build/release/apps/httpd/httpd # default is Posix stack
ab http://127.0.0.1:10000/ # default is just one request

I get a crash:

#0 reactor::get_epoll_future(pollable_fd_state&, promise<> pollable_fd_state::_, int) (this=0x7ffff7b137a0, pfd=..., pr=pr@entry=&pollable_fd_state::pollin,
event=event@entry=1) at core/reactor.cc:70
#1 0x000000000044f6bf in readable (fd=..., this=)
at core/reactor.cc:89
#2 reactor::read_some (this=, fd=..., buffer=,
len=) at core/reactor.hh:674
#3 0x000000000045f50e in read_some (size=8192, buffer=0x600000405000 "",
this=) at ./core/reactor.hh:844
#4 net::posix_data_source_impl::get (this=0x600000074460)
at net/posix-stack.cc:78
#5 0x00000000004142b7 in get (this=) at ./core/reactor.hh:537
#6 input_stream::consume<http_request_parser> (
this=this@entry=0x6000001c4808, consumer=...) at ./core/reactor.hh:783
#7 0x0000000000416a50 in http_server::connection::read (
this=this@entry=0x6000001c4800) at apps/httpd/httpd.cc:87
#8 0x0000000000416456 in http_server::connection::read()::{lambda()#1}::operator()() const (__closure=) at apps/httpd/httpd.cc:85
#9 0x000000000041670f in ZZN7promiseIIEE8scheduleIZN6futureIIEE4thenIZN11http_server10connection4readEvEUlvE_EENSt9result_ofIFT_vEE4typeEOS9_NSt9enable_ifIXsr9is_futureIISC_EE5valueEPvE4typeEEUlRT_E_EEvSD_EN15task_with_state3runEv ()
at ./core/apply.hh:36
#10 0x000000000044a608 in reactor::run (this=this@entry=0x7ffff7b137a0)
at core/reactor.cc:334
#11 0x000000000040840b in run<main(int, char*)::<lambda()> > (
func=, av=, ac=,
this=0x7fffffffd8f0) at ./core/app-template.hh:62
#12 main (ac=, av=) at apps/httpd/httpd.cc:193

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

Asias

Asias

nyh · 2014-11-02T09:08:26Z

I confirm that "ab" now seems to work correctly on both native and Posix stacks. Thanks!

Fixes assert failure during ^C: #0 0x0000003e134348c7 in raise () from /lib64/libc.so.6 #1 0x0000003e1343652a in abort () from /lib64/libc.so.6 #2 0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_ at /usr/include/boost/intrusive/list.hpp:1263 #5 0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71 #6 reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287 #7 0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974 #8 ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86 #9 ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96

Fix hang with ab test on posix stack: ab -n 1000 http://127.0.0.1/ Fixes #3

Fixes assert failure during ^C: #0 0x0000003e134348c7 in raise () from /lib64/libc.so.6 #1 0x0000003e1343652a in abort () from /lib64/libc.so.6 #2 0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_ at /usr/include/boost/intrusive/list.hpp:1263 #5 0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71 #6 reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287 #7 0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974 #8 ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86 #9 ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96

Fix hang with ab test on posix stack: ab -n 1000 http://127.0.0.1/ Fixes #3

Fixes assert failure during ^C: #0 0x0000003e134348c7 in raise () from /lib64/libc.so.6 #1 0x0000003e1343652a in abort () from /lib64/libc.so.6 #2 0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_ at /usr/include/boost/intrusive/list.hpp:1263 #5 0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71 #6 reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287 #7 0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974 #8 ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86 #9 ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96

This reverts commit 33406cf. It introduces memory leaks: Direct leak of 24 byte(s) in 1 object(s) allocated from: #0 0x7fb773b389d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) #1 0x108f0d4 in seastar::reactor::poller::~poller() ../src/core/reactor.cc:2879 #2 0x11c1e59 in std::experimental::fundamentals_v1::_Optional_base<seastar::reactor::poller, true>::~_Optional_base() /usr/include/c++/9/experimental/optional:288 #3 0x118f2d7 in std::experimental::fundamentals_v1::optional<seastar::reactor::poller>::~optional() /usr/include/c++/9/experimental/optional:491 #4 0x108c5a5 in seastar::reactor::run() ../src/core/reactor.cc:2587 #5 0xf1a822 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) ../src/core/app-template.cc:199 #6 0xf1885d in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) ../src/core/app-template.cc:115 #7 0xeb2735 in operator() ../src/testing/test_runner.cc:72 #8 0xebb342 in _M_invoke /usr/include/c++/9/bits/std_function.h:300 #9 0xf3d8b0 in std::function<void ()>::operator()() const /usr/include/c++/9/bits/std_function.h:690 #10 0x1034c72 in seastar::posix_thread::start_routine(void*) ../src/core/posix.cc:52 #11 0x7fb7738804e1 in start_thread /usr/src/debug/glibc-2.30-13-g919af705ee/nptl/pthread_create.c:479 Reported-by: Rafael Avila de Espindola <espindola@scylladb.com>

…o_with Fixes failures in debug mode: ``` $ build/debug/tests/unit/closeable_test -l all -t deferred_close_test WARNING: debug mode. Not for benchmarking or production random-seed=3064133628 Running 1 test case... Entering test module "../../tests/unit/closeable_test.cc" ../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test" ../../src/testing/seastar_test.cc(43): info: check true has passed ==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! terminate called after throwing an instance of 'seastar::broken_promise' what(): broken promise ==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592) False positive error reports may follow For details see google/sanitizers#189 ================================================================= ==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0) #0 0x7fbf45f39d0b (/lib64/libasan.so.6+0xb3d0b) #1 0x7fbf45f57d4e (/lib64/libasan.so.6+0xd1d4e) #2 0x7fbf45f3e724 (/lib64/libasan.so.6+0xb8724) #3 0x7fbf45eb3e5b (/lib64/libasan.so.6+0x2de5b) #4 0x7fbf45eb51e8 (/lib64/libasan.so.6+0x2f1e8) #5 0x7fbf45eb7694 (/lib64/libasan.so.6+0x31694) #6 0x7fbf45f39398 (/lib64/libasan.so.6+0xb3398) #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b) #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116 #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121 #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810) #11 0x7fbf44bcd897 (/lib64/libgcc_s.so.1+0xf897) #12 0x7fbf44bcea5f (/lib64/libgcc_s.so.1+0x10a5f) #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8) #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148 #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb) #16 0x7fbf45e10a21 (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21) #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0) #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094) #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921) #20 0x7fbf45e5eae1 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1) #21 0x7fbf45e5ed31 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31) #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547) #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618) #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77 #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65 #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1) #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com> Message-Id: <20210406100911.12278-1-bhalevy@scylladb.com>

When we enable the sanitizer, we get following error while running iotune: ==86505==ERROR: LeakSanitizer: detected memory leaks Direct leak of 4096 byte(s) in 1 object(s) allocated from: #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7) scylladb#1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16 scylladb#2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21 scylladb#3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28 scylladb#4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13 scylladb#5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20 scylladb#6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9 scylladb#7 0x7f30ff8547bf (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c) SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s). This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error. And this is needed to test Scylla machine image with debug mode binary, since it tries to run iotune with the sanitizer and fails.

When we enable the sanitizer, we get following error while running iotune: ==86505==ERROR: LeakSanitizer: detected memory leaks Direct leak of 4096 byte(s) in 1 object(s) allocated from: #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7) #1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16 #2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21 #3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28 #4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13 #5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20 #6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9 #7 0x7f30ff8547bf (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c) SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s). This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error. And this is needed to test Scylla machine image with debug mode binary, since it tries to run iotune with the sanitizer and fails. Closes #1284

================================================================= ==93534==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f676abad010 in thread T0 #0 0x7f677fedfdc2 in __interceptor_free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 scylladb#1 0x7f6776e81776 in operator() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1242 scylladb#2 0x7f6776e884a4 in ~deferred_action /home/sitano/Projects/seastar/include/seastar/util/defer.hh:67 scylladb#3 0x7f6776e82d31 in try_create_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1253 scylladb#4 0x7f6776e83e39 in detect_io_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1288 scylladb#5 0x7f6776e85113 in seastar::reactor_backend_selector::available() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:2039 scylladb#6 0x7f67773be492 in seastar::reactor_options::reactor_options(seastar::program_options::option_group*) /home/sitano/Projects/seastar/src/core/reactor.cc:3782 scylladb#7 0x7f6776f04c32 in seastar::app_template::seastar_options::seastar_options() /home/sitano/Projects/seastar/src/core/app-template.cc:70 scylladb#8 0x7f6776f04040 in seastar_options_from_config /home/sitano/Projects/seastar/src/core/app-template.cc:58 scylladb#9 0x7f6776f0704f in seastar::app_template::app_template(seastar::app_template::config) /home/sitano/Projects/seastar/src/core/app-template.cc:108 scylladb#10 0x55a5ff597a6e in main /home/sitano/Projects/seastar/demos/hello-world.cc:30 scylladb#11 0x7f676f03984f (/usr/lib/libc.so.6+0x2384f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#12 0x7f676f039909 in __libc_start_main (/usr/lib/libc.so.6+0x23909) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#13 0x55a5ff597614 in _start (/home/sitano/Projects/seastar/build/debug/demos/hello-world_demo+0xf0614) (BuildId: cbf8a43d1565f3bf0a48cdfb43a11ce16d7a9bc1) Address 0x7f676abad010 is a wild pointer inside of access range of size 0x000000000001. SUMMARY: AddressSanitizer: bad-free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 in __interceptor_free ==93534==ABORTING

in main(), we creates an instance of `http_server_control` using new, but we never destroy it. this is identified by ASan ``` ==2190125==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3 #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27 #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl ude/c++/14/bits/invoke.h:61:14 #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_ 64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14 #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, . ..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14 #5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../ ../include/c++/14/tuple:2936:14 #6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea star/core/future.hh:2005:28 #7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0: :operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13 #8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable _function.hh:129:20 #9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16 #10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9 #11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43 #12 0x7f5d72f8a18f (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06) ``` so, in this change, let's hold it using a smart pointer, so we can destroy it when it leaves the lexical scope. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com> Closes #2224

avikivity closed this as completed in fd08ece Oct 28, 2014

nyh mentioned this issue Oct 29, 2014

httpd crash #5

Closed

nyh mentioned this issue Nov 2, 2014

Seastar httpd incorrectly handles HTTP 0.9 requests #6

Open

avikivity pushed a commit that referenced this issue Dec 18, 2014

tcp: Support close initiated on server side for posix stack

e3b2463

Fix hang with ab test on posix stack: ab -n 1000 http://127.0.0.1/ Fixes #3

avikivity pushed a commit that referenced this issue Dec 21, 2014

tcp: Support close initiated on server side for posix stack

04dd72e

Fix hang with ab test on posix stack: ab -n 1000 http://127.0.0.1/ Fixes #3

slivne mentioned this issue Jan 19, 2015

support running on hyperthreads #15

Closed

vladzcloudius mentioned this issue Jan 28, 2015

DHCP timeout on AWS guest #20

Open

gleb-cloudius mentioned this issue Oct 18, 2015

memcached crashes on setting first key #64

Closed

shivshankardayal mentioned this issue Nov 30, 2015

seastar segfaults on Ubuntu 15.10 #81

Closed

aaalgo mentioned this issue Jan 2, 2016

httpd with DPDK segmentation fault #99

Open

kthommandra mentioned this issue Aug 24, 2016

Core dump in scylla-server with DHCP in DPDK mode #184

Closed

amnonh mentioned this issue Nov 21, 2016

Reactor unregister its metrics after the metrics shutdown #218

Closed

asimarora mentioned this issue Nov 25, 2017

Core Dump while running sample program #345

Closed

nlfox mentioned this issue Jan 13, 2018

Memcached bad performance (both using DPDK and posix network stack) #383

Open

frank8989 mentioned this issue Feb 26, 2018

Task block detector deadlocks with exception throwing #298

Open

tedqian mentioned this issue Mar 12, 2018

httpd print “std::system_error (error system:103, Software caused connection abort),” when seawreck run with multiple cores and connections #418

Closed

joeylichang mentioned this issue Mar 30, 2018

what(): network stack native not registeredterminate called recursively #423

Closed

closertotheheart mentioned this issue Jan 30, 2019

1/47 Test #1: Seastar.dist.consumer .........................***Failed 86.95 sec #596

Closed

loginhel mentioned this issue May 16, 2020

how to run tcp_demo #731

Open

WJTian mentioned this issue May 25, 2020

Native stack: cannot find sent record in _tcbs. #750

Open

hiseyxhp mentioned this issue Jun 23, 2020

when rpc client closed by server, the cpu 100% #762

Open

Howe2015 mentioned this issue Aug 31, 2020

http server dump #781

Open

zhanglistar mentioned this issue Jan 21, 2022

echo_demo core dump #1003

Closed

vladzcloudius added a commit to vladzcloudius/seastar that referenced this issue Aug 19, 2022

perftune.py: work in progress scylladb#3

94e1337

jordan-at-cy mentioned this issue Mar 4, 2023

Segmentation fault when running demos/tpc_demo.cc #1526

Open

nlescoua mentioned this issue Mar 28, 2023

Abort called when writing a temporary_buffer to a file_output_stream #1581

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

apachebench hangs on httpd #3

apachebench hangs on httpd #3

nyh commented Oct 14, 2014

asias commented Oct 22, 2014

asias commented Oct 27, 2014

nyh commented Oct 27, 2014

asias commented Oct 27, 2014

nyh commented Oct 27, 2014

nyh commented Oct 29, 2014

asias commented Oct 29, 2014

asias commented Nov 2, 2014

nyh commented Nov 2, 2014

apachebench hangs on httpd #3

apachebench hangs on httpd #3

Comments

nyh commented Oct 14, 2014

asias commented Oct 22, 2014

asias commented Oct 27, 2014

nyh commented Oct 27, 2014

asias commented Oct 27, 2014

nyh commented Oct 27, 2014

nyh commented Oct 29, 2014

asias commented Oct 29, 2014

asias commented Nov 2, 2014

nyh commented Nov 2, 2014