Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

httpd RST storm #1

Closed
asias opened this issue Oct 9, 2014 · 0 comments
Closed

httpd RST storm #1

asias opened this issue Oct 9, 2014 · 0 comments

Comments

@asias
Copy link
Contributor

asias commented Oct 9, 2014

$ wrk http://192.168.122.2:10000
Running 10s test @ http://192.168.122.2:10000
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   840.01ms  142.60us 840.28ms   70.00%
    Req/Sec     5.40      1.26     6.00     80.00%
  110 requests in 10.00s, 364.43KB read
Requests/sec:     11.00
Transfer/sec:     36.44KB

With commit 2f42ea15e97f8f6ee002876a6233031f0fdad690 + following patch to send large HTTP response:

--- a/apps/httpd/httpd.cc
+++ b/apps/httpd/httpd.cc
@@ -140,7 +140,8 @@ public:
             auto resp = std::make_unique<response>();
             resp->_response_line = "HTTP/1.1 200 OK\r\n";
             resp->_headers["Content-Type"] = "text/html";
-            resp->_body = "<html><head><title>this is the future</title></head><body><p>Future!!</p></body></html>";
+            resp->_body = std::string(1024*3, 'X');
             respond(std::move(resp));
         }
         future<size_t> write_body() {

I saw the following:

610  12.009139 192.168.122.1 -> 192.168.122.2 TCP 54 57381 > 10000 [RST] Seq=1 Win=0 Len=0
611  12.009143 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57382 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
612  12.009149 192.168.122.1 -> 192.168.122.2 TCP 54 57382 > 10000 [RST] Seq=1 Win=0 Len=0
613  12.009152 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57384 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
614  12.009159 192.168.122.1 -> 192.168.122.2 TCP 54 57384 > 10000 [RST] Seq=1 Win=0 Len=0
615  12.009162 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57386 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
616  12.009169 192.168.122.1 -> 192.168.122.2 TCP 54 57386 > 10000 [RST] Seq=1 Win=0 Len=0
617  12.009173 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57385 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
618  12.009180 192.168.122.1 -> 192.168.122.2 TCP 54 57385 > 10000 [RST] Seq=1 Win=0 Len=0
619  12.009184 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57383 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
620  12.009191 192.168.122.1 -> 192.168.122.2 TCP 54 57383 > 10000 [RST] Seq=1 Win=0 Len=0
621  12.409304 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57377 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
622  12.409343 192.168.122.1 -> 192.168.122.2 TCP 54 57377 > 10000 [RST] Seq=1 Win=0 Len=0
623  12.409358 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57378 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
624  12.409371 192.168.122.1 -> 192.168.122.2 TCP 54 57378 > 10000 [RST] Seq=1 Win=0 Len=0
625  12.409380 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57379 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
626  12.409389 192.168.122.1 -> 192.168.122.2 TCP 54 57379 > 10000 [RST] Seq=1 Win=0 Len=0
627  12.409393 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57380 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
628  12.409401 192.168.122.1 -> 192.168.122.2 TCP 54 57380 > 10000 [RST] Seq=1 Win=0 Len=0
629  12.409407 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57381 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
630  12.409415 192.168.122.1 -> 192.168.122.2 TCP 54 57381 > 10000 [RST] Seq=1 Win=0 Len=0
631  12.409419 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57382 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
632  12.409426 192.168.122.1 -> 192.168.122.2 TCP 54 57382 > 10000 [RST] Seq=1 Win=0 Len=0
633  12.409428 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57384 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
634  12.409435 192.168.122.1 -> 192.168.122.2 TCP 54 57384 > 10000 [RST] Seq=1 Win=0 Len=0
635  12.409438 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57386 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
636  12.409445 192.168.122.1 -> 192.168.122.2 TCP 54 57386 > 10000 [RST] Seq=1 Win=0 Len=0
637  12.409449 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57385 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
638  12.409455 192.168.122.1 -> 192.168.122.2 TCP 54 57385 > 10000 [RST] Seq=1 Win=0 Len=0
639  12.409459 192.168.122.2 -> 192.168.122.1 TCP 54 [TCP ZeroWindow] 10000 > 57383 [ACK] Seq=1 Ack=1 Win=0 [TCP CHECKSUM INCORRECT] Len=0
640  12.409466 192.168.122.1 -> 192.168.122.2 TCP 54 57383 > 10000 [RST] Seq=1 Win=0 Len=0
@asias asias closed this as completed Oct 27, 2014
avikivity pushed a commit that referenced this issue Nov 11, 2014
Fixes assert failure during ^C:
   #0  0x0000003e134348c7 in raise () from /lib64/libc.so.6
   #1  0x0000003e1343652a in abort () from /lib64/libc.so.6
   #2  0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6
   #3  0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6
   #4  0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_
        at /usr/include/boost/intrusive/list.hpp:1263
   #5  0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71
   #6  reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287
   #7  0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974
   #8  ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86
   #9  ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96
avikivity pushed a commit that referenced this issue Dec 18, 2014
Fixes assert failure during ^C:
   #0  0x0000003e134348c7 in raise () from /lib64/libc.so.6
   #1  0x0000003e1343652a in abort () from /lib64/libc.so.6
   #2  0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6
   #3  0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6
   #4  0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_
        at /usr/include/boost/intrusive/list.hpp:1263
   #5  0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71
   #6  reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287
   #7  0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974
   #8  ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86
   #9  ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96
avikivity pushed a commit that referenced this issue Dec 21, 2014
Fixes assert failure during ^C:
   #0  0x0000003e134348c7 in raise () from /lib64/libc.so.6
   #1  0x0000003e1343652a in abort () from /lib64/libc.so.6
   #2  0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6
   #3  0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6
   #4  0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_
        at /usr/include/boost/intrusive/list.hpp:1263
   #5  0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71
   #6  reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287
   #7  0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974
   #8  ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86
   #9  ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96
avikivity pushed a commit that referenced this issue Oct 1, 2016
Records allocation sites and sizes so that it's later possible to see
what kind of objects are taking up space on the heap.

Adds 8 bytes of memory overhead per live object when compiled-in.

Not compiled-in by default. To do so, configure with SEASTAR_HEAPPROF:

  ./configure.py --cflags=-DSEASTAR_HEAPPROF

To enable recording allocation sites, start seastar application with
the --heapprof flag:

  build/release/memcached --heapprof

It is possible to enable recording in run-time using, for instance, a
RESTful API.

scylla-gdb.py support will follow, with text-mode tree presentation
and dumping to flamegraphs.

Flamegraph example:

  https://cloud.githubusercontent.com/assets/283695/18888026/f69fc4e6-84f6-11e6-9b7b-305667d30f52.png

Text example:

(gdb) scylla heapprof -r
All (275954028, #12783)
 |-- void* memory::cpu_pages::allocate_large_and_trim<memory::cpu_pages::allocate_large_aligned(unsigned int, unsigned int)::{lambda(unsigned int, unsigned int)#1}>(unsigned int, memory::cpu_pages::allocate_large_aligned(unsigned int, unsigned int)::{lambda(unsigned int, unsigned int)#1}) + 169  (268959744, #5)
 |   memory::allocate_large_aligned(unsigned long, unsigned long) + 87
 |    |-- logalloc::segment_zone::segment_zone() + 291  (268435456, #1)
 |    |   logalloc::segment_pool::allocate_segment() + 413
 |    |   logalloc::segment_pool::segment_pool() + 296
 |    |   __tls_init.part.787 + 72
 |    |   logalloc::region_group::release_requests() + 1333
 |    |   logalloc::region_group::add(logalloc::region_group*) + 514
 |    |   database::database(db::config const&) + 4246
 |    |   (...)
 |    |
 |    \-- memory::allocate_aligned(unsigned long, unsigned long) + 13  (524288, #4)
 |         |-- memalign + 9  (262144, #2)
 |         |   db::commitlog::segment_manager::acquire_buffer(unsigned long) + 90
 |         |   db::commitlog::segment::new_buffer(unsigned long) + 113
 |         |   db::commitlog::segment::allocate(utils::UUID const&, shared_ptr<db::commitlog::entry_writer>) + 2347
 |         |    |-- db::commitlog::add_entry(utils::UUID const&, commitlog_entry_writer const&) + 1620  (131072, #1)
 |         |    |   database::do_apply(lw_shared_ptr<schema const>, frozen_mutation const&) + 182
 |         |    |   database::apply(lw_shared_ptr<schema const>, frozen_mutation const&) + 235
 |         |    |   service::storage_proxy::mutate_locally(lw_shared_ptr<schema const> const&, frozen_mutation const&) + 434

Message-Id: <1475240766-20198-1-git-send-email-tgrabiec@scylladb.com>
avikivity pushed a commit that referenced this issue May 21, 2018
When _snd.window == 0, it is not allowed to send a tcp segment with more
than 1 byte of data for zero window probe. However, it is legal to send
an empty segment, even if _snd.window == 0.

When a seastar tcp client connects to a tcp server:

With:

    void do_syn_sent() {
        _state = SYN_SENT;
        _snd.syn_tx_time = clock_type::now();
   +    print("In do_syn_sent: _snd.window = %ld\n", _snd.window);
        // Send <SYN> to remote
        output();
    }

I saw:

   In do_syn_sent: _snd.window = 0

So the callchain is:

do_syn_sent() (with a _snd.window == 0)-> output() -> get_packet() -> output_one -> get the assert

client: net/tcp.hh:1691: void seastar::net::tcp<InetTraits>::tcb::output_one(bool) [with InetTraits = seastar::net::ipv4_traits]: Assertion `(_snd.window > 0) || ((_snd.window == 0) && (len == 1))' failed.

$ addr2line -Cfpi -e ./client
  0x000000000041fa8a
  0x000000000041fb5c
  0x000000000041fc13
  /lib64/libpthread.so.0+0x000000000001241f
  /lib64/libc.so.6+0x00000000000369fa
  /lib64/libc.so.6+0x00000000000387ff
  /lib64/libc.so.6+0x000000000002f0d9
  /lib64/libc.so.6+0x000000000002f151
  0x00000000005d0cc7
  0x00000000005d41f9
  0x00000000005d46e8
  0x00000000005cc2f3
  0x00000000005a7ad0
  0x00000000005a7bfb
  0x000000000052c3b2
  0x000000000052d752
  0x000000000041a814
  0x000000000046b47e
  0x00000000004fb9f2
  0x00000000004fc642
  0x0000000000418e1d
  /lib64/libc.so.6+0x0000000000020889
  0x0000000000418fc9 seastar::backtrace_buffer::append_backtrace() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:378
 (inlined by) print_with_backtrace at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:399
seastar::print_with_backtrace(char const*) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:406
sigabrt_action at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3914
 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3896
 (inlined by) _FUN at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3892
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::net::tcp<seastar::net::ipv4_traits>::tcb::output_one(bool) at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:1691 (discriminator 2)
seastar::net::tcp<seastar::net::ipv4_traits>::tcb::get_packet() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:2046
 (inlined by) seastar::net::tcp<seastar::net::ipv4_traits>::tcp(seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>&)::{lambda()#2}::operator()() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:770
std::_Function_handler<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> (), seastar::net::tcp<seastar::net::ipv4_traits>::tcp(seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>&)::{lambda()#2}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/7/bits/std_function.h:302
std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>::register_packet_provider(std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>)::{lambda()#1}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.hh:424
 (inlined by) std::_Function_handler<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> (), seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>::register_packet_provider(std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/7/bits/std_function.h:302
std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::net::ipv4::get_packet() at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.cc:302
operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.cc:54
 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302
std::function<std::experimental::fundamentals_v1::optional<seastar::net::l3_protocol::l3packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:246
 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302
std::function<std::experimental::fundamentals_v1::optional<seastar::net::packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::net::qp::poll_tx() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:68
 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:89
 (inlined by) poll at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.hh:1147
seastar::reactor::poll_once() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3254
 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3150
 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302
std::function<bool ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::reactor::run() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3175
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:185
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:113
main at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/client.cc:167

Message-Id: <5b6fddc098a1c936dd7c6833e33059d0264bb3b7.1526611968.git.asias@scylladb.com>
avikivity pushed a commit that referenced this issue May 21, 2018
The receive window size should be adjusted according to the max receiver
buffer.

Fix the following abort. Without reducing the receive window, the sender
has no back pressure, if the application does not read the data, the
_rcv.data grows unbounded, when the application read again, all the
packet in _rcv.data will be merged into a single packet with tons of
fragments and cause the assert.

In seastar::net::tcp<seastar::net::ipv4_traits>::tcb::read():

for (auto&& q : _rcv.data) {
   p.append(std::move(q));
}

server: /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:144: static void* seastar::net::packet::impl::operator new(size_t, size_t): Assertion `nr_frags == uint16_t(nr_frags)' failed.
Aborting on shard 0.
Backtrace:
  0x000000000041fafa
  0x000000000041fbcc
  0x000000000041fc83
  /lib64/libpthread.so.0+0x000000000001241f
  /lib64/libc.so.6+0x00000000000369fa
  /lib64/libc.so.6+0x00000000000387ff
  /lib64/libc.so.6+0x000000000002f0d9
  /lib64/libc.so.6+0x000000000002f151
  0x00000000004b89dd
  0x00000000005ab01e
  0x00000000005cbc6a
  0x00000000005dbfc1
  0x00000000005db9b8
  0x0000000000570986
  0x0000000000570be5
  0x000000000069859a
  0x00000000006a16e5
  0x000000000041d136
  0x000000000041d422
  0x000000000046b527
  0x00000000004fba62
  0x0000000000418e77
  /lib64/libc.so.6+0x0000000000020889
  0x0000000000419039

$ addr2line -Cfpi -e ./server

  0x000000000041fafa
  0x000000000041fbcc
  0x000000000041fc83
  /lib64/libpthread.so.0+0x000000000001241f
  /lib64/libc.so.6+0x00000000000369fa
  /lib64/libc.so.6+0x00000000000387ff
  /lib64/libc.so.6+0x000000000002f0d9
  /lib64/libc.so.6+0x000000000002f151
  0x00000000004b89dd
  0x00000000005ab01e
  0x00000000005cbc6a
  0x00000000005dbfc1
  0x00000000005db9b8
  0x0000000000570986
  0x0000000000570be5
  0x000000000069859a
  0x00000000006a16e5
  0x000000000041d136
  0x000000000041d422
  0x000000000046b527
  0x00000000004fba62
  0x0000000000418e77
  /lib64/libc.so.6+0x0000000000020889
  0x0000000000419039  seastar::backtrace_buffer::append_backtrace() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:378
 (inlined by) print_with_backtrace at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:399
seastar::print_with_backtrace(char const*) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:406
sigabrt_action at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3914
 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3896
 (inlined by) _FUN at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3892
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::net::packet::impl::operator new(unsigned long, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:144
 (inlined by) seastar::net::packet::impl::allocate(unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:117
 (inlined by) seastar::net::packet::impl::copy(seastar::net::packet::impl*, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:121
 (inlined by) seastar::net::packet::impl::allocate_if_needed(std::unique_ptr<seastar::net::packet::impl, std::default_delete<seastar::net::packet::impl> >, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:141
seastar::net::packet::append(seastar::net::packet&&) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:490
seastar::net::tcp<seastar::net::ipv4_traits>::tcb::read() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:1760
 (inlined by) seastar::net::tcp<seastar::net::ipv4_traits>::connection::read() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:682
 (inlined by) seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/net/native-stack-impl.hh:145
 (inlined by) seastar::apply_helper<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply({lambda()#2}&&, std::tuple<>) at /home/asias/src/cloudius-systems/scylla/seastar/core/apply.hh:36
 (inlined by) auto seastar::apply<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}>(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&, std::tuple<>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/apply.hh:44
 (inlined by) seastar::future<seastar::temporary_buffer<char> > seastar::futurize<seastar::future<seastar::temporary_buffer<char> > >::apply<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}>(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&, std::tuple<>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1377
seastar::future<seastar::temporary_buffer<char> > seastar::future<>::then<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}, seastar::future<seastar::temporary_buffer<char> > >(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:943
seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get() at /home/asias/src/cloudius-systems/scylla/seastar/net/native-stack-impl.hh:149
seastar::data_source::get() at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream.hh:67
 (inlined by) seastar::input_stream<char>::read_exactly_part(unsigned long, seastar::temporary_buffer<char>, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream-impl.hh:164
seastar::input_stream<char>::read_exactly(unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream-impl.hh:191
tcp_echo_server::connection::process() at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:86
 (inlined by) tcp_echo_server::start()::{lambda()#1}::operator()() const::{lambda(seastar::connected_socket, {lambda()#1}::socket_address)#1}::operator()(seastar, seastar::connected_socket)::{lambda()#2}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:113
seastar::internal::do_until_state<tcp_echo_server::start()::{lambda()#1}::operator()() const::{lambda(seastar::connected_socket, seastar::socket_address)#1}::operator()({lambda()#1}, seastar::connected_socket)::{lambda()#1}, tcp_echo_server::start()::{lambda()#1}::operator()() const::seastar::socket_address::operator()({lambda()#1}, seastar::connected_socket)::{lambda()#2}>::run_and_dispose() at /home/asias/src/cloudius-systems/scylla/seastar/core/future-util.hh:463
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:2597
seastar::reactor::run_some_tasks() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3009
seastar::reactor::run_some_tasks() at /usr/include/c++/7/chrono:377
 (inlined by) seastar::reactor::run() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3156
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:185
main at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:179 (discriminator 5)

Message-Id: <06a6c0ccb82fd4fdde6f56a112a8eb6f659ee8ab.1526611968.git.asias@scylladb.com>
denesb added a commit to denesb/seastar that referenced this issue Aug 1, 2018
Instead, assign an id to each backtrace and print that before printing
the backtrace. The id is simply a counter that starts from 0 and is
incremented at each seen backtrace. Also separate backtraces with an
empty line. The new output looks like this:

Backtrace #0:
...
...
...

Backtrace scylladb#1:
...
...
...
...
denesb added a commit to denesb/seastar that referenced this issue Aug 2, 2018
Instead, assign an id to each backtrace and print that before printing
the backtrace. The id is simply a counter that starts from 0 and is
incremented at each seen backtrace. Also separate backtraces with an
empty line. The new output looks like this:

Backtrace #0:
...
...
...

Backtrace scylladb#1:
...
...
...
...
avikivity pushed a commit that referenced this issue Apr 6, 2021
…o_with

Fixes failures in debug mode:
```
$ build/debug/tests/unit/closeable_test -l all -t deferred_close_test
WARNING: debug mode. Not for benchmarking or production
random-seed=3064133628
Running 1 test case...
Entering test module "../../tests/unit/closeable_test.cc"
../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test"
../../src/testing/seastar_test.cc(43): info: check true has passed
==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
terminate called after throwing an instance of 'seastar::broken_promise'
  what():  broken promise
==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592)
False positive error reports may follow
For details see google/sanitizers#189
=================================================================
==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
    #0 0x7fbf45f39d0b  (/lib64/libasan.so.6+0xb3d0b)
    #1 0x7fbf45f57d4e  (/lib64/libasan.so.6+0xd1d4e)
    #2 0x7fbf45f3e724  (/lib64/libasan.so.6+0xb8724)
    #3 0x7fbf45eb3e5b  (/lib64/libasan.so.6+0x2de5b)
    #4 0x7fbf45eb51e8  (/lib64/libasan.so.6+0x2f1e8)
    #5 0x7fbf45eb7694  (/lib64/libasan.so.6+0x31694)
    #6 0x7fbf45f39398  (/lib64/libasan.so.6+0xb3398)
    #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b)
    #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116
    #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121
    #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810)
    #11 0x7fbf44bcd897  (/lib64/libgcc_s.so.1+0xf897)
    #12 0x7fbf44bcea5f  (/lib64/libgcc_s.so.1+0x10a5f)
    #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8)
    #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148
    #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb)
    #16 0x7fbf45e10a21  (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21)
    #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0)
    #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094)
    #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921)
    #20 0x7fbf45e5eae1  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1)
    #21 0x7fbf45e5ed31  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31)
    #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547)
    #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618)
    #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77
    #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65
    #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20210406100911.12278-1-bhalevy@scylladb.com>
xemul added a commit to xemul/seastar that referenced this issue May 25, 2021
This is step scylladb#1 in removing all classes manipulations API from
io_queue to io_priority_class.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
xemul added a commit to xemul/seastar that referenced this issue Jun 11, 2021
The io_queue_topology describes "numerical" topology -- the
numbers of queues and groups and their mappings to each other.
Later this info is materialized into device_io_topology by
creating vectors of queues and groups.

There's no real need in this split. The io_queue_topology can
just create the needed vectors with no objects in them, so
that later queues and groups are put into _it_ instead of on
the device_io_topology.

This is step scylladb#1 -- move vector of queues into io_queue_topology.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
xemul added a commit to xemul/seastar that referenced this issue Jun 11, 2021
This is a tiny leak fix -- if queues initialization fails in the
middle some allocated, but not yet assigned queues will not be
deleted.

(side note: this patch is the beneficiary of the explicit outline
constructors for io_queue_topology from patch scylladb#1 of this set)

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
xemul added a commit to xemul/seastar that referenced this issue Jun 24, 2021
The io_queue_topology describes "numerical" topology -- the
numbers of queues and groups and their mappings to each other.
Later this info is materialized into device_io_topology by
creating vectors of queues and groups.

There's no real need in this split. The io_queue_topology can
just create the needed vectors with no objects in them, so
that later queues and groups are put into _it_ instead of on
the device_io_topology.

This is step scylladb#1 -- move vector of queues into io_queue_topology.

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
xemul added a commit to xemul/seastar that referenced this issue Jun 24, 2021
This is a tiny leak fix -- if queues initialization fails in the
middle some allocated, but not yet assigned queues will not be
deleted.

(side note: this patch is the beneficiary of the explicit outline
constructors for io_queue_topology from patch scylladb#1 of this set)

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Sep 20, 2021
… enqueue_op throws

If enqueue_op fails to allocate, op_func ends up setting `fut`
error to broken promise and we get the following warning:
```
WARN  2021-09-20 11:02:33,437 [shard 0] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace: 0x57a609 0x57a932 0x57ace9 0x488b57 0x488c7c 0x44a970 0x41418e 0x4645e9 0x47f99a 0x484dfb 0x485acc 0x44f6ef 0x44fcfb 0x45342d 0x4464f8 0x42a33c
```

Decoded:
```
seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) [clone .cold] at /home/bhalevy/dev/seastar/build/release/../../src/core/file-impl.hh:258
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:811
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/file.hh:351
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:431
seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}::operator()() at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:387
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&, seastar::internal::monostate) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::future<void>&, seastar::internal::future_result&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1520
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:406
seastar::data_sink::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream.hh:142
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:454
seastar::output_stream<char>::flush() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:420
seastar::output_stream<char>::close() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:512
```

To prevent this, catch the error around enqueue_op and handle
it by fut.ignore_ready_future().

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Sep 20, 2021
… enqueue_op throws

If enqueue_op fails to allocate, op_func ends up setting `fut`
error to broken promise and we get the following warning:
```
WARN  2021-09-20 11:02:33,437 [shard 0] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace: 0x57a609 0x57a932 0x57ace9 0x488b57 0x488c7c 0x44a970 0x41418e 0x4645e9 0x47f99a 0x484dfb 0x485acc 0x44f6ef 0x44fcfb 0x45342d 0x4464f8 0x42a33c
```

Decoded:
```
seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) [clone .cold] at /home/bhalevy/dev/seastar/build/release/../../src/core/file-impl.hh:258
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:811
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/file.hh:351
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:431
seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}::operator()() at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:387
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&, seastar::internal::monostate) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::future<void>&, seastar::internal::future_result&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1520
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:406
seastar::data_sink::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream.hh:142
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:454
seastar::output_stream<char>::flush() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:420
seastar::output_stream<char>::close() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:512
```

To prevent this, catch the error around enqueue_op and handle
it by fut.ignore_ready_future().

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Nov 15, 2021
… enqueue_op throws

If enqueue_op fails to allocate, op_func ends up setting `fut`
error to broken promise and we get the following warning:
```
WARN  2021-09-20 11:02:33,437 [shard 0] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace: 0x57a609 0x57a932 0x57ace9 0x488b57 0x488c7c 0x44a970 0x41418e 0x4645e9 0x47f99a 0x484dfb 0x485acc 0x44f6ef 0x44fcfb 0x45342d 0x4464f8 0x42a33c
```

Decoded:
```
seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) [clone .cold] at /home/bhalevy/dev/seastar/build/release/../../src/core/file-impl.hh:258
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:811
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/file.hh:351
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:431
seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}::operator()() at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:387
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&, seastar::internal::monostate) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::future<void>&, seastar::internal::future_result&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1520
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:406
seastar::data_sink::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream.hh:142
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:454
seastar::output_stream<char>::flush() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:420
seastar::output_stream<char>::close() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:512
```

To prevent this, catch the error around enqueue_op and handle
it by fut.ignore_ready_future().

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Nov 15, 2021
… enqueue_op throws

If enqueue_op fails to allocate, op_func ends up setting `fut`
error to broken promise and we get the following warning:
```
WARN  2021-09-20 11:02:33,437 [shard 0] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace: 0x57a609 0x57a932 0x57ace9 0x488b57 0x488c7c 0x44a970 0x41418e 0x4645e9 0x47f99a 0x484dfb 0x485acc 0x44f6ef 0x44fcfb 0x45342d 0x4464f8 0x42a33c
```

Decoded:
```
seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) [clone .cold] at /home/bhalevy/dev/seastar/build/release/../../src/core/file-impl.hh:258
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:811
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/file.hh:351
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:431
seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}::operator()() at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:387
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&, seastar::internal::monostate) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::future<void>&, seastar::internal::future_result&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1520
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:406
seastar::data_sink::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream.hh:142
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:454
seastar::output_stream<char>::flush() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:420
seastar::output_stream<char>::close() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:512
```

To prevent this, catch the error around enqueue_op and handle
it by fut.ignore_ready_future().

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Nov 24, 2021
… enqueue_op throws

If enqueue_op fails to allocate, op_func ends up setting `fut`
error to broken promise and we get the following warning:
```
WARN  2021-09-20 11:02:33,437 [shard 0] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace: 0x57a609 0x57a932 0x57ace9 0x488b57 0x488c7c 0x44a970 0x41418e 0x4645e9 0x47f99a 0x484dfb 0x485acc 0x44f6ef 0x44fcfb 0x45342d 0x4464f8 0x42a33c
```

Decoded:
```
seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) [clone .cold] at /home/bhalevy/dev/seastar/build/release/../../src/core/file-impl.hh:258
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:811
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/file.hh:351
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:431
seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}::operator()() at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:387
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&, seastar::internal::monostate) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}>(seastar::future<void>&, seastar::internal::future_result&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}, seastar::future<void> >(seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>)::{lambda()scylladb#1}&&) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/future.hh:1520
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../src/core/fstream.cc:406
seastar::data_sink::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream.hh:142
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:454
seastar::output_stream<char>::flush() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:420
seastar::output_stream<char>::close() at /home/bhalevy/dev/seastar/build/release/../../include/seastar/core/iostream-impl.hh:512
```

To prevent this, catch the error around enqueue_op and handle
it by fut.ignore_ready_future().

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Sep 7, 2022
Some may contain space characters around the '+' operator in, e.g.:
```
    scylladb#1  0x00007fd2dab4f950 abort (libc.so.6 + 0x26950)
```

Fixes scylladb#1206

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/seastar that referenced this issue Sep 7, 2022
Some may contain space characters around the '+' operator in, e.g.:
```
    scylladb#1  0x00007fd2dab4f950 abort (libc.so.6 + 0x26950)
```

Fixes scylladb#1206

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
xemul added a commit that referenced this issue Sep 9, 2022
…es' from Benny Halevy

Some may contain space characters around the '+' operator in, e.g.:
```
    #1  0x00007fd2dab4f950 abort (libc.so.6 + 0x26950)
```

Fixes #1206

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #1207

* https://github.com/scylladb/seastar:
  seastar-addr2line: strip input lines
  seastar-addr2line: support more flexible syslog-style backtraces
dotnwat referenced this issue in dotnwat/seastar Oct 24, 2022
When `posix_server_socket_impl::accept()` runs it may start a cross-core
background fiber that inserts a pending connection into the thread local
container posix_ap_server_socket_impl::conn_q.

However, the continuation that enqueues the pending connection may not
aactually run until after the target core calls abort_accept() (e.g.
parallel shutdown via a seastar::sharded<server>::stop).

This can leave an entry in the conn_q container that is destroyed when
the reactor thread exits. Unfortunately the conn_q container holds
conntrack::handle type that schedules additional work in its destructor.

```
   class handle {
       foreign_ptr<lw_shared_ptr<load_balancer>> _lb;
       ~handle() {
           (void)smp::submit_to(_host_cpu, [cpu = _target_cpu, lb = std::move(_lb)] {
               lb->closed_cpu(cpu);
           });
       }
       ...
```

When this race occurs and the destructor runs the reactor is no longer
available, leading to the following memory leak in which the continuation that
is scheduled onto the reactor is leaked:

Direct leak of 88 byte(s) in 1 object(s) allocated from:
    #0 0x557c91ca5b7d in operator new(unsigned long) /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:95:3

    #1 0x557ca3e3cc08 in void seastar::future<void>::schedule<seastar::internal::promise_ba...
    ...
    // the unordered map here is conn_q
    scylladb#19 0x557ca47034d8 in std::__1::unordered_multimap<std::__1::tuple<int, seastar::socket...
    scylladb#20 0x7f98dcaf238e in __call_tls_dtors (/lib64/libc.so.6+0x4038e) (BuildId: 6e3c087aca9...

fixes: scylladb#738

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
dotnwat referenced this issue in dotnwat/seastar Oct 24, 2022
When `posix_server_socket_impl::accept()` runs it may start a cross-core
background fiber that inserts a pending connection into the thread local
container posix_ap_server_socket_impl::conn_q.

However, the continuation that enqueues the pending connection may not
aactually run until after the target core calls abort_accept() (e.g.
parallel shutdown via a seastar::sharded<server>::stop).

This can leave an entry in the conn_q container that is destroyed when
the reactor thread exits. Unfortunately the conn_q container holds
conntrack::handle type that schedules additional work in its destructor.

```
   class handle {
       foreign_ptr<lw_shared_ptr<load_balancer>> _lb;
       ~handle() {
           (void)smp::submit_to(_host_cpu, [cpu = _target_cpu, lb = std::move(_lb)] {
               lb->closed_cpu(cpu);
           });
       }
       ...
```

When this race occurs and the destructor runs the reactor is no longer
available, leading to the following memory leak in which the continuation that
is scheduled onto the reactor is leaked:

Direct leak of 88 byte(s) in 1 object(s) allocated from:
    #0 0x557c91ca5b7d in operator new(unsigned long) /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:95:3

    #1 0x557ca3e3cc08 in void seastar::future<void>::schedule<seastar::internal::promise_ba...
    ...
    // the unordered map here is conn_q
    scylladb#19 0x557ca47034d8 in std::__1::unordered_multimap<std::__1::tuple<int, seastar::socket...
    scylladb#20 0x7f98dcaf238e in __call_tls_dtors (/lib64/libc.so.6+0x4038e) (BuildId: 6e3c087aca9...

fixes: scylladb#738

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 8, 2022
When we enable the sanitizer, we get following error while running
iotune:

==86505==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4096 byte(s) in 1 object(s) allocated from:
    #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7)
    scylladb#1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16
    scylladb#2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21
    scylladb#3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28
    scylladb#4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13
    scylladb#5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20
    scylladb#6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9
    scylladb#7 0x7f30ff8547bf  (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c)

SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s).

This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error.

And this is needed to test Scylla machine image with debug mode binary,
since it tries to run iotune with the sanitizer and fails.
syuu1228 added a commit to syuu1228/seastar-1 that referenced this issue Nov 14, 2022
When we enable the sanitizer, we get following error while running
iotune:

==86505==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4096 byte(s) in 1 object(s) allocated from:
    #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7)
    scylladb#1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16
    scylladb#2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21
    scylladb#3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28
    scylladb#4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13
    scylladb#5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20
    scylladb#6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9
    scylladb#7 0x7f30ff8547bf  (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c)

SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s).

This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error.

And this is needed to test Scylla machine image with debug mode binary,
since it tries to run iotune with the sanitizer and fails.
xemul pushed a commit that referenced this issue Nov 14, 2022
When we enable the sanitizer, we get following error while running
iotune:

==86505==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4096 byte(s) in 1 object(s) allocated from:
    #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7)
    #1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16
    #2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21
    #3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28
    #4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13
    #5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20
    #6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9
    #7 0x7f30ff8547bf  (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c)

SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s).

This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error.

And this is needed to test Scylla machine image with debug mode binary,
since it tries to run iotune with the sanitizer and fails.

Closes #1284
dotnwat referenced this issue in dotnwat/seastar Dec 21, 2022
When `posix_server_socket_impl::accept()` runs it may start a cross-core
background fiber that inserts a pending connection into the thread local
container posix_ap_server_socket_impl::conn_q.

However, the continuation that enqueues the pending connection may not
aactually run until after the target core calls abort_accept() (e.g.
parallel shutdown via a seastar::sharded<server>::stop).

This can leave an entry in the conn_q container that is destroyed when
the reactor thread exits. Unfortunately the conn_q container holds
conntrack::handle type that schedules additional work in its destructor.

```
   class handle {
       foreign_ptr<lw_shared_ptr<load_balancer>> _lb;
       ~handle() {
           (void)smp::submit_to(_host_cpu, [cpu = _target_cpu, lb = std::move(_lb)] {
               lb->closed_cpu(cpu);
           });
       }
       ...
```

When this race occurs and the destructor runs the reactor is no longer
available, leading to the following memory leak in which the continuation that
is scheduled onto the reactor is leaked:

Direct leak of 88 byte(s) in 1 object(s) allocated from:
    #0 0x557c91ca5b7d in operator new(unsigned long) /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:95:3

    #1 0x557ca3e3cc08 in void seastar::future<void>::schedule<seastar::internal::promise_ba...
    ...
    // the unordered map here is conn_q
    scylladb#19 0x557ca47034d8 in std::__1::unordered_multimap<std::__1::tuple<int, seastar::socket...
    scylladb#20 0x7f98dcaf238e in __call_tls_dtors (/lib64/libc.so.6+0x4038e) (BuildId: 6e3c087aca9...

fixes: scylladb#738

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
avikivity pushed a commit that referenced this issue Jan 9, 2023
this change is a follow-up of 338ba97.

before this change, ${CMAKE_CURRENT_BINARY_DIR} is used for
Seastar_BINARY_DIR. if Seastar is a top-level project, the values
of ${CMAKE_CURRENT_BINARY_DIR} and ${CMAKE_BINARY_DIR} are
identical. but if Seastar is embedded in a parent project,
${CMAKE_BINARY_DIR} would be somewhere like "bulid/seastar" where
"build" is the build directory of the parent project.
but we are still referencing the build directory with
${Seastar_BINARY_DIR} and issuing commands like

cmake --build ${Seastar_BINARY_DIR} --target ${target}

if this would fail as the build directory is not ${Seastar_BINARY_DIR}
anymore. if the cmake generator is make, the failure would look like:

> gmake: *** No rule to make target 'test_unit_abort_source_run'.  Stop.

if the cmake generator is ninja, the failure would look like:

> 1/95 Test  #1: Seastar.unit.abort_source .....................***Failed    0.02 sec
> ninja: error: loading 'build.ninja': No such file or directory

after this change, all occurrences of

cmake --build ${Seastar_BINARY_DIR}

are changed to

cmake --build ${CMAKE_BINARY_DIR}

this ensure that these commands can find the build directory
at the top level of the build tree.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
sitano pushed a commit to sitano/seastar that referenced this issue Jun 23, 2023
=================================================================
==93534==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f676abad010 in thread T0
    #0 0x7f677fedfdc2 in __interceptor_free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52
    scylladb#1 0x7f6776e81776 in operator() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1242
    scylladb#2 0x7f6776e884a4 in ~deferred_action /home/sitano/Projects/seastar/include/seastar/util/defer.hh:67
    scylladb#3 0x7f6776e82d31 in try_create_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1253
    scylladb#4 0x7f6776e83e39 in detect_io_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1288
    scylladb#5 0x7f6776e85113 in seastar::reactor_backend_selector::available() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:2039
    scylladb#6 0x7f67773be492 in seastar::reactor_options::reactor_options(seastar::program_options::option_group*) /home/sitano/Projects/seastar/src/core/reactor.cc:3782
    scylladb#7 0x7f6776f04c32 in seastar::app_template::seastar_options::seastar_options() /home/sitano/Projects/seastar/src/core/app-template.cc:70
    scylladb#8 0x7f6776f04040 in seastar_options_from_config /home/sitano/Projects/seastar/src/core/app-template.cc:58
    scylladb#9 0x7f6776f0704f in seastar::app_template::app_template(seastar::app_template::config) /home/sitano/Projects/seastar/src/core/app-template.cc:108
    scylladb#10 0x55a5ff597a6e in main /home/sitano/Projects/seastar/demos/hello-world.cc:30
    scylladb#11 0x7f676f03984f  (/usr/lib/libc.so.6+0x2384f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)
    scylladb#12 0x7f676f039909 in __libc_start_main (/usr/lib/libc.so.6+0x23909) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e)
    scylladb#13 0x55a5ff597614 in _start (/home/sitano/Projects/seastar/build/debug/demos/hello-world_demo+0xf0614) (BuildId: cbf8a43d1565f3bf0a48cdfb43a11ce16d7a9bc1)

Address 0x7f676abad010 is a wild pointer inside of access range of size 0x000000000001.
SUMMARY: AddressSanitizer: bad-free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 in __interceptor_free
==93534==ABORTING
xemul pushed a commit that referenced this issue May 7, 2024
in main(), we creates an instance of `http_server_control` using
new, but we never destroy it. this is identified by ASan

```
==2190125==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3
    #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27
    #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl
ude/c++/14/bits/invoke.h:61:14
    #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_
64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14
    #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, .
..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14
    #5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../
../include/c++/14/tuple:2936:14
    #6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea
star/core/future.hh:2005:28
    #7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0:
:operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13
    #8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()()
 const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable
_function.hh:129:20
    #9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16
    #10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9
    #11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43
    #12 0x7f5d72f8a18f  (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06)
```

so, in this change, let's hold it using a smart pointer, so we
can destroy it when it leaves the lexical scope.

Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>

Closes #2224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant