Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

heap-buffer-overflow in do_stop_gossiping #5701

Closed
bhalevy opened this issue Feb 3, 2020 · 1 comment
Closed

heap-buffer-overflow in do_stop_gossiping #5701

bhalevy opened this issue Feb 3, 2020 · 1 comment

Comments

@bhalevy
Copy link
Member

@bhalevy bhalevy commented Feb 3, 2020

Seen in https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-debug/376/testReport/bootstrap_test/TestBootstrap/add_node_test/
Scylla version 4839ca8

node1.log:

INFO  2020-02-03 01:13:07,227 [shard 0] gossip - InetAddress 127.0.53.2 is now DOWN, status = shutdown
=================================================================
==2263==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000921748 at pc 0x000011107219 bp 0x7f61bfa86990 sp 0x7f61bfa86980
READ of size 24 at 0x603000921748 thread T0
    #0 0x11107218 in operator() gms/gossiper.cc:1938
    #1 0x11198124 in apply seastar/include/seastar/core/apply.hh:36
    #2 0x111981ab in apply<gms::gossiper::do_stop_gossiping()::<lambda()> > seastar/include/seastar/core/apply.hh:44
    #3 0x1119823b in apply_tuple<gms::gossiper::do_stop_gossiping()::<lambda()> > seastar/include/seastar/core/future.hh:1599
    #4 0x111768ee in apply<gms::gossiper::do_stop_gossiping()::<lambda()> > seastar/include/seastar/core/future.hh:1633
    #5 0x111528ee in operator() seastar/include/seastar/core/thread.hh:252
    #6 0x111cc81b in call seastar/include/seastar/util/noncopyable_function.hh:101
    #7 0xc1beacd in seastar::noncopyable_function<void ()>::operator()() const seastar/include/seastar/util/noncopyable_function.hh:184
    #8 0x159dee3a in seastar::thread_context::main() seastar/src/core/thread.cc:286

0x603000921748 is located 0 bytes to the right of 24-byte region [0x603000921730,0x603000921748)
allocated by thread T0 here:
    #0 0x7f62567a6c58 in __interceptor_malloc (/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x10dc58)
    #1 0x1127fd95 in utils::chunked_vector<gms::inet_address, 131072ul>::new_chunk(unsigned long) utils/chunked_vector.hh:336
    #2 0x1124a82e in utils::chunked_vector<gms::inet_address, 131072ul>::shrink_to_fit() utils/chunked_vector.hh:429
    #3 0x11227665 in utils::chunked_vector<gms::inet_address, 131072ul>::resize(unsigned long) utils/chunked_vector.hh:411
    #4 0x110f667f in gms::gossiper::mark_dead(gms::inet_address, gms::endpoint_state&) gms/gossiper.cc:1478
    #5 0x1110af34 in gms::gossiper::mark_as_shutdown(gms::inet_address const&) gms/gossiper.cc:2071
    #6 0x110d2a24 in operator() gms/gossiper.cc:430
    #7 0x1117bb03 in apply seastar/include/seastar/core/apply.hh:36
    #8 0x1117bb8a in apply<gms::gossiper::handle_shutdown_msg(gms::inet_address)::<lambda()> > seastar/include/seastar/core/apply.hh:44
    #9 0x1117bc1a in apply_tuple<gms::gossiper::handle_shutdown_msg(gms::inet_address)::<lambda()> > seastar/include/seastar/core/future.hh:1599
    #10 0x1115b772 in apply<gms::gossiper::handle_shutdown_msg(gms::inet_address)::<lambda()> > seastar/include/seastar/core/future.hh:1633
    #11 0x1112c48e in operator() seastar/include/seastar/core/thread.hh:252
    #12 0x111bf676 in call seastar/include/seastar/util/noncopyable_function.hh:101
    #13 0xc1beacd in seastar::noncopyable_function<void ()>::operator()() const seastar/include/seastar/util/noncopyable_function.hh:184
    #14 0x159dee3a in seastar::thread_context::main() seastar/src/core/thread.cc:286
    #15 0x159deb95 in seastar::thread_context::s_main(int, int) seastar/src/core/thread.cc:264
    #16 0x7f6254357d1f  (/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libc.so.6+0x51d1f)

SUMMARY: AddressSanitizer: heap-buffer-overflow gms/gossiper.cc:1938 in operator()
Shadow bytes around the buggy address:
  0x0c068011c290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c068011c2a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c068011c2b0: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
  0x0c068011c2c0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x0c068011c2d0: fd fa fa fa fd fd fd fd fa fa fd fd fd fd fa fa
=>0x0c068011c2e0: 00 00 00 00 fa fa 00 00 00[fa]fa fa fd fd fd fd
  0x0c068011c2f0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x0c068011c300: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068011c310: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068011c320: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068011c330: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==2263==ABORTING
Aborting on shard 0.
Backtrace:
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x000000000006ce7f
  0x0000000015541800
  0x000000001551ec28
  0x00000000153e7cdd
  0x00000000153e7e23
  0x000000001543ed42
  0x0000000015479553
  0x00000000154795d4
  0x00007f62556e0b1f
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libc.so.6+0x000000000003c624
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libc.so.6+0x00000000000258d8
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x000000000012b731
  0x00007f62567cf2db
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x000000000011797b
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x00000000001173f2
  /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/4839ca849136841b4545931876b1345ee09affa1/scylla//opt/scylladb/libreloc/libasan.so.5+0x0000000000118a17
  0x0000000011107218
  0x0000000011198124
  0x00000000111981ab
  0x000000001119823b
  0x00000000111768ee
  0x00000000111528ee
  0x00000000111cc81b
  0x000000000c1beacd
  0x00000000159dee3a

gossiper::_live_endpoints was resized by mark_dead while traversed by do_stop_gossiping.

I think that the simplest solution would be to simply traverse a copy of _live_endpoints in do_stop_gossiping.
tgrabiec pushed a commit that referenced this issue Feb 3, 2020
@bhalevy @tgrabiec
gossiper: do_stop_gossiping: copy live endpoints vector
d01ffcf 
It can be resized asynchronously by mark_dead.

Fixes #5701

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200203091344.229518-1-bhalevy@scylladb.com>
avikivity pushed a commit that referenced this issue Feb 26, 2020
@bhalevy @avikivity
gossiper: do_stop_gossiping: copy live endpoints vector
ff2e108 
It can be resized asynchronously by mark_dead.

Fixes #5701

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200203091344.229518-1-bhalevy@scylladb.com>
(cherry picked from commit f45faba)
avikivity pushed a commit that referenced this issue Feb 26, 2020
@bhalevy @avikivity
gossiper: do_stop_gossiping: copy live endpoints vector
63e9311 
It can be resized asynchronously by mark_dead.

Fixes #5701

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200203091344.229518-1-bhalevy@scylladb.com>
(cherry picked from commit f45faba)
avikivity pushed a commit that referenced this issue Feb 26, 2020
@bhalevy @avikivity
gossiper: do_stop_gossiping: copy live endpoints vector
8a94f6b 
It can be resized asynchronously by mark_dead.

Fixes #5701

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200203091344.229518-1-bhalevy@scylladb.com>
(cherry picked from commit f45faba)
@avikivity
Copy link
Member

@avikivity avikivity commented Feb 26, 2020

Backported to 3.3, 3.2, 3.1.

avikivity pushed a commit to avikivity/scylla that referenced this issue Apr 28, 2020
@bhalevy @avikivity
gossiper: do_stop_gossiping: copy live endpoints vector
1d8ab5f 
It can be resized asynchronously by mark_dead.

Fixes scylladb#5701

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20200203091344.229518-1-bhalevy@scylladb.com>
(cherry picked from commit f45faba)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@avikivity @bhalevy @scylladb-promoter