Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nodetool cleanup segfaults #2537

Closed
glommer opened this issue Jun 29, 2017 · 0 comments
Closed

nodetool cleanup segfaults #2537

glommer opened this issue Jun 29, 2017 · 0 comments
Labels
Milestone

Comments

@glommer
Copy link
Contributor

glommer commented Jun 29, 2017

Installation details
Scylla version (or git commit hash): 1.7.0

Calling nodetool cleanup can segfault a node. Here's the trace that is generated:

0x00000000004e05fa
0x00000000004e06a3
0x00007fb9a4dd736f
0x00000000009797f0
0x00000000008e5356
0x00000000008e78de
0x00000000004e206f
0x0000000000531bc2
0x0000000000533400
0x000000000058ed2d
0x00007fb9a4dcfdc4
0x00007fb9a4afe76c

That translates to:

print_with_backtrace(char const*) at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:2374
 (inlined by) print_with_backtrace at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:3198
void install_oneshot_signal_handler<11, &(sigsegv_action())>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:3202
 (inlined by) operator() at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:3178
 (inlined by) _FUN at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:3174
?? ??:0
lw_shared_ptr<sstables::sstable::shareable_components>::operator*() const at /usr/src/debug/scylla-1.7.0/seastar/core/shared_ptr.hh:212
 (inlined by) foreign_ptr<lw_shared_ptr<sstables::sstable::shareable_components> >::operator->() const at /usr/src/debug/scylla-1.7.0/seastar/core/sharded.hh:520
 (inlined by) sstables::sstable::get_stats_metadata() const at /usr/src/debug/scylla-1.7.0/sstables/sstables.hh:653
sstables::sstable::get_sstable_level() const at /usr/src/debug/scylla-1.7.0/sstables/sstables.hh:679
 (inlined by) operator() at /usr/src/debug/scylla-1.7.0/database.cc:1394
 (inlined by) apply<column_family::cleanup_sstables(sstables::compaction_descriptor)::<lambda(auto:120&)> [with auto:120 = lw_shared_ptr<sstables::sstable>]::<lambda()> > at /usr/src/debug/scylla-1.7.0/seastar/core/future.hh:1277
 (inlined by) futurize_apply<column_family::cleanup_sstables(sstables::compaction_descriptor)::<lambda(auto:120&)> [with auto:120 = lw_shared_ptr<sstables::sstable>]::<lambda()> > at /usr/src/debug/scylla-1.7.0/seastar/core/future.hh:1335
 (inlined by) operator()<semaphore_units<> > at /usr/src/debug/scylla-1.7.0/seastar/core/semaphore.hh:410
 (inlined by) apply at /usr/src/debug/scylla-1.7.0/seastar/core/apply.hh:34
_ZN12continuationIZN6futureIJ15semaphore_unitsI35semaphore_default_exception_factoryNSt6chrono3_V212steady_clockEEEE4thenIZ14with_semaphoreIS2_ZZN13column_family16cleanup_sstablesEN8sstables21compaction_descriptorEENKUlRT_E_clI13lw_shared_ptrINSB_7sstableEEEEDaSE_EUlvE_S5_EN8futurizeINSt9result_ofIFT0_vEE4typeEE4typeER15basic_semaphoreISD_T1_EmOSN_EUlSD_E_S0_IJEEEESN_OSD_EUlS10_E_JS6_EE3runEv at /usr/src/debug/scylla-1.7.0/seastar/core/apply.hh:42
 (inlined by) apply<with_semaphore(basic_semaphore<ExceptionFactory, Clock>&, size_t, Func&&) [with ExceptionFactory = semaphore_default_exception_factory; Func = column_family::cleanup_sstables(sstables::compaction_descriptor)::<lambda(auto:120&)> [with auto:120 = lw_shared_ptr<sstables::sstable>]::<lambda()>; Clock = std::chrono::_V2::steady_clock; futurize_t<typename std::result_of<Func()>::type> = future<>; size_t = long unsigned int]::<lambda(auto:28)>, semaphore_units<semaphore_default_exception_factory, std::chrono::_V2::steady_clock> > at /usr/src/debug/scylla-1.7.0/seastar/core/future.hh:1267
 (inlined by) operator()<future_state<semaphore_units<semaphore_default_exception_factory, std::chrono::_V2::steady_clock> > > at /usr/src/debug/scylla-1.7.0/seastar/core/future.hh:872
 (inlined by) run at /usr/src/debug/scylla-1.7.0/seastar/core/future.hh:390
reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&) at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:2025
reactor::run() at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:2467
smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const [clone .constprop.4079] at /usr/src/debug/scylla-1.7.0/seastar/core/reactor.cc:3396
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/5.3.1/functional:2271
 (inlined by) posix_thread::start_routine(void*) at /usr/src/debug/scylla-1.7.0/seastar/core/posix.cc:50

I have a backtrace in case someone wants to look at it, but will not publish as it contains user data.

@tzach tzach added this to the 1.7 milestone Jul 2, 2017
avikivity pushed a commit that referenced this issue Jul 3, 2017
when do_for_each is in its last iteration and with_semaphore defers
because there's an ongoing cleanup, sstable object will be used after
freed because it was taken by ref and the container it lives in was
destroyed prematurely.

Let's fix it with a do_with, also making code nicer.

Fixes #2537.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20170630035324.19881-1-raphaelsc@scylladb.com>
(cherry picked from commit b9d0645)
avikivity pushed a commit that referenced this issue Jul 3, 2017
when do_for_each is in its last iteration and with_semaphore defers
because there's an ongoing cleanup, sstable object will be used after
freed because it was taken by ref and the container it lives in was
destroyed prematurely.

Let's fix it with a do_with, also making code nicer.

Fixes #2537.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20170630035324.19881-1-raphaelsc@scylladb.com>
(cherry picked from commit b9d0645)
avikivity added a commit to avikivity/scylladb that referenced this issue Mar 20, 2023
…ojnowski

This patch introduces link-time optimization (LTO) to the build.

The performance gains from LTO alone are modest (~7%), but it's vital ingredient of effective profile-guided optimization, which will be introduced later.

In general, use of LTO is quite simple and transparent to build systems. It is sufficient to add the -flto flag to compile and link steps, and use a LTO-aware linker. At compile time, -flto will cause the compiler to emit .o files containing not machine code, but compiler's internal representation (IR). At link time, those pieces of IR will be compiled together, allowing cross-object optimization.

Therefore compiling Scylla with LTO can be as simple as running configure.py with --cflags=-flto. Unfortunately this will also cause all other executables (like tests) to be linked with LTO. In other words, the entire project will be largely recompiled from scratch for every test, and that's impractical.

GCC has the ability to emit "fat" objects containing both LTO-ready IR and machine code, which can be used for both LTO and non-LTO linking, which solves the above problem. Unfortunately LLVM doesn't have such option and we have to implement an alternative ourselves, which complicates the situation.

This patch settles for a dual compilation of all objects: from source to IR files and from IR files to ELF objects. The rule for scylla then uses the IR files for linking and rules for tests use the ELF files.

Due to it's high compile time cost, the optimization can be toggled with a configure.py option. As of this patch, it's disabled by default.

Refs scylladb#2537
Refs scylladb#2538
Refs scylladb#10808

Closes scylladb#2597

* github.com:scylladb/scylla-enterprise:
  configure.py: enable LTO in release builds by default
  configure.py: introduce link-time optimization
  configure.py: add a `default` to `add_tristate`.
  configure.py: unify build rules for cxxbridge .cc files and regular .cc files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants