Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

seastar_memory - oversized allocation in reconcilable_result #4780

Closed
r4fek opened this issue Jul 30, 2019 · 9 comments
Closed

seastar_memory - oversized allocation in reconcilable_result #4780

r4fek opened this issue Jul 30, 2019 · 9 comments
Labels
Milestone

Comments

@r4fek
Copy link

@r4fek r4fek commented Jul 30, 2019

Installation details
Scylla version (or git commit hash): 3.0.8
Cluster size: 23
OS (RHEL/CentOS/Ubuntu/AWS AMI): Debian Stretch

Log:

lip 30 10:45:42 ams-dbs1 scylla[39322]:  [shard 3] seastar_memory - oversized allocation: 6291456 bytes, please report: at   0x59be3e
                                          0x527cd7
                                          0x52d6b7
                                          0x52dcc4
                                          0x14c0f27
                                          0x144f6d1
                                          0x1508c3c
                                          0x37993e
                                          0x379b01
                                          0x44a00f
                                          0x46d43a
                                          0x5266ad
                                          /lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
                                          /lib/x86_64-linux-gnu/libc.so.6+0xe8d0e

Backtrace:

[Backtrace #0]
backtrace<seastar::current_backtrace()::<lambda(seastar::frame)> > at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/util/backtrace.hh:56
 (inlined by) seastar::current_backtrace() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/util/backtrace.cc:84
seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:609
seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:617
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:623
 (inlined by) seastar::memory::allocate_large(unsigned long) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:1194
 (inlined by) seastar::memory::allocate(unsigned long) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:1244
operator new(unsigned long) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/memory.cc:1647
__gnu_cxx::new_allocator<partition>::allocate(unsigned long, void const*) at /opt/scylladb/include/c++/7/ext/new_allocator.h:111
 (inlined by) std::allocator_traits<std::allocator<partition> >::allocate(std::allocator<partition>&, unsigned long) at /opt/scylladb/include/c++/7/bits/alloc_traits.h:436
 (inlined by) std::_Vector_base<partition, std::allocator<partition> >::_M_allocate(unsigned long) at /opt/scylladb/include/c++/7/bits/stl_vector.h:172
 (inlined by) void std::vector<partition, std::allocator<partition> >::_M_realloc_insert<partition>(__gnu_cxx::__normal_iterator<partition*, std::vector<partition, std::allocator<partition> > >, partition&&) at /opt/scylladb/include/c++/7/bits/vector.tcc:406
partition& std::vector<partition, std::allocator<partition> >::emplace_back<partition>(partition&&) at /opt/scylladb/include/c++/7/bits/vector.tcc:105
 (inlined by) reconcilable_result_builder::consume_end_of_partition() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/mutation_partition.cc:2083
query::clustering_position_tracker<reconcilable_result_builder>::consume_end_of_partition() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/querier.hh:62
 (inlined by) seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0>::consume_end_of_partition<query::clustering_position_tracker<reconcilable_result_builder> >(query::clustering_position_tracker<reconcilable_result_builder>&) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/mutation_compactor.hh:274
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)0, query::clustering_position_tracker<reconcilable_result_builder> >::consume_end_of_partition() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/mutation_compactor.hh:393
 (inlined by) stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > >::consume_end_of_partition() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/mutation_reader.hh:365
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >::consume(partition_end&&) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/flat_mutation_reader.hh:240
 (inlined by) _ZNO17mutation_fragment7consumeIN20flat_mutation_reader4impl16consumer_adapterI35stable_flattened_mutations_consumerI17compact_for_queryIL19emit_only_live_rows0EN5query27clustering_position_trackerI27reconcilable_result_builderEEEEEEEEDcRT_ at /build/scylla-server-3.0.8-0.20190627.23da53c4f/mutation_fragment.hh:444
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >::operator()(mutation_fragment&&) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/flat_mutation_reader.hh:220
 (inlined by) seastar::bool_class<seastar::stop_iteration_tag> std::__invoke_impl<seastar::bool_class<seastar::stop_iteration_tag>, flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >&, mutation_fragment>(std::__invoke_other, flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >&, mutation_fragment&&) at /opt/scylladb/include/c++/7/bits/invoke.h:60
 (inlined by) std::__invoke_result<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >&, mutation_fragment>::type std::__invoke<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >&, mutation_fragment>(flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >&, mutation_fragment&&) at /opt/scylladb/include/c++/7/bits/invoke.h:96
 (inlined by) std::result_of<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > >& (mutation_fragment&&)>::type std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >::operator()<mutation_fragment>(mutation_fragment&&) const at /opt/scylladb/include/c++/7/bits/refwrap.h:356
 (inlined by) seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/flat_mutation_reader.hh:172
 (inlined by) seastar::internal::do_until_state<seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>::run_and_dispose() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/./seastar/core/future-util.hh:462
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/reactor.cc:2695
seastar::reactor::run_some_tasks() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/reactor.cc:3118
seastar::reactor::run_some_tasks() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/util/log.hh:313
 (inlined by) seastar::reactor::run() at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/reactor.cc:3265
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/reactor.cc:4334
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::posix_thread::start_routine(void*) at /build/scylla-server-3.0.8-0.20190627.23da53c4f/seastar/core/posix.cc:52
start_thread at ??:?
clone at ??:?
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Jul 30, 2019

The vector being expanded is

class reconcilable_result_builder {
    const schema& _schema;
    const query::partition_slice& _slice;

    std::vector<partition> _result;
@tzach

This comment has been minimized.

Copy link
Contributor

@tzach tzach commented Aug 1, 2019

@r4fek do you know what CQL query cause the oversized allocation?

@r4fek

This comment has been minimized.

Copy link
Author

@r4fek r4fek commented Aug 1, 2019

@tzach no idea. All I know is that this warning usually pops up after restarting the node.

@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 1, 2019

Do you have many small partitions that you delete? It looks like we fetched a large number of tombstones from another node.

@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 1, 2019

Or alternatively, can your driver not be using paging?

@avikivity avikivity changed the title seastar_memory - oversized allocation seastar_memory - oversized allocation in reconcilable_result Aug 1, 2019
@r4fek

This comment has been minimized.

Copy link
Author

@r4fek r4fek commented Aug 2, 2019

@avikivity I use cassandra-driver for Python. It supports automatic query paging. Having large amounts of tombstones is highly possible though.

@slivne slivne added this to the 3.2 milestone Aug 8, 2019
nyh added a commit that referenced this issue Aug 11, 2019
Merged patch series from Avi Kivity:

In rare but valid cases (reconciling many tombstones, paging disabled),
a reconciled_result can grow large. This triggers large allocation
warnings. Switch to chunked_vector to avoid the large allocation.
In passing, fix chunked_vector's begin()/end() const correctness, and
add the reverse iterator function family which is needed by the conversion.

Fixes #4780.

Tests: unit (dev)

Commit Summary

    utils: chunked_vector: make begin()/end() const correct
    utils::chunked_vector: add rbegin() and related iterators
    reconcilable_result: use chunked_vector to hold partitions
@r4fek

This comment has been minimized.

Copy link
Author

@r4fek r4fek commented Aug 12, 2019

Can we backport this to 3.0 @avikivity?

@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 12, 2019

I'd like to, but I also want to let it cook for a while in master to see it doesn't cause regressions.

avikivity added a commit that referenced this issue Nov 17, 2019
Usually, a reconcilable_result holds very few partitions (1 is common),
since the page size is limited by 1MB. But if we have paging disabled or
if we are reconciling a range full of tombstones, we may see many more.
This can cause large allocations.

Change to chunked_vector to prevent those large allocations, as they
can be quite expensive.

Fixes #4780.

(cherry picked from commit 093d2cd)
avikivity added a commit that referenced this issue Nov 17, 2019
Usually, a reconcilable_result holds very few partitions (1 is common),
since the page size is limited by 1MB. But if we have paging disabled or
if we are reconciling a range full of tombstones, we may see many more.
This can cause large allocations.

Change to chunked_vector to prevent those large allocations, as they
can be quite expensive.

Fixes #4780.

(cherry picked from commit 093d2cd)
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Nov 17, 2019

Backported to 3.0 and 3.1; removing backport candidate label.

avikivity added a commit that referenced this issue Nov 17, 2019
begin() of a const vector should return a const_iterator, to avoid
giving the caller the ability to mutate it.

This slipped through since iterator's constructor does a const_cast.

Noticed by code inspection.

(cherry picked from commit df6faae)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 17, 2019
Needed as an std::vector replacement.

(cherry picked from commit eaa9a5b)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 17, 2019
Usually, a reconcilable_result holds very few partitions (1 is common),
since the page size is limited by 1MB. But if we have paging disabled or
if we are reconciling a range full of tombstones, we may see many more.
This can cause large allocations.

Change to chunked_vector to prevent those large allocations, as they
can be quite expensive.

Fixes #4780.

(cherry picked from commit 093d2cd)
avikivity added a commit that referenced this issue Nov 17, 2019
begin() of a const vector should return a const_iterator, to avoid
giving the caller the ability to mutate it.

This slipped through since iterator's constructor does a const_cast.

Noticed by code inspection.

(cherry picked from commit df6faae)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 17, 2019
Needed as an std::vector replacement.

(cherry picked from commit eaa9a5b)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 17, 2019
Usually, a reconcilable_result holds very few partitions (1 is common),
since the page size is limited by 1MB. But if we have paging disabled or
if we are reconciling a range full of tombstones, we may see many more.
This can cause large allocations.

Change to chunked_vector to prevent those large allocations, as they
can be quite expensive.

Fixes #4780.

(cherry picked from commit 093d2cd)
avikivity added a commit that referenced this issue Nov 19, 2019
begin() of a const vector should return a const_iterator, to avoid
giving the caller the ability to mutate it.

This slipped through since iterator's constructor does a const_cast.

Noticed by code inspection.

(cherry picked from commit df6faae)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 19, 2019
Needed as an std::vector replacement.

(cherry picked from commit eaa9a5b)

Prerequisite for #4780.
avikivity added a commit that referenced this issue Nov 19, 2019
Usually, a reconcilable_result holds very few partitions (1 is common),
since the page size is limited by 1MB. But if we have paging disabled or
if we are reconciling a range full of tombstones, we may see many more.
This can cause large allocations.

Change to chunked_vector to prevent those large allocations, as they
can be quite expensive.

Fixes #4780.

(cherry picked from commit 093d2cd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.