Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Oversized allocation: 409600 bytes in kafka::create_topics_api #16758

Closed
ballard26 opened this issue Feb 27, 2024 · 1 comment · Fixed by #16982
Closed

Oversized allocation: 409600 bytes in kafka::create_topics_api #16758

ballard26 opened this issue Feb 27, 2024 · 1 comment · Fixed by #16982
Assignees
@BenPope
Labels
area/oversized-allocation kind/bug Something isn't working sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low.

Comments

@ballard26
Copy link
Contributor

Occurs while sending a topic creation request for 4k+ topics in #16463 .
Backtrace;

/home/ubuntu/tests/results/2024-02-26--002/ManyPartitionsTest/test_many_topics_throughput/1/RedpandaService-0-139946360787280/ip-172-31-1-54/redpanda.log:861:WARN  2024-02-26 21:05:38,101 [shard 0:main] seastar_memory - oversized allocation: 409600 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #53]
{/opt/redpanda/libexec/redpanda} 0x93c83d3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
{/opt/redpanda/libexec/redpanda} 0x902fefc: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839
 (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911
 (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645
{/opt/redpanda/libexec/redpanda} 0x903d1d0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
{/opt/redpanda/libexec/redpanda} 0x4737618: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<cluster::custom_assignable_topic_configuration>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<cluster::custom_assignable_topic_configuration> >::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<cluster::custom_assignable_topic_configuration> >(std::__1::allocator<cluster::custom_assignable_topic_configuration>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<cluster::custom_assignable_topic_configuration>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration> >::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) std::__1::vector<decltype(to_cluster_type(*(fp))), std::__1::allocator<decltype(to_cluster_type(*(fp)))> > kafka::to_cluster_type<fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false> >(fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false>, fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/kafka/server/handlers/topics/topic_utils.h:127
 (inlined by) kafka::handler_template<kafka::create_topics_api, (short)0, (short)6, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/kafka/server/handlers/create_topics.cc:320
{/opt/redpanda/libexec/redpanda} 0x43ab80a: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:83
{/opt/redpanda/libexec/redpanda} 0x90fd3ff: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x9100b71: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x90fdd06: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
{/opt/redpanda/libexec/redpanda} 0x8ffce80: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
{/opt/redpanda/libexec/redpanda} 0x8ffb278: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
{/opt/redpanda/libexec/redpanda} 0x2ccffb6: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/redpanda/application.cc:416
{/opt/redpanda/libexec/redpanda} 0x941e759: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/lib/libc.so.6} 0x2d58f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2d648: __libc_start_main at ??:0
{/opt/redpanda/libexec/redpanda} 0x2cc87e4: _start at ??:0
@michael-redpanda michael-redpanda added the sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. label Mar 1, 2024
@piyushredpanda
Copy link
Contributor

We want to get this fixed within the next two weeks, @michael-redpanda and @aanthony-rp

BenPope added a commit to BenPope/redpanda that referenced this issue Mar 8, 2024
@BenPope
cluster: Avoid oversize alloc for topic configuration
a86f07a 
`topic_configuration` is 392 bytes
`custom_assignable_topic_configuration`	is 416 bytes

Switch vectors of them to `chunked_vector`

Fixes redpanda-data#16758

Signed-off-by: Ben Pope <ben@redpanda.com>
BenPope added a commit to BenPope/redpanda that referenced this issue Mar 8, 2024
@BenPope
cluster: Avoid oversize alloc for topic configuration
d58c9fa 
`topic_configuration` is 392 bytes
`custom_assignable_topic_configuration`	is 416 bytes

Switch vectors of them to `chunked_vector`

Fixes redpanda-data#16758

Signed-off-by: Ben Pope <ben@redpanda.com>
@BenPope BenPope mentioned this issue Mar 8, 2024
Merged
6 tasks
BenPope added a commit to BenPope/redpanda that referenced this issue Mar 12, 2024
@BenPope
cluster: Avoid oversize alloc for topic configuration
fd81a9d 
`topic_configuration` is 392 bytes
`custom_assignable_topic_configuration`	is 416 bytes

Switch vectors of them to `chunked_vector`

Fixes redpanda-data#16758

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit d58c9fa)

Conflicts:
  src/v/compat/cluster_generator.h (didn't provide the size)
BenPope added a commit to BenPope/redpanda that referenced this issue Mar 12, 2024
@BenPope
cluster: Avoid oversize alloc for topic configuration
01e2011 
`topic_configuration` is 392 bytes
`custom_assignable_topic_configuration`	is 416 bytes

Switch vectors of them to `chunked_vector`

Fixes redpanda-data#16758

Signed-off-by: Ben Pope <ben@redpanda.com>
(cherry picked from commit d58c9fa)

Conflicts:
  src/v/compat/cluster_generator.h (didn't provide the size)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BenPope BenPope

Labels
area/oversized-allocation kind/bug Something isn't working sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cluster: Avoid oversize allocs for topic creation and configuration BenPope/redpanda
4 participants
@BenPope @ballard26 @piyushredpanda @michael-redpanda