[PoC] Jemalloc profiler

[squadgazzz]

Description

This is a proof of concept for integrating the Jemalloc memory profiler into various services to collect memory dumps on demand. Profiling with the Jemalloc allocator is currently considered the most resource-efficient option (though this still needs to be validated in prod) and doesn’t require running additional applications, which is a major advantage.

The implementation is based on various open-source projects(e.g. https://github.com/tikv/tikv/blob/master/components/tikv_alloc/src/jemalloc.rs#L327)

Changes

• Allocator selection: Mimalloc is currently used across all services and provides the best performance, so Jemalloc is only considered for collecting memory dumps. The allocator is selected based on the feature selected during the compilation. The Dockerfile is updated accordingly. This allows granular selection of the Jemalloc allocator. This PR contains implementation only for autopilot. Other crates will be supported in follow-up PRs. The major disadvantage for this approach is that the binary needs to be recompiled. Selecting the memory allocator after seems to be impossible.
• Profiler activation: The Jemalloc profiler only records allocations that occur while profiling is active. By default, profiling is disabled. This is useful when allocations during service warm-up consume most of the memory and memory leaks slowly later.
• Profiler control: This is implemented using a combination of a USR2 signal and environment variables. TODO: update it. This approach is much easier than introducing an HTTP API with auth, etc:
  • Set the MEM_DUMP_PATH ENV param to specify the dump output directory.
  • Set the PROFILER_COMMAND ENV param with one of the values: TODO: update it
    • enable - activates profiling
    • disable - disables profiling
    • dump - stores the recorded dump
    • run_for(<Duration, e.g. 1h>) - automatically activates profiling, records the dump for the provided duration, stores the dump, and disables profiling.
  • Send USR2 using kill -USR2 <pid> to execute the command specified in the previous step.

Further automation

Based on the control flexibility, some infra automation can be implemented based on the resource consumption. For example, once memory reaches 50%, start profiling and record a dump when it reaches 90%.

How to test

Try running it locally and in staging. That would require an infra change with PVC creation(it was already done for the Heaptrack profiler)

[Copilot]

Pull Request Overview

This PR introduces jemalloc memory profiling capabilities to replace the existing mimalloc allocator. The implementation adds signal-based memory profiling that can be triggered via SIGUSR2 to collect heap dumps for performance analysis.

• Replaces mimalloc with jemalloc as the global allocator
• Adds a JemallocMemoryProfiler that responds to SIGUSR2 signals to collect heap dumps
• Integrates the profiler into the autopilot service startup process

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
crates/shared/src/lib.rs	Adds new alloc module to shared library
crates/shared/src/alloc.rs	Implements JemallocMemoryProfiler with signal handling and dump functionality
crates/shared/Cargo.toml	Adds tikv-jemalloc-ctl dependency for profiling controls
crates/autopilot/src/run.rs	Integrates memory profiler into autopilot startup
crates/autopilot/src/main.rs	Switches global allocator from mimalloc to jemalloc
crates/autopilot/Cargo.toml	Replaces mimalloc with tikv-jemallocator dependency
Dockerfile	Adds make package for build dependencies

[github-actions]

This pull request has been marked as stale because it has been inactive a while. Please update this pull request or it will be automatically closed.

[github-actions]

This pull request has been marked as stale because it has been inactive a while. Please update this pull request or it will be automatically closed.

[MartinquaXD]

Given that you are waiting for the maintainers to unblock you, can this PR and the other one temporarily be closed?