kafka_consumer: return freed memory to the OS with malloc_trim after each run

[piochelepiotr]

What does this PR do?

Call glibc malloc_trim(0) after every check run (in a finally), so per-arena free memory is returned to the OS each cycle. It is resolved once at import via ctypes and is a no-op on platforms without it (macOS, musl).

Motivation

librdkafka runs roughly one thread per broker, and on high-core hosts glibc spreads allocations across many per-thread arenas (default cap 8 x ncpu). glibc only ever returns memory from the top of an arena's heap and never coalesces free chunks across arenas, so the transient allocations each run leaves behind accumulate as retained-but-unused memory and agent RSS ratchets upward. malloc_trim(0) walks the arenas and hands their free memory back to the OS, keeping RSS bounded without requiring a MALLOC_ARENA_MAX environment variable. Overhead is negligible for this check (runs are network-bound at several seconds).

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 77.27%
• Overall Coverage: 92.10% (+3.81%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c235f70 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 43a61df3fd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[temporal-github-worker-1]

Backport evaluation: View Slack thread

[dd-octo-sts]

The backport to 7.82.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-7.82.x 7.82.x
# Navigate to the new working tree
cd .worktrees/backport-7.82.x
# Create a new branch
git switch --create backport-24553-to-7.82.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 433b81a58be72bd80dc86bdfc618a1af28412aac
# Push it to GitHub
git push --set-upstream origin backport-24553-to-7.82.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-7.82.x

Then, create a pull request where the base branch is 7.82.x and the compare/head branch is backport-24553-to-7.82.x.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run