Skip to content

feat: respect CPU and cgroup limits in server-ng and connectors stats#3647

Merged
mmodzelewski merged 5 commits into
masterfrom
connectors-scoped-stats
Jul 11, 2026
Merged

feat: respect CPU and cgroup limits in server-ng and connectors stats#3647
mmodzelewski merged 5 commits into
masterfrom
connectors-scoped-stats

Conversation

@mmodzelewski

@mmodzelewski mmodzelewski commented Jul 10, 2026

Copy link
Copy Markdown
Member

The server-ng and connectors runtime stats endpoints reported
host-wide sysinfo numbers: a cpuset-confined or memory-capped
process showed its neighbors' CPU load and a memory total it can
never allocate, leaking host sizing on multi-tenant machines. The
legacy server already scoped these (#3615) but kept the logic
inline.

Extract that logic into a shared system_stats crate: SystemProbe
scopes total CPU usage to the allowed core set (snapshotted at
startup, before shard threads pin themselves) and memory totals
to the effective cgroup cap, falling back to host-wide values for
unconfined processes. All three binaries now probe through it.

On confined hosts the reported total_cpu_usage, total_memory and
available_memory now reflect the confinement, so dashboards and
alerts keyed to the old host-wide values will see lower numbers.

The connectors runtime also keeps its sysinfo System alive across
captures, so CPU usage reflects real deltas instead of the zero
first sample of a freshly created System on every request.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 10, 2026
CPU set and cgroup memory cap, but server-ng and the connectors
runtime kept reporting host-wide sysinfo numbers: a confined
instance showed its neighbors' CPU load and a memory total it
can never allocate, leaking host sizing to anyone who can read
the stats endpoint.

Extract the scoping into a shared system_stats crate
(scoped_total_cpu_usage, cgroup_scoped_memory) and use it from
the legacy server, server-ng and the connectors runtime.

The connectors runtime also built a fresh sysinfo System on
every call, so its CPU fields (deltas over the previous refresh)
were always zero. Keep one process-wide System so readings are
real from the second probe on.
@mmodzelewski mmodzelewski force-pushed the connectors-scoped-stats branch from fbd018f to 6f2c3fa Compare July 10, 2026 14:22
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.98678% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.79%. Comparing base (a860cef) to head (a3338e2).

Files with missing lines Patch % Lines
core/system_stats/src/lib.rs 84.74% 14 Missing and 4 partials ⚠️
core/server/src/shard/system/stats.rs 91.04% 6 Missing ⚠️
core/server-ng/src/main.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3647       +/-   ##
=============================================
- Coverage     73.42%   57.79%   -15.64%     
  Complexity      937      937               
=============================================
  Files          1285     1284        -1     
  Lines        140958   124703    -16255     
  Branches     116848   100634    -16214     
=============================================
- Hits         103503    72071    -31432     
- Misses        34228    49570    +15342     
+ Partials       3227     3062      -165     
Components Coverage Δ
Rust Core 54.15% <88.98%> (-19.49%) ⬇️
Java SDK 62.44% <ø> (ø)
C# SDK 71.40% <ø> (-0.71%) ⬇️
Python SDK 91.65% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 91.26% <ø> (-0.10%) ⬇️
Go SDK 42.28% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/runtime/src/main.rs 85.71% <100.00%> (+0.05%) ⬆️
core/connectors/runtime/src/stats.rs 70.58% <100.00%> (-1.64%) ⬇️
core/connectors/sdk/src/api.rs 100.00% <ø> (ø)
core/server-ng/src/responses.rs 32.42% <100.00%> (-0.72%) ⬇️
core/server/src/main.rs 63.34% <100.00%> (+0.10%) ⬆️
core/system_stats/src/cgroup_memory.rs 86.30% <ø> (ø)
core/server-ng/src/main.rs 0.00% <0.00%> (ø)
core/server/src/shard/system/stats.rs 90.27% <91.04%> (+6.94%) ⬆️
core/system_stats/src/lib.rs 84.74% <84.74%> (ø)

... and 316 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one finding lands outside the diff so it goes here: ConnectorRuntimeStats in core/connectors/sdk/src/api.rs now carries cpuset/cgroup-scoped total_cpu_usage / total_memory / available_memory but the fields have no rustdoc - legacy Stats got per-field scoping notes in #3615, worth mirroring the memory ones here (hold the cpu wording until the affinity issue below is fixed, right now it would document a value the servers don't produce). release notes should also mention the value change for server-ng stats: total/available memory become the cgroup cap on confined hosts. wire-compatible, but dashboards and SDK consumers see different numbers.

Comment thread core/server-ng/src/responses.rs Outdated
Comment thread core/server/src/shard/system/stats.rs Outdated
Comment thread core/system_stats/src/lib.rs
Comment thread core/system_stats/src/lib.rs Outdated
Comment thread core/system_stats/src/lib.rs Outdated
Comment thread core/connectors/runtime/src/stats.rs Outdated
Comment thread core/connectors/runtime/src/stats.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 10, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all findings from the earlier review are fixed as of fb6d2f8 - verified each one: boot-time capture_allowed_cpus() snapshot runs on the main thread before any shard pinning in all three binaries, and the .get()-not-get_or_init read means a missed capture falls back to host-global instead of lazily grabbing a pinned thread's mask; System::new() swap landed everywhere including the two pre-existing sites; positive averaging test + None-branch assertion added; cgroup_scoped_memory got the calling-process doc and went private; Pid bound once. cargo test -p system_stats green, fmt + clippy clean.

two leftovers, neither blocking:

  • ConnectorRuntimeStats in core/connectors/sdk/src/api.rs still has no per-field rustdoc on total_cpu_usage / total_memory / available_memory, which now report cpuset/cgroup-scoped values on confined hosts. legacy Stats in core/common/src/types/stats/mod.rs got per-field scoping notes in #3615 - worth mirroring, and the cpu wording is safe to write now that the affinity snapshot landed.
  • the PR description still names scoped_total_cpu_usage / cgroup_scoped_memory as the crate's api (both are private now - the surface is SystemProbe::capture + capture_allowed_cpus) and doesn't mention the affinity boot snapshot or that stats values change for dashboards on confined hosts. worth a refresh before merge, it doubles as the release note.

@mmodzelewski mmodzelewski merged commit 2ec7d89 into master Jul 11, 2026
95 checks passed
@mmodzelewski mmodzelewski deleted the connectors-scoped-stats branch July 11, 2026 05:00
@github-actions github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants