[FLINK-39404][runtime] HardwareDescription reports incorrect CPU cores in containerized environments with fractional CPU limits

[Dennis-Mircea]

What is the purpose of the change

This PR fixes inaccurate CPU core reporting in containerized environments (Kubernetes, YARN) where fractional CPU limits are used (e.g. 0.5 cores).

Currently, Hardware.getNumberCPUCores() relies on Runtime.getRuntime().availableProcessors(), which returns an int, ceiling-rounding fractional container CPU limits (0.5 → 1). This causes:

1. Misleading Web UI: The Task Manager pages display "1 CPU core" when only 0.5 is allocated.
2. Thread pool over-provisioning: ClusterEntrypointUtils computes 4 * ceil(0.5) = 4 threads instead of ceil(4 * 0.5) = 2.

Brief change log

• Hardware.java:

  • Added getContainerCpuLimit() - reads the actual container CPU limit from Linux cgroup files (v2: /sys/fs/cgroup/cpu.max, v1: cpu.cfs_quota_us / cpu.cfs_period_us). Returns the fractional value (e.g. 0.5) or -1 if not in a container / no limit set.
  • Added getNumberCPUCoresAsDouble() - returns the fractional CPU core count (container limit with fallback to availableProcessors()). Use this when fractional precision matters (display, arithmetic before rounding).

• HardwareDescription.java:

  • Changed numberOfCPUCores from int to double (field, constructor, getter, equals, toString).
  • extractFromSystem() now uses Hardware.getNumberCPUCoresAsDouble().
  • The REST API JSON field cpuCores now emits fractional values (e.g. 0.5 instead of 1).

• ClusterEntrypointUtils.java:

  • Changed 4 * Hardware.getNumberCPUCores() to (int) Math.ceil(4 * Hardware.getNumberCPUCoresAsDouble()) so the multiplication happens before ceiling, avoiding thread over-provisioning (0.5 CPU → 2 threads instead of 4).

Verifying this change

• On a non-container environment (macOS, bare-metal Linux without cgroup limits): getContainerCpuLimit() returns -1, getNumberCPUCores() falls back to availableProcessors(), representing no behavioral change.
• On a container with fractional CPU (e.g. Kubernetes with cpu: 0.5):
  • getContainerCpuLimit() returns 0.5
  • getNumberCPUCoresAsDouble() returns 0.5
  • getNumberCPUCores() returns 1 (ceiling)
  • Web UI displays 0.5 instead of 1
  • IO executor pool: ceil(4 * 0.5) = 2 threads instead of 4

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• d8cc821 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[Dennis-Mircea]

@flinkbot run azure re-run the last Azure build

[Dennis-Mircea]

@flinkbot run azure re-run the last Azure build

[milindl]

Hi @Dennis-Mircea , this PR changes the public-facing REST API in a non-backwards compatible way. For something like this, we might need to wait for a new major release. Or else, instead of changing an existing field, we might need to add a new one.

[Dennis-Mircea]

Hi @Dennis-Mircea , this PR changes the public-facing REST API in a non-backwards compatible way. For something like this, we might need to wait for a new major release. Or else, instead of changing an existing field, we might need to add a new one.

Hi @milindl, thanks for the comment! Sounds good both ways. I also open the https://lists.apache.org/thread/ny2vj4b8yofl9j0q8zh2cwg4rjpdc566 thread discussion. I would appreciate some support in-here to push forward and find the best approach that we can take on this one.

[davidradl]

Please can you add tests for this change I am curious what happens if we have a third of a CPU - i.e. irrational

[Dennis-Mircea]

Good point. A truly irrational CPU share isn't actually expressible through cgroups: the kernel stores quota / period in microseconds, so a user request like "one-third of a CPU" ends up on disk as something like 33333 / 100000. The realistic equivalents are therefore truncated decimals such as 0.3 or 0.33333.

I added tests in HardwareTest covering both:

• testCgroupV{1,2}SubCoreLimit - typical sub-core input: 30000 / 100000 -> returns 0.3 exactly.
• testCgroupV{1,2}RepeatingFractionalLimit / testCgroupV1OneThirdCpuLimit - the "one-third" case: 33333 / 100000 -> returns 0.33333, also asserted to stay within 1e-4 of the mathematical 1.0 / 3.0

[davidradl]

my AI says :
High Priority Issues
🟠 HIGH - Division by Zero Risk: Period validation in cgroup v1 code needs verification to ensure it happens before division operation

Medium Priority Issues
Missing unit tests for new cgroup detection methods
No handling for cgroup v2 hybrid mode scenarios
Missing documentation distinguishing when to use getNumberCPUCores() vs getNumberCPUCoresAsDouble()
Thread pool multiplier (4x) may still over-provision for fractional CPUs
Low Priority Issues
Magic string "max" should be constant
Redundant file existence checks
Format specifier inconsistency in toString()
Path traversal validation missing
Test fixtures need fractional value testing

[Dennis-Mircea]

Thanks for the thorough review! Going through each point:

🟠 HIGH - Division by zero in cgroup v1

This is already guarded. In Hardware#getCpuLimitFromCgroupV1, both values are validated before the division:

if (quota <= 0) { ... return -1; }      // v1 "-1" sentinel + negative/zero guard
if (period <= 0) { ... return -1; }     // explicit zero/negative period guard
double cpuLimit = (double) quota / period;

The same is true for v2 (if (quota > 0 && period > 0)). No change needed.

Missing unit tests / fractional test fixtures

Addressed. The two cgroup helpers are now @VisibleForTesting and take injected Path parameters, which lets HardwareTest cover them with temp files. Added 13 new tests covering: fractional limits (0.5, 1.5), integer limits (2.0), "max" unlimited (v2), -1 unlimited (v1), zero period, malformed content, non-numeric values, and missing files.

cgroup v2 hybrid mode

The existing v2 -> v1 fallback already covers this. In hybrid mode the cpu controller typically stays on the v1 hierarchy, so the v1 fallback picks it up correctly. Added a note to the getContainerCpuLimit Javadoc to make this explicit.

Docs: getNumberCPUCores() vs getNumberCPUCoresAsDouble()

The guidance already existed on getNumberCPUCoresAsDouble ("Use this method when the fractional value matters… For call sites that need an integer, use getNumberCPUCores() instead"). Added the reverse cross-reference on getNumberCPUCores as well.

Thread pool 4× multiplier may over-provision for fractional CPUs

The only thread-pool site updated by this PR is ClusterEntrypointUtils#getPoolSize (cluster IO executor), which now computes:

(int) Math.ceil(4 * Hardware.getNumberCPUCoresAsDouble())

This actually reduces over-provisioning compared to the previous 4 * getNumberCPUCores(): on a container limited to 0.5 CPU, the old code produced 4 * 1 = 4 threads, while the new code produces ceil(4 * 0.5) = 2. The multiplication happens before the cast to int, so fractional limits are honored rather than rounded up first. All other thread-pool sites still use the unchanged int variant (getNumberCPUCores).

Magic string "max"

Extracted to Hardware.CGROUP_V2_UNLIMITED.

Redundant file existence checks

This is kept on purpose as they avoid throwing and logging NoSuchFileException on the common non-Linux / non-containerized path, which would be noisy in normal operation.

Format specifier inconsistency in toString()

Fixed: HardwareDescription#toString now uses %.2f for numberOfCPUCores instead of %s.

Path traversal validation

N/A in this case. The cgroup paths are hardcoded private static final String constants (/sys/fs/cgroup/cpu.max, /sys/fs/cgroup/cpu/cpu.cfs_quota_us, /sys/fs/cgroup/cpu/cpu.cfs_period_us). There is no user-supplied input in the path construction.

Let me know if anything's unclear or you'd like a different approach on any of the above.

[Dennis-Mircea]

@flinkbot run azure re-run the last Azure build