[improve][broker] Support cgroup v2 by using jdk.internal.platform.Metrics in Pulsar Loadbalancer

[coderzc]

Master Issue: #16601

Motivation

The Pulsar load balancer detects CPU limits using cgroup v1 API, and the jdk.internal.platform.Metrics already support cgroup (V1, v2) so we should use jdk.internal.platform.Metrics to get the cgroup metrics.

Reference: https://code.yawk.at/java/17/java.base/jdk/internal/platform/

Modifications

Use jdk.internal.platform.Metrics to get the cgroup metrics in the LinuxInfoUtils.

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as testCGroupMetrics.

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API: (yes / no)
• The schema: (yes / no / don't know)
• The default values of configurations: (yes / no)
• The wire protocol: (yes / no)
• The rest endpoints: (yes / no)
• The admin cli options: (yes / no)
• Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

• doc-required
  (Your PR needs to update docs and you will update later)

• doc-not-needed
  (Please explain why)

• doc
  (Your PR contains doc changes)

• doc-complete
  (Docs have been already added)

[coderzc]

/pulsarbot run-failure-checks

[nodece]

LGTM

[coderzc]

/pulsarbot run-failure-checks

[codelipenghui]

We should check if the test really uses the jdk.internal.platform.Metrics not the old way?

[coderzc]

Good idea, we can assert metrics != null.

[coderzc]

/pulsarbot run-failure-checks

[asafm]

However, LB also requires other signals such as dict_memory_usage_in_percent_cgroup, network_usage_in_percent_cgroup, which require separate limits for their percentage computation.

Can you please expand on that? I don't any method providing usage and limit of direct memory in Pulsar. I do see usage and limit for physical memory (in the container) which as we said we can take from the mxBean which provides total and usage of memory.

Regarding network usage - yes, that I was surprised to see that the operating system didn't include, and we resort to reading operating system specific files to obtain that. I posted a question in the OpenJDK general discussion mailing to get the motivation for that ( I did search their JIRA issue database and also googled the mailing list site but found nothing).

Maybe we can tweak the code to ignore the limit and use the signal as-is if they are already in the *_usage_in_percent_cgroup form.

I guess that requires a separate PR. Maybe we can push that PR in but add an issue to refactor (delete it out) the dependency on limit and only use percentage (which should be enough) or refactor to avoid using limit and be more specific like ResourceUsage.CpuUsage, MemoryUsage, NetworkUsage, each providing it's own method, where the CPU would only provide percentage.

[heesung-sn]

Can you please expand on that? I don't any method providing usage and limit of direct memory in Pulsar. I do see usage and limit for physical memory (in the container) which as we said we can take from the mxBean which provides total and usage of memory.

I see this method is used to collect direct memory usage.

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LoadManagerShared.java

Lines 228 to 229 in 5354153

	systemResourceUsage.setDirectMemory(new ResourceUsage((double) (getJvmDirectMemoryUsed() / MIBI),
	(double) (DirectMemoryUtils.jvmMaxDirectMemory() / MIBI)));

I guess that requires a separate PR. Maybe we can push that PR in but add an issue to refactor (delete it out) the dependency on limit and only use percentage (which should be enough) or refactor to avoid using limit and be more specific like ResourceUsage.CpuUsage, MemoryUsage, NetworkUsage, each providing it's own method, where the CPU would only provide percentage.

Yes, a separate PR makes sense to me.

[github-actions]

The pr had no activity for 30 days, mark with Stale label.

[nodece]

Any updates? I think this PR makes sense to use the cgroup v2.

[heesung-sn]

For backward compatibility, don't we need to multiply the limit in calculateBrokerHostUsage?

public void calculateBrokerHostUsage() {
... 
       double totalCpuLimit = getTotalCpuLimit(isCGroupsEnabled);
        if (isCGroupsEnabled && metrics != null && getCpuUsageMethod != null) {
            // cgroup cpuUsage is already scaled, [0.0, 1.0]
            usage.setCpu(new ResourceUsage(cpuUsage * totalCpuLimit, totalCpuLimit));
        } else {
            usage.setCpu(new ResourceUsage(cpuUsage, totalCpuLimit));
        }
        

[coderzc]

The Metrics.getCpuUsage will return the aggregate time, so I think we don't need to multiply the limit.

    /**
     * Returns the aggregate time, in nanoseconds, consumed by all
     * tasks in the Isolation Group.
     *
     * @return Time in nanoseconds, -1 if unknown or
     *         -2 if the metric is not supported.
     *
     */
    public long getCpuUsage();

[heesung-sn]

I see. Thanks for the check.

Ref:
https://github.com/openjdk/jdk17/blob/master/src/java.base/share/classes/jdk/internal/platform/Metrics.java

https://github.com/openjdk/jdk17/tree/master/src/java.base/linux/classes/jdk/internal/platform

[coderzc]

I will update this PR as soon.

[coderzc]

I guess that requires a separate PR. Maybe we can push that PR in but add an issue to refactor (delete it out) the dependency on limit and only use percentage (which should be enough) or refactor to avoid using limit and be more specific like ResourceUsage.CpuUsage, MemoryUsage, NetworkUsage, each providing it's own method, where the CPU would only provide percentage.

Yes, we need a separate PR to refactor it.

[lhotari]

We need this to be included in branch-3.0 and branch-2.11 asap. I backported the changes to branch-2.10 with all dependencies in #20659.

This is becoming urgent.

AKS Kubernetes 1.25+ switches to use cgroup v2:
https://github.com/Azure/AKS/releases/tag/2023-03-05

AKS Kubernetes 1.24 goes End-of-life on July 30, 2023.

GKE contains to have a way to select between cgroup v1 & cgroup v2:
https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options
GKE will default to cgroup v2 in new Kubernetes 1.26 clusters or node pools.
AWS EKS v1.26 nodes will default to cgroup v2.