[Bug] Fix concurrent modify opentelemetry record in calculate consumer lag

[lizhimins]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

Linux 5.19

RocketMQ version

[5.3.2, 5.3.3]

JDK Version

JDK17

Describe the Bug

The 'record' method uses HashMap which may cause concurrent access issues when Pull thread executes Pop callbacks

Steps to Reproduce

The 'record' method uses HashMap which may cause concurrent access issues when Pull thread executes Pop callbacks

What Did You Expect to See?

No concurrent access

What Did You See Instead?

Concurrent access hashmap

io.opentelemetry.sdk.metrics.internal.state.AsynchronousMetricStorage.recordPoint(AsynchronousMetricStorage.java:152)
	at io.opentelemetry.sdk.metrics.internal.state.AsynchronousMetricStorage.record(AsynchronousMetricStorage.java:116)
	at io.opentelemetry.sdk.metrics.internal.state.SdkObservableMeasurement.doRecord(SdkObservableMeasurement.java:132)
	at io.opentelemetry.sdk.metrics.internal.state.SdkObservableMeasurement.record(SdkObservableMeasurement.java:107)
	at org.apache.rocketmq.broker.metrics.BrokerMetricsManager.lambda$initLagAndDlqMetrics$15(BrokerMetricsManager.java:606)
	at org.apache.rocketmq.broker.metrics.BrokerMetricsManager$$Lambda$548/0x000000080077e040.accept(Unknown Source)
	at org.apache.rocketmq.broker.metrics.ConsumerLagCalculator.calculate(ConsumerLagCalculator.java:252)
	at org.apache.rocketmq.broker.metrics.ConsumerLagCalculator$$Lambda$589/0x00000008007d1c40.accept(Unknown Source)
	at org.apache.rocketmq.broker.longpolling.PopCommandCallback.accept(PopCommandCallback.java:47)
	at org.apache.rocketmq.broker.processor.PopMessageProcessor$$Lambda$590/0x00000008007d3440.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.14.13-AJDKfp3/ArrayList.java:1541)
	at org.apache.rocketmq.broker.processor.PopMessageProcessor.lambda$processRequest$6(PopMessageProcessor.java:547)
	at org.apache.rocketmq.broker.processor.PopMessageProcessor$$Lambda$461/0x0000000800632040.apply(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniApplyNow(java.base@11.0.14.13-AJDKfp3/CompletableFuture.java:680)
	at java.util.concurrent.CompletableFuture.uniApplyStage(java.base@11.0.14.13-AJDKfp3/CompletableFuture.java:658)
	at java.util.concurrent.CompletableFuture.thenApply(java.base@11.0.14.13-AJDKfp3/CompletableFuture.java:2094)
	at org.apache.rocketmq.broker.processor.PopMessageProcessor.processRequest(PopMessageProcessor.java:544)
	at org.apache.rocketmq.broker.longpolling.PopLongPollingService.lambda$wakeUp$1(PopLongPollingService.java:251)
	at org.apache.rocketmq.broker.longpolling.PopLongPollingService$$Lambda$488/0x00000008006f7040.run(Unknown Source)
	at org.apache.rocketmq.remoting.netty.RequestTask.run(RequestTask.java:80)
	at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.14.13-AJDKfp3/Executors.java:515)
	at java.util.concurrent.FutureTask.run(java.base@11.0.14.13-AJDKfp3/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.14.13-AJDKfp3/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.14.13-AJDKfp3/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.14.13-AJDKfp3/Thread.java:991)

Additional Context

No response