Skip to content

Load: Fix InstanceAlreadyExistsException caused by concurrent registration on JMXService#11755

Merged
SteveYurongSu merged 1 commit intoapache:masterfrom
Caideyipi:jmx-bug
Dec 20, 2023
Merged

Load: Fix InstanceAlreadyExistsException caused by concurrent registration on JMXService#11755
SteveYurongSu merged 1 commit intoapache:masterfrom
Caideyipi:jmx-bug

Conversation

@Caideyipi
Copy link
Copy Markdown
Collaborator

@Caideyipi Caideyipi commented Dec 20, 2023

Description

Previously, the concurrent call to "registerMBean" or "deregisterMBean" may cause after 2 threads seeing the same state simultaneously, that one thread changes the state and make another one throw InstanceAlreadyExistsException or InstanceNotFoundException. This PR fixed this problem by adding synchronize to the 2 functions.

Log example when exception occurs:

2023-11-02 15:52:49,880 [pool-34-IoTDB-ClientRPC-Processor-44$20231102_075249_05132_1] ERROR o.a.i.c.s.JMXService:56 - Failed to registerMBean org.apache.iotdb.threadpool:type=org.apache.iotdb.db.queryengine.plan.scheduler.load.LoadTsFileDispatcherImpl
javax.management.InstanceAlreadyExistsException: org.apache.iotdb.threadpool:type=org.apache.iotdb.db.queryengine.plan.scheduler.load.LoadTsFileDispatcherImpl
        at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.apache.iotdb.commons.service.JMXService.registerMBean(JMXService.java:50)
        at org.apache.iotdb.commons.concurrent.threadpool.WrappedThreadPoolExecutor.<init>(WrappedThreadPoolExecutor.java:72)
        at org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory.newCachedThreadPool(IoTDBThreadPoolFactory.java:149)
        at org.apache.iotdb.db.queryengine.plan.scheduler.load.LoadTsFileDispatcherImpl.<init>(LoadTsFileDispatcherImpl.java:82)
        at org.apache.iotdb.db.queryengine.plan.scheduler.load.LoadTsFileScheduler.<init>(LoadTsFileScheduler.java:127)
        at org.apache.iotdb.db.queryengine.plan.execution.QueryExecution.schedule(QueryExecution.java:321)
        at org.apache.iotdb.db.queryengine.plan.execution.QueryExecution.start(QueryExecution.java:233)
        at org.apache.iotdb.db.queryengine.plan.Coordinator.execute(Coordinator.java:160)
        at org.apache.iotdb.db.pipe.receiver.thrift.IoTDBThriftReceiverV1.executeStatement(IoTDBThriftReceiverV1.java:535)
        at org.apache.iotdb.db.pipe.receiver.thrift.IoTDBThriftReceiverV1.handleTransferFileSeal(IoTDBThriftReceiverV1.java:476)
        at org.apache.iotdb.db.pipe.receiver.thrift.IoTDBThriftReceiverV1.receive(IoTDBThriftReceiverV1.java:140)
        at org.apache.iotdb.db.pipe.receiver.thrift.IoTDBThriftReceiverAgent.receive(IoTDBThriftReceiverAgent.java:43)
        at org.apache.iotdb.db.protocol.thrift.impl.ClientRPCServiceImpl.pipeTransfer(ClientRPCServiceImpl.java:2555)
        at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$pipeTransfer.getResult(IClientRPCService.java:5467)
        at org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$pipeTransfer.getResult(IClientRPCService.java:5447)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

@SteveYurongSu SteveYurongSu self-assigned this Dec 20, 2023
@SteveYurongSu SteveYurongSu changed the title Pipe: Fixed the JMXService to avoid concurrent bug causing InstanceAlreadyExistsException Pipe: Fix InstanceAlreadyExistsException caused by concurrent registration on JMXService Dec 20, 2023
@SteveYurongSu SteveYurongSu changed the title Pipe: Fix InstanceAlreadyExistsException caused by concurrent registration on JMXService Load: Fix InstanceAlreadyExistsException caused by concurrent registration on JMXService Dec 20, 2023
@SteveYurongSu SteveYurongSu merged commit 13ea7e6 into apache:master Dec 20, 2023
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
…ation on JMXService (apache#11755)

Previously, the concurrent call to "registerMBean" or "deregisterMBean" may cause after 2 threads seeing the same state simultaneously, that one thread changes the state and make another one throw InstanceAlreadyExistsException or InstanceNotFoundException. This PR fixed this problem by adding synchronize to the 2 functions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants