Fix: deadlock while closing non-persistent topic

[rdhabalia]

Motivation

Deadlock while closing non-persistent topic.

• Unload topic iterates over topics map and removing from the map with the same thread can create deadlock.

"pulsar-web-72-23" #251 prio=5 os_prio=0 tid=0x00002aba90001000 nid=0x2428 waiting on condition [0x00002ab907b1a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000568e0ba18> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
        at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
        at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:354)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.remove(ConcurrentOpenHashMap.java:305)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.access$200(ConcurrentOpenHashMap.java:181)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.remove(ConcurrentOpenHashMap.java:136)
        at org.apache.pulsar.broker.service.BrokerService.removeTopicFromCache(BrokerService.java:931)
        at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.lambda$10(NonPersistentTopic.java:450)
        at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic$$Lambda$493/445153470.run(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705)
        at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:717)
        at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2010)
        at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.close(NonPersistentTopic.java:448)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$275/707822372.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
        at org.apache.pulsar.broker.service.BrokerService.lambda$21(BrokerService.java:885)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$274/1896064809.accept(Unknown Source)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:386)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:160)
        at org.apache.pulsar.broker.service.BrokerService.unloadServiceUnit(BrokerService.java:880)
        at org.apache.pulsar.broker.namespace.OwnedBundle.handleUnloadRequest(OwnedBundle.java:125)
        at org.apache.pulsar.broker.namespace.NamespaceService.unloadNamespaceBundle(NamespaceService.java:482)
        at org.apache.pulsar.broker.namespace.NamespaceService.unloadNamespaceBundle(NamespaceService.java:478)
        at org.apache.pulsar.broker.admin.Namespaces.unloadNamespaceBundle(Namespaces.java:842)
        at sun.reflect.GeneratedMethodAccessor180.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Modifications

Remove topic from topicCacheMap in a different thread.

Result

It will avoid possible deadlock for non-persistent topic.

[merlimat]

👍

[sijie]

retest this please