Skip to content

Commit

Permalink
Fix thread-blocked problem of EtcdDataSource (#2991)
Browse files Browse the repository at this point in the history
* fix thread blocked problem of etcd watcher

in the previous version of EtcdDataSource, when the flowRules defined in the etcd change and trigger etcd watcher, there will be a following error:
2022-12-07 14:36:03.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2336 ms, time limit is 2000 ms
2022-12-07 14:36:04.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3337 ms, time limit is 2000 ms
2022-12-07 14:36:05.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4337 ms, time limit is 2000 ms
2022-12-07 14:36:06.595 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5337 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
 at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3334)
 at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:92)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:23)
 at com.alibaba.csp.sentinel.datasource.AbstractDataSource.loadConfig(AbstractDataSource.java:44)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.lambda$initWatcher$0(EtcdDataSource.java:74)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource$$Lambda$480/1375681611.accept(Unknown Source)
 at io.etcd.jetcd.Watch$1.onNext(Watch.java:183)
 at io.etcd.jetcd.impl.WatchImpl$WatcherImpl.onNext(WatchImpl.java:310)
 at io.etcd.jetcd.impl.WatchImpl$WatcherImpl$$Lambda$488/2088661957.handle(Unknown Source

I guess the reason is : when the watcher hold etcd Client, you can't use etcd Client in the Watch's consumer function. 
Besides the problem above, use getKeyValue() in the watchEvent can reduce once invoke with etcd server to getting the latest flowRules :)

* Update EtcdDataSource.java

Add charset adaptation -- accept the suggestion from @LProDreamAll
  • Loading branch information
Yan1025 committed Dec 23, 2022
1 parent 141d54f commit a7cd8a1
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ private void initWatcher() {
WatchEvent.EventType eventType = watchEvent.getEventType();
if (eventType == WatchEvent.EventType.PUT) {
try {
T newValue = loadConfig();
String newValueJson = watchEvent.getKeyValue().getValue().toString(charset);
T newValue = parser.convert(newValueJson);
getProperty().updateValue(newValue);
} catch (Exception e) {
RecordLog.warn("[EtcdDataSource] Failed to update config", e);
Expand Down

0 comments on commit a7cd8a1

Please sign in to comment.