VirtualStorage implementation for MSQ in druid 37 causing threadpool leak causing native threads exhausted

[simha03]

We have using druid 37 and we are facing issue with threadpool not closing for MSQ as its not releasing the threads on task exit. we are exhausting the native threads due to this threads leak. once the threads hit the ulimit value it doesnt allow any new threads creation until we restart the indexer service. The issue is more visible in indexer as MSQ runs on same jvm unlike middle manager where each MSQ will fork different JVM.

[root@xxxxx ~]# ls /proc/32589/task | wc -l
254,869
[root@xxxxx ~]#

254,869 ← the leaker, ~99% thread are VirtualStorageOnDemandLoadingThread

Looks the issue was identified and fixed : dedicated ephemeral storage loading pool for tasks- #19658
#19658
I could see the backporting is done only to druid 38. Can we backport to druid 37 as its is a major break on druid stability running MSQ running on indexers ?

[simha03]

@gianm @FrankChen021
#19658 you had made the changes. Can you please check if this can also be backported to druid 37 ?

[FrankChen021]

@simha03 Thanks for reporting. I think for v37, it's not easy to backport #19658 because that PR depends on other changes in v38.

A quick way is to upgrading to v38

[simha03]

@FrankChen021 thanks for response.
When can we expect druid 38 release ?

[FrankChen021]

this work started last month, and 38 release branch has been created, many bugs have already been backported to that branch. It will be soon released I believe, but I don't have a concrete date.