Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenTSDB 2.1.0 is stuck - requiring a restart #516

Open
liorsav opened this issue Jun 1, 2015 · 10 comments
Open

OpenTSDB 2.1.0 is stuck - requiring a restart #516

liorsav opened this issue Jun 1, 2015 · 10 comments
Labels
bug

Comments

@liorsav
Copy link

liorsav commented Jun 1, 2015

Hi,
We are using version 2.1.0 (with netty-3.9.4.Final.jar), the TSD daemon is stuck few times a day. Seems like the netty issue described here: #304

Please advise,
Lior

netstat | grep 4242:
tcp 2494 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:37309 CLOSE_WAIT
tcp 2150 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:32819 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:42450 CLOSE_WAIT
tcp 2402 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47955 CLOSE_WAIT
tcp 2134 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44851 CLOSE_WAIT
tcp 2518 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:49431 CLOSE_WAIT
tcp 2173 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:54101 CLOSE_WAIT
tcp 2369 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:38001 CLOSE_WAIT
tcp 464 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56558 CLOSE_WAIT
tcp 532 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:56176 CLOSE_WAIT
tcp 2304 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:60386 CLOSE_WAIT
tcp 2330 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:44911 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:43024 CLOSE_WAIT
tcp 341 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:33464 CLOSE_WAIT
tcp 333 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:34899 CLOSE_WAIT
tcp 2145 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:52186 CLOSE_WAIT
tcp 2186 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:55596 CLOSE_WAIT
tcp 2130 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:39876 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:44268 CLOSE_WAIT
tcp 379 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36235 CLOSE_WAIT
tcp 2133 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:46839 CLOSE_WAIT
tcp 2101 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:33178 CLOSE_WAIT
tcp 2099 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:34925 CLOSE_WAIT
tcp 2541 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:39266 CLOSE_WAIT
tcp 380 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:42558 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:41396 CLOSE_WAIT
tcp 346 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:37886 CLOSE_WAIT
tcp 2264 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53946 CLOSE_WAIT
tcp 2254 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:38948 CLOSE_WAIT
tcp 448 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38695 CLOSE_WAIT
tcp 2189 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:41989 CLOSE_WAIT
tcp 2132 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:53534 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46783 CLOSE_WAIT
tcp 2516 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:42615 CLOSE_WAIT
tcp 2212 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:57742 CLOSE_WAIT
tcp 2238 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:60469 CLOSE_WAIT
tcp 345 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43834 CLOSE_WAIT
tcp 428 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39398 CLOSE_WAIT
tcp 547 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46914 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:43406 CLOSE_WAIT
tcp 2101 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:34072 CLOSE_WAIT
tcp 2523 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:54324 CLOSE_WAIT
tcp 2456 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59161 CLOSE_WAIT
tcp 2185 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:52370 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:37211 CLOSE_WAIT
tcp 2393 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55969 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:32891 CLOSE_WAIT
tcp 352 0 ::ffff:192.168.4.12:4242 ::ffff:172.126.2.115:63628 CLOSE_WAIT
tcp 2167 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46435 CLOSE_WAIT
tcp 2240 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:35305 CLOSE_WAIT
tcp 2146 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:44850 CLOSE_WAIT
tcp 2265 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53531 CLOSE_WAIT
tcp 559 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43535 CLOSE_WAIT
tcp 2144 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:38711 CLOSE_WAIT
tcp 2516 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:33357 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40990 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:34264 CLOSE_WAIT
tcp 2141 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:38218 CLOSE_WAIT
tcp 579 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53048 CLOSE_WAIT
tcp 2226 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:36410 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:35905 CLOSE_WAIT
tcp 2512 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60955 CLOSE_WAIT
tcp 355 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55842 CLOSE_WAIT
tcp 2541 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:41285 ESTABLISHED
tcp 2460 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:37633 CLOSE_WAIT
tcp 580 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:54731 CLOSE_WAIT
tcp 2229 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:50869 CLOSE_WAIT
tcp 2532 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:40528 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:58004 CLOSE_WAIT
tcp 2189 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:40339 CLOSE_WAIT
tcp 412 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60932 CLOSE_WAIT
tcp 424 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:52109 CLOSE_WAIT
tcp 2527 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:39449 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47561 CLOSE_WAIT
tcp 2500 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:53247 CLOSE_WAIT
tcp 2443 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:58863 CLOSE_WAIT
tcp 2356 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43466 CLOSE_WAIT
tcp 2255 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:58885 CLOSE_WAIT
tcp 2141 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:39477 CLOSE_WAIT
tcp 2183 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:52547 CLOSE_WAIT
tcp 2221 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:57206 CLOSE_WAIT
tcp 2301 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:32970 CLOSE_WAIT
tcp 2353 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49279 CLOSE_WAIT
tcp 2294 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:50932 CLOSE_WAIT
tcp 2534 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:51854 CLOSE_WAIT
tcp 2473 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47017 CLOSE_WAIT
tcp 2488 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:40713 CLOSE_WAIT
tcp 574 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48236 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44496 CLOSE_WAIT
tcp 383 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40342 CLOSE_WAIT
tcp 2265 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:56586 CLOSE_WAIT
tcp 0 0 localhost:53990 localhost:4242 ESTABLISHED
tcp 533 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:33205 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53756 CLOSE_WAIT
tcp 365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:58645 CLOSE_WAIT
tcp 366 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:52094 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:34814 CLOSE_WAIT
tcp 2242 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:59422 CLOSE_WAIT
tcp 424 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:51604 CLOSE_WAIT
tcp 2557 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:60970 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46402 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:49251 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:43881 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:39162 CLOSE_WAIT
tcp 2408 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:48838 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53448 CLOSE_WAIT
tcp 2140 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44979 CLOSE_WAIT
tcp 559 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48630 CLOSE_WAIT
tcp 428 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50914 CLOSE_WAIT
tcp 353 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56718 CLOSE_WAIT
tcp 2223 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59930 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:48140 CLOSE_WAIT
tcp 567 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:52076 CLOSE_WAIT
tcp 2201 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:54022 CLOSE_WAIT
tcp 2554 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:39761 CLOSE_WAIT
tcp 2306 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:33473 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:49798 CLOSE_WAIT
tcp 2174 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:55485 CLOSE_WAIT
tcp 582 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:50363 CLOSE_WAIT
tcp 450 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59837 CLOSE_WAIT
tcp 2128 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:61000 CLOSE_WAIT
tcp 531 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:40773 CLOSE_WAIT
tcp 420 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:57695 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49938 CLOSE_WAIT
tcp 2227 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:51804 CLOSE_WAIT
tcp 2224 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:49902 CLOSE_WAIT
tcp 584 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39662 CLOSE_WAIT
tcp 2294 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:57072 CLOSE_WAIT
tcp 2391 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50381 CLOSE_WAIT
tcp 2356 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:33629 CLOSE_WAIT
tcp 432 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:35597 CLOSE_WAIT
tcp 2240 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44064 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60041 CLOSE_WAIT
tcp 379 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59683 CLOSE_WAIT
tcp 2527 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:60280 CLOSE_WAIT
tcp 2102 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:48453 CLOSE_WAIT
tcp 2502 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:35304 CLOSE_WAIT
tcp 2483 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:36403 CLOSE_WAIT
tcp 2128 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:38296 CLOSE_WAIT
tcp 421 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47572 CLOSE_WAIT
tcp 2255 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:58454 CLOSE_WAIT
tcp 557 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43778 CLOSE_WAIT
tcp 533 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:58215 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44230 CLOSE_WAIT
tcp 2368 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34556 CLOSE_WAIT
tcp 2435 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:37411 CLOSE_WAIT
tcp 2468 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:40510 CLOSE_WAIT
tcp 2102 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:43588 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53863 CLOSE_WAIT
tcp 2214 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46226 CLOSE_WAIT
tcp 376 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36391 CLOSE_WAIT
tcp 2200 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:54340 CLOSE_WAIT
tcp 420 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45282 CLOSE_WAIT
tcp 2226 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44853 CLOSE_WAIT
tcp 2523 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:54179 CLOSE_WAIT
tcp 2184 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:58656 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53784 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:53278 CLOSE_WAIT
tcp 2445 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47569 CLOSE_WAIT
tcp 2529 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:34265 CLOSE_WAIT
tcp 2345 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48114 CLOSE_WAIT
tcp 365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53174 CLOSE_WAIT
tcp 2293 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53425 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34338 CLOSE_WAIT
tcp 2184 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:58090 CLOSE_WAIT
tcp 424 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:48308 CLOSE_WAIT
tcp 2481 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:42973 CLOSE_WAIT
tcp 379 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45224 CLOSE_WAIT
tcp 557 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:60248 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53870 CLOSE_WAIT
tcp 2276 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49820 CLOSE_WAIT
tcp 2440 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:47562 CLOSE_WAIT
tcp 2365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:41503 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:59710 CLOSE_WAIT
tcp 2493 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:40534 CLOSE_WAIT
tcp 2163 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:33815 CLOSE_WAIT
tcp 2515 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:53819 CLOSE_WAIT
tcp 2134 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:38820 CLOSE_WAIT
tcp 2200 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49089 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:45741 CLOSE_WAIT
tcp 2146 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:41661 CLOSE_WAIT
tcp 2518 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:59997 CLOSE_WAIT
tcp 2101 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:38393 CLOSE_WAIT
tcp 2138 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59222 CLOSE_WAIT
tcp 2146 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:39468 CLOSE_WAIT
tcp 432 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:43497 CLOSE_WAIT
tcp 2326 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:37038 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:40225 CLOSE_WAIT
tcp 565 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:47313 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:36055 CLOSE_WAIT
tcp 2169 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:60614 CLOSE_WAIT
tcp 2276 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:51833 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53458 CLOSE_WAIT
tcp 2429 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:59499 CLOSE_WAIT
tcp 2097 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:34203 CLOSE_WAIT
tcp 549 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49351 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40555 CLOSE_WAIT
tcp 2539 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:41724 CLOSE_WAIT
tcp 431 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46338 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53751 CLOSE_WAIT
tcp 435 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46786 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:34180 CLOSE_WAIT
tcp 2184 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:35324 CLOSE_WAIT
tcp 431 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:54505 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:52185 CLOSE_WAIT
tcp 2271 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:38638 CLOSE_WAIT
tcp 2535 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:48288 CLOSE_WAIT
tcp 419 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56694 CLOSE_WAIT
tcp 392 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:32980 CLOSE_WAIT
tcp 2505 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:42634 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:52638 CLOSE_WAIT
tcp 2229 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:44379 CLOSE_WAIT
tcp 2174 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:34860 CLOSE_WAIT
tcp 2174 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:45168 CLOSE_WAIT
tcp 344 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:38579 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36142 CLOSE_WAIT
tcp 2203 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:33224 CLOSE_WAIT
tcp 2492 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46569 CLOSE_WAIT
tcp 444 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36337 CLOSE_WAIT
tcp 2128 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:40546 CLOSE_WAIT
tcp 591 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:40401 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36275 CLOSE_WAIT
tcp 2438 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46605 CLOSE_WAIT
tcp 2150 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46009 CLOSE_WAIT
tcp 2105 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:41825 CLOSE_WAIT
tcp 2362 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:38669 CLOSE_WAIT
tcp 2535 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:36657 CLOSE_WAIT
tcp 2102 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:51792 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45659 CLOSE_WAIT
tcp 2172 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:39913 CLOSE_WAIT
tcp 546 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:56699 CLOSE_WAIT
tcp 2145 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:34823 CLOSE_WAIT
tcp 2105 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:43759 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59786 CLOSE_WAIT
tcp 2169 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:39803 CLOSE_WAIT
tcp 2428 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34605 CLOSE_WAIT
tcp 376 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:44338 CLOSE_WAIT
tcp 592 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:33551 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:55375 CLOSE_WAIT
tcp 2130 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46791 CLOSE_WAIT
tcp 2521 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:59564 CLOSE_WAIT
tcp 2524 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:51922 CLOSE_WAIT
tcp 366 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56735 CLOSE_WAIT
tcp 530 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43530 CLOSE_WAIT
tcp 78 0 localhost:4242 localhost:53990 ESTABLISHED
tcp 2426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:49089 CLOSE_WAIT
tcp 2358 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:57637 CLOSE_WAIT
tcp 2185 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:44893 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:49815 CLOSE_WAIT
tcp 423 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55259 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:57315 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:33713 CLOSE_WAIT
tcp 2102 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:53494 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:34684 CLOSE_WAIT
tcp 549 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:37956 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:47236 CLOSE_WAIT
tcp 440 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36808 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39551 CLOSE_WAIT
tcp 344 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:47275 CLOSE_WAIT
tcp 429 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60575 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53908 CLOSE_WAIT
tcp 2222 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:53079 CLOSE_WAIT
tcp 2169 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:52660 CLOSE_WAIT
tcp 2460 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56796 CLOSE_WAIT
tcp 2362 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:54476 CLOSE_WAIT
tcp 2504 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:55314 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:48529 CLOSE_WAIT
tcp 2160 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44566 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:52459 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:48235 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53035 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36884 CLOSE_WAIT
tcp 2175 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:37556 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:42115 CLOSE_WAIT
tcp 2273 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43668 CLOSE_WAIT
tcp 365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:49282 CLOSE_WAIT
tcp 2100 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:56030 CLOSE_WAIT
tcp 420 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:33379 CLOSE_WAIT
tcp 2265 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43319 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53481 CLOSE_WAIT
tcp 2294 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:58265 CLOSE_WAIT
tcp 2222 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:57002 CLOSE_WAIT
tcp 2306 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34161 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:58469 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50553 CLOSE_WAIT
tcp 2417 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:56452 CLOSE_WAIT
tcp 2174 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59147 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46662 CLOSE_WAIT
tcp 375 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:41780 CLOSE_WAIT
tcp 2455 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38845 CLOSE_WAIT
tcp 380 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:58324 CLOSE_WAIT
tcp 2160 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:49619 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53357 CLOSE_WAIT
tcp 2213 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:57314 CLOSE_WAIT
tcp 2528 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:42688 CLOSE_WAIT
tcp 2533 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:46796 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:47284 CLOSE_WAIT
tcp 2431 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:38791 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53757 CLOSE_WAIT
tcp 2477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53656 CLOSE_WAIT
tcp 2507 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:41605 CLOSE_WAIT
tcp 422 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47338 CLOSE_WAIT
tcp 2133 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:37926 CLOSE_WAIT
tcp 2330 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:56841 CLOSE_WAIT
tcp 2187 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:52950 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:57532 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53508 CLOSE_WAIT
tcp 2224 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:40343 CLOSE_WAIT
tcp 2428 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:37222 CLOSE_WAIT
tcp 2333 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53480 CLOSE_WAIT
tcp 2488 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:44001 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:56604 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:35620 CLOSE_WAIT
tcp 2253 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39086 CLOSE_WAIT
tcp 2462 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:57347 CLOSE_WAIT
tcp 2524 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:55150 CLOSE_WAIT
tcp 2486 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53781 ESTABLISHED
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:41810 CLOSE_WAIT
tcp 2229 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:57533 CLOSE_WAIT
tcp 452 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39329 CLOSE_WAIT
tcp 464 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:52723 CLOSE_WAIT
tcp 353 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40703 CLOSE_WAIT
tcp 2404 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50452 CLOSE_WAIT
tcp 2214 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:60394 CLOSE_WAIT
tcp 418 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59185 CLOSE_WAIT
tcp 2186 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:48444 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:47781 CLOSE_WAIT
tcp 408 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:43981 CLOSE_WAIT
tcp 2145 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:51194 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:44221 CLOSE_WAIT
tcp 2132 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44767 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34712 CLOSE_WAIT
tcp 2478 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:48706 CLOSE_WAIT
tcp 2421 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36041 CLOSE_WAIT
tcp 366 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:35496 CLOSE_WAIT
tcp 2129 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:41729 CLOSE_WAIT
tcp 2173 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:51336 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:46740 CLOSE_WAIT
tcp 2145 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:37765 CLOSE_WAIT
tcp 2101 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:35501 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53509 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50037 CLOSE_WAIT
tcp 2531 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:43219 CLOSE_WAIT
tcp 2228 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59533 CLOSE_WAIT
tcp 334 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:49655 CLOSE_WAIT
tcp 422 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60472 CLOSE_WAIT
tcp 2319 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39903 CLOSE_WAIT
tcp 2532 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:48173 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39853 CLOSE_WAIT
tcp 2185 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:58508 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53449 CLOSE_WAIT
tcp 2212 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:56292 CLOSE_WAIT
tcp 2390 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:41955 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55611 CLOSE_WAIT
tcp 2169 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:39030 CLOSE_WAIT
tcp 2394 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:51372 CLOSE_WAIT
tcp 2463 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53543 CLOSE_WAIT
tcp 397 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60763 CLOSE_WAIT
tcp 555 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:37624 CLOSE_WAIT
tcp 2536 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:33100 CLOSE_WAIT
tcp 2509 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:34356 CLOSE_WAIT
tcp 353 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40255 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55210 CLOSE_WAIT
tcp 2289 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:36586 CLOSE_WAIT
tcp 556 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:58753 ESTABLISHED
tcp 2502 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:58119 CLOSE_WAIT
tcp 2185 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:36755 CLOSE_WAIT
tcp 2228 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:36173 CLOSE_WAIT
tcp 2132 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:35348 CLOSE_WAIT
tcp 573 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46000 CLOSE_WAIT
tcp 552 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:34161 CLOSE_WAIT
tcp 579 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48056 CLOSE_WAIT
tcp 2132 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:55716 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44821 CLOSE_WAIT
tcp 2203 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:49722 CLOSE_WAIT
tcp 421 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59733 CLOSE_WAIT
tcp 2160 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:54555 CLOSE_WAIT
tcp 387 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:50729 ESTABLISHED
tcp 1 0 localhost:4242 localhost:53674 CLOSE_WAIT
tcp 2523 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:35964 CLOSE_WAIT
tcp 2330 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:57308 CLOSE_WAIT
tcp 424 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:42748 CLOSE_WAIT
tcp 2392 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:48553 CLOSE_WAIT
tcp 2212 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:52546 CLOSE_WAIT
tcp 2202 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:37777 CLOSE_WAIT
tcp 2144 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:48759 CLOSE_WAIT
tcp 2256 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:35509 CLOSE_WAIT
tcp 353 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59833 CLOSE_WAIT
tcp 2202 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:50837 CLOSE_WAIT
tcp 365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36909 CLOSE_WAIT
tcp 2395 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38444 CLOSE_WAIT
tcp 2186 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:47035 CLOSE_WAIT
tcp 2443 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45309 CLOSE_WAIT
tcp 2520 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:60536 CLOSE_WAIT
tcp 2565 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:38337 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53576 CLOSE_WAIT
tcp 2358 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46337 CLOSE_WAIT
tcp 564 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:54783 CLOSE_WAIT
tcp 375 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55013 CLOSE_WAIT
tcp 2195 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:35841 CLOSE_WAIT
tcp 2541 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:50783 CLOSE_WAIT
tcp 2509 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:35961 CLOSE_WAIT
tcp 2253 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46622 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38651 CLOSE_WAIT
tcp 2311 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:54296 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:43942 CLOSE_WAIT
tcp 424 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39799 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:55810 CLOSE_WAIT
tcp 556 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53384 CLOSE_WAIT
tcp 2129 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:41965 CLOSE_WAIT
tcp 2319 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39833 CLOSE_WAIT
tcp 2518 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:51469 CLOSE_WAIT
tcp 2175 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:54337 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:35654 CLOSE_WAIT
tcp 2293 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:52413 CLOSE_WAIT
tcp 420 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38354 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:54760 CLOSE_WAIT
tcp 2100 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:34138 CLOSE_WAIT
tcp 355 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:58208 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:59139 CLOSE_WAIT
tcp 2130 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:45553 CLOSE_WAIT
tcp 2226 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:37290 CLOSE_WAIT
tcp 2144 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:43721 CLOSE_WAIT
tcp 451 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36345 CLOSE_WAIT
tcp 2527 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:38720 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:57380 CLOSE_WAIT
tcp 428 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59493 CLOSE_WAIT
tcp 2292 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:50104 CLOSE_WAIT
tcp 464 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53191 CLOSE_WAIT
tcp 2242 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:42543 CLOSE_WAIT
tcp 2211 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59252 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:44384 CLOSE_WAIT
tcp 2132 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:42071 CLOSE_WAIT
tcp 2303 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:57396 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:35255 CLOSE_WAIT
tcp 567 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:55052 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:33837 CLOSE_WAIT
tcp 2478 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:44159 CLOSE_WAIT
tcp 2389 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:35306 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53364 CLOSE_WAIT
tcp 2185 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:56530 CLOSE_WAIT
tcp 2171 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:51803 CLOSE_WAIT
tcp 2540 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60239 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53675 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53482 CLOSE_WAIT
tcp 2143 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:42232 CLOSE_WAIT
tcp 2496 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:57442 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:44419 CLOSE_WAIT
tcp 2363 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:60070 CLOSE_WAIT
tcp 2303 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:45670 CLOSE_WAIT
tcp 2538 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:52856 CLOSE_WAIT
tcp 355 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45014 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:35903 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:53939 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:34294 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40609 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53989 CLOSE_WAIT
tcp 2173 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:35172 CLOSE_WAIT
tcp 554 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:53339 CLOSE_WAIT
tcp 2104 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:46292 CLOSE_WAIT
tcp 1 0 localhost:4242 localhost:53907 CLOSE_WAIT
tcp 557 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48709 CLOSE_WAIT
tcp 417 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:41651 ESTABLISHED
tcp 2541 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:44563 CLOSE_WAIT
tcp 459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40035 CLOSE_WAIT
tcp 2549 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:59517 CLOSE_WAIT
tcp 2394 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:37177 CLOSE_WAIT
tcp 560 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:34233 CLOSE_WAIT
tcp 2144 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:40783 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53577 CLOSE_WAIT
tcp 2227 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:33450 CLOSE_WAIT
tcp 2505 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:35619 CLOSE_WAIT
tcp 449 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:38327 CLOSE_WAIT
tcp 2289 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:60823 CLOSE_WAIT
tcp 2100 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:59948 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53459 CLOSE_WAIT
tcp 2530 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:47795 CLOSE_WAIT
tcp 2131 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:45957 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55712 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53752 CLOSE_WAIT
tcp 2298 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45589 CLOSE_WAIT
tcp 2409 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:41404 CLOSE_WAIT
tcp 379 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:41411 CLOSE_WAIT
tcp 533 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:42920 CLOSE_WAIT
tcp 2146 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:58367 CLOSE_WAIT
tcp 2523 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:41505 CLOSE_WAIT
tcp 350 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:35882 CLOSE_WAIT
tcp 2483 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:60533 CLOSE_WAIT
tcp 391 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:35853 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:32812 CLOSE_WAIT
tcp 2562 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:60787 CLOSE_WAIT
tcp 2146 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:55639 CLOSE_WAIT
tcp 535 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:55366 CLOSE_WAIT
tcp 580 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:51699 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53540 CLOSE_WAIT
tcp 2130 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:55106 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53785 CLOSE_WAIT
tcp 378 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:36804 CLOSE_WAIT
tcp 2498 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:57938 CLOSE_WAIT
tcp 426 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:32782 CLOSE_WAIT
tcp 366 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:48268 CLOSE_WAIT
tcp 2141 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:60470 CLOSE_WAIT
tcp 553 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:39289 CLOSE_WAIT
tcp 2173 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:47313 CLOSE_WAIT
tcp 552 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:58872 CLOSE_WAIT
tcp 2459 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53835 CLOSE_WAIT
tcp 558 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:48293 CLOSE_WAIT
tcp 2501 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:45111 CLOSE_WAIT
tcp 2407 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40836 CLOSE_WAIT
tcp 2318 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:46236 CLOSE_WAIT
tcp 420 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:59783 CLOSE_WAIT
tcp 2216 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:59408 CLOSE_WAIT
tcp 2501 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39477 CLOSE_WAIT
tcp 2130 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:46844 CLOSE_WAIT
tcp 553 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:51356 CLOSE_WAIT
tcp 422 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:53645 CLOSE_WAIT
tcp 2215 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:48349 CLOSE_WAIT
tcp 2474 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:55704 CLOSE_WAIT
tcp 388 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:54001 CLOSE_WAIT
tcp 2546 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:38744 CLOSE_WAIT
tcp 2159 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:36101 CLOSE_WAIT
tcp 2142 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:36545 CLOSE_WAIT
tcp 416 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:54771 CLOSE_WAIT
tcp 2159 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:49396 CLOSE_WAIT
tcp 2103 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:58199 ESTABLISHED
tcp 581 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:45672 CLOSE_WAIT
tcp 2128 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:50924 CLOSE_WAIT
tcp 2520 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:32935 CLOSE_WAIT
tcp 366 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60246 CLOSE_WAIT
tcp 2530 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:40322 CLOSE_WAIT
tcp 2298 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:43959 CLOSE_WAIT
tcp 535 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.88:34176 CLOSE_WAIT
tcp 247 0 localhost:4242 localhost:53864 CLOSE_WAIT
tcp 365 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:60973 CLOSE_WAIT
tcp 2187 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.86:37809 CLOSE_WAIT
tcp 477 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:40052 CLOSE_WAIT
tcp 2203 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.64:48432 CLOSE_WAIT
tcp 427 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.62:39737 CLOSE_WAIT
tcp 2173 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.87:51894 CLOSE_WAIT
tcp 2520 0 ::ffff:172.20.1.58:4242 ::ffff:172.20.1.63:45110 CLOSE_WAIT

jstack sample:
"New I/O worker #38" prio=10 tid=0x00007fec5499c000 nid=0x423f in Object.wait() [0x00007fec1ce4c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007a92e81a0> (a com.stumbleupon.async.Deferred$Signal)
at java.lang.Object.wait(Object.java:503)
at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1138)
- locked <0x00000007a92e81a0> (a com.stumbleupon.async.Deferred$Signal)
at com.stumbleupon.async.Deferred.joinUninterruptibly(Deferred.java:1064)
at net.opentsdb.tsd.QueryRpc.handleQuery(QueryRpc.java:162)
at net.opentsdb.tsd.QueryRpc.execute(QueryRpc.java:84)
at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273)
at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at net.opentsdb.tsd.ConnectionManager.handleUpstream(ConnectionManager.java:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
@manolama
Copy link
Member

Hm, are you gracefully closing your connections to the TSD after completing your query? All of those half-open sockets are indicative of that. And were those queries or writes to the TSD? There is code to handle the idle connections but I forgot if it made it into 2.1 or is in 2.2. I'll look this weekend.

@manolama manolama added the bug label Aug 28, 2015
@jroyalty
Copy link
Contributor

jroyalty commented Sep 3, 2015

@manolama Anecdotally, I've observed this happening on TSDs that do only writes. And writing via the Telnet API at that.

@mxk1235
Copy link
Contributor

mxk1235 commented Sep 3, 2015

this sounds like when telnet client doesn't read error messages from OpenTsdb, which get queued up on the tsd side. are you using any of the open-source clients, or did you write your own?

@jroyalty
Copy link
Contributor

jroyalty commented Sep 3, 2015

@mxk1235 It's a custom thing in Python+Twisted. It does read the response though.

@manolama
Copy link
Member

manolama commented Sep 3, 2015

@jroyalty It could be that a bad region server creates a flood of messages that backs up the write buffer on the socket. Netty helpfully stores all that data for you. 2.2 already has a patch that will block writes to the telnet socket while it's full to prevent heap bloating.

@rclaire
Copy link

rclaire commented Sep 4, 2015

@manolama We've been seeing a similar problem with openTSDB 2.1.0 (netty-3.9.4.Final), where openTSDB becomes unresponsive if it experiences read or write interruption from a Region Server.

We divided our openTSDB servers into read and write roles, and it is the read instances where we see this hanging behavior

with LSOF, we see a massive jump in in file descriptions, with most of them being 'can't identify protocol' for SOCK connections, as others pointing out, we then hit the ulimit -n threshold. openTSDB is then hung until the process is restarted

Example distribution of open files we see when from a hung openTSDB process:

39528 sock
2323 IPv4
  60 FIFO
  52 REG
  30 0000
   3 CHR
   2 unix
   2 DIR

jstack thread dumps that we take when openTSDB is hung show about 50 % of the threads being blocked. Some examples of blocked threads are:

Thread 26611: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
 - com.stumbleupon.async.Deferred.doJoin(boolean, long) @bci=77, line=1138 (Compiled frame)
 - com.stumbleupon.async.Deferred.joinUninterruptibly() @bci=3, line=1064 (Compiled frame)
 - net.opentsdb.tsd.QueryRpc.handleQuery(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=302, line=162 (Compiled frame)
 - net.opentsdb.tsd.QueryRpc.execute(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=109, line=84 (Interpreted frame)
 - net.opentsdb.tsd.RpcHandler.handleHttpQuery(net.opentsdb.core.TSDB, org.jboss.netty.channel.Channel, org.jboss.netty.handler.codec.http.HttpRequest) @bci=354, line=273 (Interpreted frame)
 - net.opentsdb.tsd.RpcHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=56, line=180 (Interpreted frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=36 (Interpreted frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=19, line=294 (Interpreted frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=85, line=82 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=95, line=435 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=123, line=462 (Interpreted frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=97, line=443 (Interpreted frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=62, line=303 (Interpreted frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=2, line=142 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - net.opentsdb.tsd.ConnectionManager.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=87 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=55, line=559 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=372, line=318 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 26607: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
 - com.stumbleupon.async.Deferred.doJoin(boolean, long) @bci=77, line=1138 (Compiled frame)
 - com.stumbleupon.async.Deferred.joinUninterruptibly() @bci=3, line=1064 (Compiled frame)
 - net.opentsdb.tsd.QueryRpc.handleQuery(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=302, line=162 (Compiled frame)
 - net.opentsdb.tsd.QueryRpc.execute(net.opentsdb.core.TSDB, net.opentsdb.tsd.HttpQuery) @bci=109, line=84 (Compiled frame)
 - net.opentsdb.tsd.RpcHandler.handleHttpQuery(net.opentsdb.core.TSDB, org.jboss.netty.channel.Channel, org.jboss.netty.handler.codec.http.HttpRequest) @bci=354, line=273 (Compiled frame)
 - net.opentsdb.tsd.RpcHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=56, line=180 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=36 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=19, line=294 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=85, line=82 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=285, line=108 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=99, line=145 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Compiled frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=95, line=435 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=2, line=142 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - net.opentsdb.tsd.ConnectionManager.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=87 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=55, line=559 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=372, line=318 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
: @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=85, line=82 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=285, line=108 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=99, line=145 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Compiled frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=95, line=435 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=2, line=142 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - net.opentsdb.tsd.ConnectionManager.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=87 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=55, line=559 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=372, line=318 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
: @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=85, line=82 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=285, line=108 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=99, line=145 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Compiled frame)
 - org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.jboss.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.Channel, org.jboss.netty.buffer.ChannelBuffer, org.jboss.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Compiled frame)
 - org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=95, line=435 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.messageReceived(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.MessageEvent) @bci=2, line=142 (Compiled frame)
 - org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=13, line=88 (Compiled frame)
 - net.opentsdb.tsd.ConnectionManager.handleUpstream(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=22, line=87 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.jboss.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(org.jboss.netty.channel.ChannelEvent) @bci=55, line=559 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Compiled frame)
 - org.jboss.netty.channel.Channels.fireMessageReceived(org.jboss.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=372, line=318 (Compiled frame)
 - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

@manolama
Copy link
Member

manolama commented Sep 4, 2015

Thanks @rclaire! Could you also capture stdout for your readers please? We're seeing issues popping up around stack overflows that can cause deadlocks like these but they aren't logged normally since such calls can cause additional stack growth.

@rclaire
Copy link

rclaire commented Sep 4, 2015

Thanks @manolama This is what we see in opentsdb.log during the onset of the problem.

19:55:02.523 INFO  [ConnectionManager.handleUpstream] - [id: 0xb7226d29, /10.1.46.24:49230 => /10.1.224.239:4242] OPEN
19:55:05.752 INFO  [ConnectionManager.handleUpstream] - [id: 0xd2fb04be, /10.1.239.63:63740 => /10.1.224.239:4242] OPEN
19:55:05.939 INFO  [ConnectionManager.handleUpstream] - [id: 0xf25b9e71, /10.1.100.176:51478 => /10.1.224.239:4242] OPEN
19:55:08.522 INFO  [ConnectionManager.handleUpstream] - [id: 0xef4c38af, /10.1.46.24:49252 => /10.1.224.239:4242] OPEN
19:55:11.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x2afcb116, /10.1.239.63:63763 => /10.1.224.239:4242] OPEN
19:55:11.939 INFO  [ConnectionManager.handleUpstream] - [id: 0x83eee26b, /10.1.100.176:51500 => /10.1.224.239:4242] OPEN
19:55:14.522 INFO  [ConnectionManager.handleUpstream] - [id: 0x0a2387e9, /10.1.46.24:49275 => /10.1.224.239:4242] OPEN
19:55:17.753 INFO  [ConnectionManager.handleUpstream] - [id: 0x611edc2b, /10.1.239.63:63785 => /10.1.224.239:4242] OPEN
19:55:17.940 INFO  [ConnectionManager.handleUpstream] - [id: 0xc260fea4, /10.1.100.176:51522 => /10.1.224.239:4242] OPEN
19:55:20.522 INFO  [ConnectionManager.handleUpstream] - [id: 0x22e3d35d, /10.1.46.24:49297 => /10.1.224.239:4242] OPEN
19:55:23.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x1d9ea361, /10.1.239.63:63807 => /10.1.224.239:4242] OPEN
19:55:23.939 INFO  [ConnectionManager.handleUpstream] - [id: 0x4700ad8c, /10.1.100.176:51546 => /10.1.224.239:4242] OPEN
19:55:26.522 INFO  [ConnectionManager.handleUpstream] - [id: 0x2d227a21, /10.1.46.24:49321 => /10.1.224.239:4242] OPEN
19:55:29.752 INFO  [ConnectionManager.handleUpstream] - [id: 0xd0b0e2ca, /10.1.239.63:63831 => /10.1.224.239:4242] OPEN
19:55:29.939 INFO  [ConnectionManager.handleUpstream] - [id: 0x8d2961a6, /10.1.100.176:51571 => /10.1.224.239:4242] OPEN
19:55:31.013 WARN  [Scanner.call] - RegionInfo(table="tsdb", region_name="tsdb,\x90\xC3eUX\xC9\x10\x00\x00\x04Ku\xA8\x00\x00\x0CK\x1A\xFA,1432959378150.a032227ad1500eec1d0ed108c52bc31c.", stop_key=[-112, -50, 88, 85, 69, -85, 80, 0, 0, 4, 78, -43, -17, 0, 0, 5, 0, 1, -91, 0, 0, 11, 0, 1, -79, 0, 0, 12, 78, -43, -15, 0, 0, 16, 0, 1, 97]) pretends to not know Scanner(table="tsdb", start_key=[-112, -60, -26, 85, -31, 18, 32, 0, 0, 3, 0, 0, 14, 0, 0, 4, 79, -118, 40, 0, 0, 12, 78, -105, 10, 0, 0, -122, 75, 27, -73], stop_key=[-112, -60, -26, 85, -22, 3, -41], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x000A0680974E0453).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)
19:55:31.013 WARN  [Scanner.call] - RegionInfo(table="tsdb", region_name="tsdb,\x90\xC3eUX\xC9\x10\x00\x00\x04Ku\xA8\x00\x00\x0CK\x1A\xFA,1432959378150.a032227ad1500eec1d0ed108c52bc31c.", stop_key=[-112, -50, 88, 85, 69, -85, 80, 0, 0, 4, 78, -43, -17, 0, 0, 5, 0, 1, -91, 0, 0, 11, 0, 1, -79, 0, 0, 12, 78, -43, -15, 0, 0, 16, 0, 1, 97]) pretends to not know Scanner(table="tsdb", start_key=[-112, -60, -26, 85, -31, 60, 80, 0, 0, 3, 0, 0, 14, 0, 0, 4, 79, -118, 38, 0, 0, 12, 78, -105, 10, 0, 0, -122, 75, 18, -124], stop_key=[-112, -60, -26, 85, -22, 3, -56], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x00C73EFA96395D27).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)
19:55:31.015 WARN  [Scanner.call] - RegionInfo(table="tsdb", region_name="tsdb,\x90\xC3eUX\xC9\x10\x00\x00\x04Ku\xA8\x00\x00\x0CK\x1A\xFA,1432959378150.a032227ad1500eec1d0ed108c52bc31c.", stop_key=[-112, -50, 88, 85, 69, -85, 80, 0, 0, 4, 78, -43, -17, 0, 0, 5, 0, 1, -91, 0, 0, 11, 0, 1, -79, 0, 0, 12, 78, -43, -15, 0, 0, 16, 0, 1, 97]) pretends to not know Scanner(table="tsdb", start_key=[-112, -60, -26, 85, -30, 71, -128, 0, 0, 3, 0, 0, 14, 0, 0, 4, 79, 85, 95, 0, 0, 12, 78, -105, 10, 0, 0, -122, 75, 18, -83], stop_key=[-112, -60, -26, 85, -22, 3, -112], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x074CDBB8CF9D4B66).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)
19:55:31.016 WARN  [Scanner.call] - RegionInfo(table="tsdb", region_name="tsdb,\x90\xC3eUX\xC9\x10\x00\x00\x04Ku\xA8\x00\x00\x0CK\x1A\xFA,1432959378150.a032227ad1500eec1d0ed108c52bc31c.", stop_key=[-112, -50, 88, 85, 69, -85, 80, 0, 0, 4, 78, -43, -17, 0, 0, 5, 0, 1, -91, 0, 0, 11, 0, 1, -79, 0, 0, 12, 78, -43, -15, 0, 0, 16, 0, 1, 97]) pretends to not know Scanner(table="tsdb", start_key=[-112, -60, -26, 85, -32, -67, -64, 0, 0, 3, 0, 0, 14, 0, 0, 4, 79, 85, 95, 0, 0, 12, 78, -105, 10, 0, 0, -122, 75, 18, 126], stop_key=[-112, -60, -26, 85, -22, 4, 4], columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=null, scanner_id=0x01F2959F20940678).  I will retry to open a scanner but this is typically because you've been holding the scanner open and idle for too long (possibly due to a long GC pause on your side or in the RegionServer)
19:55:32.522 INFO  [ConnectionManager.handleUpstream] - [id: 0x88d6e756, /10.1.46.24:49343 => /10.1.224.239:4242] OPEN
19:55:35.753 INFO  [ConnectionManager.handleUpstream] - [id: 0x5776f628, /10.1.239.63:63853 => /10.1.224.239:4242] OPEN
19:55:35.939 INFO  [ConnectionManager.handleUpstream] - [id: 0xf981ba4e, /10.1.100.176:51593 => /10.1.224.239:4242] OPEN
19:55:38.523 INFO  [ConnectionManager.handleUpstream] - [id: 0x0660b5b5, /10.1.46.24:49365 => /10.1.224.239:4242] OPEN
19:55:41.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x2503a5ac, /10.1.239.63:63875 => /10.1.224.239:4242] OPEN
19:55:41.939 INFO  [ConnectionManager.handleUpstream] - [id: 0x6cd78336, /10.1.100.176:51617 => /10.1.224.239:4242] OPEN
19:55:44.522 INFO  [ConnectionManager.handleUpstream] - [id: 0x03e2bbea, /10.1.46.24:49391 => /10.1.224.239:4242] OPEN
19:55:47.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x5f58c06f, /10.1.239.63:63900 => /10.1.224.239:4242] OPEN
19:55:47.939 INFO  [ConnectionManager.handleUpstream] - [id: 0x06e0b9f2, /10.1.100.176:51640 => /10.1.224.239:4242] OPEN

When openTSDB is hung, we don't see anything interesting in the logs

19:59:11.940 INFO  [ConnectionManager.handleUpstream] - [id: 0x86d1418c, /10.1.100.176:52447 => /10.1.224.239:4242] OPEN
19:59:14.523 INFO  [ConnectionManager.handleUpstream] - [id: 0xd0763505, /10.1.46.24:50223 => /10.1.224.239:4242] OPEN
19:59:17.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x26550a1f, /10.1.239.63:64733 => /10.1.224.239:4242] OPEN
19:59:17.939 INFO  [ConnectionManager.handleUpstream] - [id: 0xa96d96ad, /10.1.100.176:52469 => /10.1.224.239:4242] OPEN
19:59:20.523 INFO  [ConnectionManager.handleUpstream] - [id: 0xb0e52033, /10.1.46.24:50245 => /10.1.224.239:4242] OPEN
19:59:23.752 INFO  [ConnectionManager.handleUpstream] - [id: 0x2058401f, /10.1.239.63:64755 => /10.1.224.239:4242] OPEN
19:59:23.940 INFO  [ConnectionManager.handleUpstream] - [id: 0x2788c409, /10.1.100.176:52493 => /10.1.224.239:4242] OPEN
19:59:26.523 INFO  [ConnectionManager.handleUpstream] - [id: 0x21aac410, /10.1.46.24:50269 => /10.1.224.239:4242] OPEN
19:59:29.753 INFO  [ConnectionManager.handleUpstream] - [id: 0x2e21c04e, /10.1.239.63:64779 => /10.1.224.239:4242] OPEN

@utkarshcmu
Copy link
Contributor

@manolama I am facing the same problem in v2.1.1. Also, when I stop the service opentsdb, I still have open connections left. This is weird and this happens when the Hbase cluster is too big I guess. We were not having problems when the cluster was less than 50/60 nodes. Now, when the cluster is 80 nodes, this problem is very frequent that we have to restart opentsdb to reset connections.

@pinifaran
Copy link

From what we've experienced, it happens for read (queries) operations and when enable_compaction is set to true. When compaction is set to false, the problem does not happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rclaire @jroyalty @mxk1235 @utkarshcmu @manolama @pinifaran @liorsav