TiFlash cop thread pool can not handle request with high QPS

[JaySon-Huang]

Enhancement

One of our users execute queries like select count(*) from table where (`url` like 'xxx%') and `uid` in (....) in tidb. If the size of uid is more than several hundred, tidb choose to route that request to TiFlash, with about 15 QPS.

However, TiFlash can not handle those queries quickly, the request is lined up by the coprocessor thread pool. Requests are stacking up while TiDB sees all requests are "timeout" and retry, which makes more requests sent to TiFlash. Finally, it makes TiFlash out of memory.

"cop_dag" means those coprocessor requests are being executing and ...

"cop" means the sum of those coprocessor requests are being executed and those requests are lined up.

[JaySon-Huang]

/cc @LittleFall @windtalker @zanmato1984

[JaySon-Huang]

I think if the pending number of cop requests is more than k times the size of the coprocessor thread pool, then just simple reply something like "TiFlash is busy" to the caller instead of pending by the thread pool. So that TiFlash can recover from large amount of useless retry requests.

[LittleFall]

I think if the pending number of cop requests is more than k times the size of the coprocessor thread pool, then just simple reply something like "TiFlash is busy" to the caller instead of pending by the thread pool. So that TiFlash can recover from large amount of useless retry requests.

this behavior LGTM

[JaySon-Huang]

Are there any plans to implement this behavior? @LittleFall

[JaySon-Huang]

Should also take consideration for Elastic Thread Pool/Dynamic Thread Pool model. /cc @bestwoody @fuzhe1989

[fuzhe1989]

@JaySon-Huang It depends on both TiFlash and TiDB. Do we use exponential backoff retry strategy?

[JaySon-Huang]

Reproduce when running a QA test that only use 8c for TiKV, and the CPU usage of all TiKV is high, making all read index timeout.

[JaySon-Huang]

Another similar problem from asktug: https://asktug.com/t/topic/694336/24
In this case, the number of cop task reach about 30k and make tiflash reach the limit of /proc/sys/vm/max_map_count, no more thread can be created and tiflash crash.

[LittleFall]

closed because #6438 has been basically implemented