prometheus do not work on large kubernetes cluster with error: query timed out in expression evaluation

[garyyang85]

I am using prometheus 1.6.3 on kubernetes which have 800 nodes and more than forty thousand containers. I would like to get the "container_cpu_usage_seconds_total" value under the "graph" tab.
When I select the "container_cpu_usage_seconds_total" and do the "Execute", it return "Error executing query: query timed out in expression evaluation". I can not find any error in the logs.
Something I confirm:

1. this works on a 40 nodes kubernetes env
2. I have configured enough memory with 72G storage.local.target-heap-size and there are no warnings like "Storage has entered rushed mode" now.
   Could help to advise? Thank you very much.
3. I have configured 54 vcpus for the prometheu's vm, the cpu looks a little busy but I do not think it is the root cause.

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.