P3 - ApiServer Overload when task number to much

[yqwang-ms]

What would you like to be added:

Why is this needed:

Without this feature, how does the current module work:

Components that may involve changes:

Potential Solutions:

1. Tune ApiServer to serve more QPS
   https://github.com/kubernetes/kubernetes/blob/4fb75e2f0d9a36c47edcf65f89bb92f20274ee56/cluster/gce/gci/configure-helper.sh#L1811-L1822

2. Tune WebPortal, K8S Dashboard, FC and Runtime to send less QPS
   WebPortal control the request concurrency.
   Runtime use downward-api: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api
   Runtime use kubelet local web server: https://applatix.com/making-kubernetes-production-ready-part-3/

3. Cache ApiServer in RestServer

4. RestServer throttling

Find Culprit Tests:
2000 Tasks

• FC: No ApiServer slowdown or problems observed
• RestServer + FC + Runtime: No ApiServer slowdown or problems observed, but met FC issue P2 - Framework object is larger than ETCD can hold (1.5MB) #3706
• Whole PAI (including WebPortal): No ApiServer slowdown or problems observed, but met FC issue P2 - Framework object is larger than ETCD can hold (1.5MB) #3706

[yqwang-ms]

Cannot reproduce 1k task ApiServer unresponse issue, will tune ApiServer in case future overload.

[yqwang-ms]

Change to low priority due to cannot reproduce

[fanyangCS]

Close it. Feel free to reopen when the issue comes up again.