increasing node and pod worker number

[haouc]

Issue #, if available:
#295
Description of changes:
We are seeing pod queue latency in some extreme use case. Increasing the workers to help on performance.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[sushrk]

General comment for my understanding, how did we choose these values?

[haouc]

We are still in the process to tune the values. I don't think there is any standards or guidance from upstream on those values. Since they are pretty lightweight and increasing to 20 for pods and 10 for nodes are not large number, I think it is worth to make the change and watch how it works.
In small/medium scale load tests, I didn't see significant improvement by increasing the goroutines. But using 3 for pods and 4 for nodes are likely constraining queue performance during large scaling. Also considering the queueing frequency, pods need more goroutines than nodes to work on queues.

[haouc]

After offline sync with @sushrk , we agree to increase the concurrency on node and pod controller by stages to minimize risks. If this change works stably, we will increase them to 10 for node and 20 for pod controller in the next release.

[sushrk]

LGTM, thanks Hao.
We will be tuning this in the future as per performance metrics.