mirrord hangs when agent is not created due to OutOfCpu

[shaikustin]

Bug Description

when the agent pod is not created due to OutOfCPU error, mirrord keeps on trying to create the agent and hangs on waiting for pods to be ready.
error on cluster is

 Warning  OutOfcpu  6s    kubelet  Node didn't have enough resource: cpu, requested: 100, used: 1858, capacity: 1930                                                                                                                      

Steps to Reproduce

highly active node when running mirrord

Backtrace

No response

Relevant Logs

No response

Your operating system and version

non relevant

Local process

non relevant

Local process version

No response

Additional Info

No response

[aviramha]

This is interesting - we don't have any resources request as we're not a SLA-based service, so we just use "extra" resources (atleast what we want).
I assume your cluster defines a default requests policy, so even if a pod spawns with no requests specified it adds it.
I'm thinking to address this issue with two changes:

1. add requests to our pod, so we won't get the default 100 which isn't probably fully used (I'd assume we're around 0-20)
2. better error handling in this case.

I'll take care of the default requests.

[eyalb181]

Should be resolved in #599