Exponentially reduce Kubernetes API calls

[RaviHari]

Is your feature request related to a problem? Please describe.
In Active-monitor the status of the workflow is polled every second which results in too many kubernetes api calls. As the number of monitors or self-healing use cases increases, in a managed kubernetes clusters this can result in additional cost, in non-managed kubernetes clusters the master nodes might have to be scaled up to accomodate these api calls.

Describe the solution you'd like
This can be solved by leveraging https://github.com/keikoproj/inverse-exp-backoff library. There is an API in this library where we can use inverse exponential backoff with timeout. This will allow us to reduce the API calls made to the kubernetes API server exponentially.

Have you thought about contributing yourself?

Yes I will implement it.

If you want to get involved, check out the
contributing guide, then reach out to us on Slack so we can see how to get you started.