RFE: add rate limiter to the broker's API handler

[jmrodri]

Feature:

There are times that APIs are getting called for what seems like forever. For example, a provision fails which will cause the service catalog to issue a deprovision. If the deprovision fails, this cycle can continue forever. Normally the broker is configured to save failed APB pods so that they may be debugged later. At the moment we save ALL APB pods which can create a lot of unwanted namespaces.

There are 2 parts to this feature:

1. look into only saving the latest failed APB pod. For example, if a deprovision call fails, and a subsequent one fails with the same parameters, then we should delete the older pod and keep the newer one

2. rate limit the API, such that after a certain period of time we don't spawn new pods at all for the API and just return that it failed always. Basically becoming a no-op.

[jmrodri]

Downstream bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1649076

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jmrodri]

/close

[openshift-ci-robot]

@jmrodri: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.