push federation retry limiter down to matrixfederationclient

[richvdh]

rather than having to instrument everywhere we make a federation call, make
the MatrixFederationHttpClient manage the retry limiter.

This might result in a couple of places getting NotRetryingDestination excepitons where they weren't previously expected - but I suspect that most of those code paths are like the send-invite path and don't have any exception handling for any other exceptions either (#2047), so this isn't making things much worse.

Hopefully this will fix #1737, #1829, #1463 and #1569.

[richvdh]

balls, the tests are failing

[erikjohnston]

+1 on moving this logic down.

I think we should seriously think about the UX of limiting all requests to a domain. E.g. if I can't invite someone or lookup an alias on a domain for several hours after it coming back up its going to be quite frustrating. I suggest we add a param (that defaults to off) that allows us to specify that some requests shouldn't be subject to retry

[richvdh]

shouldn't be subject to retry

s/retry/backoff/, I assume

I certainly wouldn't be averse to such a parameter. Presumably if such a request succeeded, we'd need to reset the backoff so that subsequent requests also didn't get backed off. (If I invite someone, it's no good being able to successfully invite them if I can't then speak to them).

[richvdh]

@erikjohnston: PTAL?

[erikjohnston]

lgtm, modulo test errors