we should backoff on 40x errors

[richvdh]

Currently we do not increment the per-host backoff counter on 405 (method not allowed) responses. If we get a 405, it suggests a poorly-configured destination, so we should back off.

Indeed, the only 40x codes we do back off for are 401, 429, and (sometimes) 404. If our spec actually properly defined the valid error responses, we could do much better here.

[richvdh]

currently the federation sender will back off sending transactions to a host which returns:

• a 50x error
• a 401
• a 429
• a 404, but only for certain endpoints

(see https://github.com/matrix-org/synapse/blob/v1.26.0/synapse/util/retryutils.py#L176-L183)

This seems... arbitrary. For example, a homeserver which has been configured with a federation whitelist will return a 403 to any homeserver not on that whitelist. Surely we should back off in that case.

[richvdh]

related: #8915, #8917