Unhandled error in Deferred: BodyExceededMaxSize

[richvdh]

We're seeing a few of these in the logs from the federation sender:

2021-01-12 21:41:24,475 - twisted - 254 - CRITICAL - None - Unhandled error in Deferred:
2021-01-12 21:41:24,485 - twisted - 254 - CRITICAL - None - 
synapse.http.client.BodyExceededMaxSize

[richvdh]

it looks like we should be catching BodyExceededMaxSize, so maybe it's something like: we are aborting the request, but failing to cancel it: it then goes on in the background until we get a huge response?

[richvdh]

or possibly: we are correctly catching the BodyExceededMaxSize, but it is being thrown again into a second deferred:

    def dataReceived(self, data: bytes) -> None:
        self.stream.write(data)
        self.length += len(data)
        if self.max_size is not None and self.length >= self.max_size:
            self.deferred.errback(BodyExceededMaxSize())
            self.deferred = defer.Deferred()
            self.transport.loseConnection()

... replacing self.deferred in that way seems risky. Calling loseConnection does not, afaik, guarantee that dataReceived will not be called again.

[clokep]

This is probably happening more now due to checking the well known size. (I had noticed there's a few servers that are offline and redirect to large 404 pages...) I'll take a look at this.