Clarify response completion

[hannahhoward]

Goals

There is a lot of confusion in the ResponseManager about when we're actually "done" with the request. This is cause the query executor completes before we've truly finished sending data -- that only happens once the message queue confirms everything has gone over the wire.

As a result it's very unclear in the code where a request ends -- has it gone over the wire or is it just done processing queries. Even most ways a request can abort still have a response code sent, meaning we need to wait till that gets sent to cleanup the request.

What's worse, cleanup tasks are spread out every where, with somewhat inconsistent implementation.

Implementation

The main change here is to add a "CompletingSend" status that tracks the time from when the query finishes executing to when the last message goes over the wire.

This makes it clearer if we're DONE DONE or still waiting for confirmation from the network queue.

We also add a method terminateRequest that cleans up the response when it's truly done, being careful to
only delete requests when they finish going over the network.

Put requests that are not processing but still going over the network in a state of CompletingSend.

[rvagg]

👌 very nice!
The main thing I don't follow here though is how we get from the dangling graphsync.CompletingSend = true states (bottom of finishTask() being the main one) to a final terminateRequest(). Is it guaranteed that we'll end up in terminateRequest() from there via responsemanager/subscriber.go#OnNext() having sent the data? Are there other paths that might get us there and is there any way that it could remain in that state and not be properly terminated? (Which I guess the new diagnostics might tell us in practice!).

[hannahhoward]

@rvagg

So: OnNext would only not fire in a case where everything was shutting down (i.e. the global graphsync context got cancelled. And it wouldn't matter anyway then :). Anyway, if there are other times, this is how we'll find that out!