add client_wait_for_response option to some method calls

[risicle]

This is a part of a potential way of addressing https://trello.com/c/LABe8BKA

The pattern it introduces is something I like to call async-not-async. It's simply allows a user of the apiclient to perform a highly simplified form of async http request by ignoring the response from the server. It does this by setting the read timeout to something tiny and allowing that timeout to happen. It relies on the fact that web server stacks tend not to readily propagate "the client is gone" information, and a request will continue to be processed fully, even after the client has given up.

In the case we're proposing to use it it gives us basically all the advantages of true async but also all the perils: how easy it is to launch hundreds of requests in a couple of milliseconds without considering how the recipient will deal with it. Pushed too far, what we'd effectively end up doing is using our http servers' connection buffers as a not-very-good task queue, which could lead to healthcheck failures.

Why prefix the kwarg with client_? Because previously all arguments passed to apiclient methods implied they were referring to server options. This is the first relating to client behaviour, and the distinction could start to get ambiguous if and when we start doing things like adding a server-side option to the api's update_service endpoint to instruct it to perform its daisychained index request to the search-api without waiting for the response.

[risicle]

I've also added the wait_for_index option to DataAPIClient.update_service() which in turn uses the above, wait_for_response, option (inside the api's view).

[risicle]

Ok, this does not work IRL, though I think I know what I've got to do to fix it, so don't bother testing quite yet...

[benvand]

What commands are we going to issue as client_wait_for_response in production?

[risicle]

Currently the only one planned is SearchAPIClient.index(), but had there been an easy existing mechanism to add is as a common option across all calls, I might have added it to everything. But there wasn't.

[risicle]

Ok it works now, but when used in this way it emits a bunch of log events to the effect that the request failed. Which it kinda didn't. I'll see what I can do.

[risicle]

This works "fully" now, only through a lot of investigation into what exceptions requests really lets through to the caller. Short answer: it varies. And a lot of them wrap other exceptions in slightly non-standard ways so I've had to build a method _iter_exceptions_by_cause to find these exceptions in the most flexible way possible.

[katstevens]

I will try and make some time today to test this out locally (with the corresponding API branch).

[katstevens]

OK - have tested this locally, let's try it 👍