Async execution, ResultSet paging, and NoHostAvailableException

[d-t-w]

The datastax java driver pages results with a default fetch-size of 5000.

If the cluster is unavailable as the ResultSet requests the next page of results a NoHostAvailableException is thrown, there's a ticket to make paging resumable (https://datastax-oss.atlassian.net/browse/JAVA-277), but there will always be a possibility of this exception being thrown.

As much as it is unlikely to occur, there are two broad implications:

• Consumers of query execution results should know that it is possible an exception can be thrown while iterating through those results.
• It is currently possible in each of the asynchronous execution functions (-async, -chan, -chan-buffered) to throw this NoHostAvailableException within the onSuccess callback method on the ResultSetFuture and either not call the success function with a result (-async) or not place anything on the channel and close it (-chan, -chan-buffered)

In the case of -async and -chan there's only a vanishingly small window of triggering the latter case, fetch-size needs to be less than thunk-size which is probably a misconfiguration anyway. You're more likely to encounter this case when using -chan-buffered and paging through large amounts of data when a network partition occurs, then you end up waiting on a channel which will never close.

We use ccm (https://github.com/SMX-LTD/ccm-clj) for stopping/starting clusters while testing and I've been able to reproduce both cases for -chan and -chan-buffered by executing a query and stopping the cluster temporarily, I'll raise a PR which addresses it, the only further thing to add is consumers of a -chan-buffered execution result should know an exception may be placed on the channel at any time, even after some results have been successfully returned.

I feel like -chan-buffered is the most 'correct' implementation of the core.async way of querying C* and maybe should replace -chan?

[mpenet]

Hi Derek,

Good catch! While this is true that in the -chan and -async this is would be relatively rare, this is something that had to be fixed.

About chan-buffered, you're perfectly right, but in practise it's often more convenient to use one of the other functions, I think having multiple options is the way to go.

I am still waiting for promise-chan to eventually be released in core.async to replace some of the -chan machinery, but I am not holding my breath anymore...

Your PR didn't include the fix for the manifold interface, that would be welcomed too (the lamina interface is going away on the next release, no need to bother).
I am on a trip at the moment, if you dont do it, I will in the next day or so.

Thanks a lot,

Max

[d-t-w]

No worries Max, there you go.

[mpenet]

Thanks again. It's on clojars as 2.5.3