retry and timeout

[kailuowang]

Two decisions:

1. shall we really allow retry when backend service return failure? (leaning towards no)
2. shall we allow retry when timeout? (leaning towards yes)

[ngbinh]

Maybe exposing something like

def shouldRetry(error: Throwable): Boolean

to let the application decide what to do? It may also return something that can dictates how to retry

[kailuowang]

@ngbinh yeah a plugable retry policy would be nice. or we can bake this into the new handler API, so that the service (or service adaptor) can determine the retry policy.

[kailuowang]

I think we it's easy to implement two policies.

1. at least once (with an optional expiration)
2. at most once (or try once).

And for at least once to work we need to wrap the work in Queue to contain information about expiration time. Also we should try deliver to a different handler if possible.