Add docs describing retry behavior

docs/architecture/retry.md

@@ -0,0 +1,38 @@
+# Retry Mechanisms
+
+## gRPC
+
+### Interceptor
+
+All services use the gRPC interceptor `interceptor.RetryableInterceptor` that can retry a failed gRPC request.
+
+Its behavior is defined by:
+- `backoff.IsRetryable` which decides whether to retry based on the service error type
+- `backoff.RetryPolicy` which decides how long to backoff first - or not retry at all
+
+The `RetryableInterceptor` delegates the actual retry behavior to `backoff.ThrottleRetryContext` which:
+- never retries any context timeout/cancellation errors
+- retries `ResourceExhausted` service errors using a custom retry policy
+
+Tip: To inspect each service's retry behavior, look for its call to `NewRetryableInterceptor`.
+
+### Service Error
+
+Service errors are specific Go errors that can generate a gRCP `Status` (see [status.proto](https://github.com/grpc/grpc/blob/master/src/proto/grpc/status/status.proto)). 
+A gRPC status contains a gRPC `Code` (see [code.proto](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto)), a message and (optionally) a payload with more details.
+
+```go
+type ServiceError interface {
+    error
+    Status() *status.Status
+}
+```
+
+[api-go](https://github.com/temporalio/api-go/tree/master/serviceerror) defines most service errors:
+- general-purpose errors
+(such as `Canceled`, `NotFound` or `Unavailable`)
+- specialized errors which carry more details 
+(such as `NamespaceNotActive` with the gRPC code `FailedPrecondition`)
+
+Furthermore, a few more Server-specific service errors are defined in this repository, such as
+`ShardOwnershipLost` or `TaskAlreadyStarted`.