Improve experience on pg serialization errors

[vroldanbet]

ℹ️ Depends on authzed/authzed-go#147 and authzed/api#88 merging first

I tried to improve two aspects of serialization errors:

• retrying them on the server side to spare the client doing it themselves
• do exponential backoff
• better logging and more concise logging
• better gRPC response (Aborted instead of Unknown)

This greatly improved the odds of transactions to succeed.

Details

serialization errors are not well handled in the Postgres datastore:

• they tend to be logged by the pgx driver with potentially very large SQL statements
• they are not retried, nor there is a backoff. There are also other types of errors that are not being retried, like conn closed (which pgx considers safe to retry)
• they return to the client as unknown instead of the recommended aborted gRPC code
• no information is logged on retries in debug level
• logs can get extremely verbose due to large SQL and args

Serialization errors can also manifest as a transaction rollback, and so code is added to retry and wrap the error accordingly, in which case gRPC aborted is returned.

Additionally, retries after serialization can lead to longer requests, and longer requests can hit the
deadline. Sometimes a cancellation leads to the pgx connection being asynchronously closed, losing information about the original cause that closed it. Unfortunately, there isn't an exported type for this, so it's detected via the string. The error is turned into a multi-error that has cancellation. In a local load test, evidence shows that in all cases the original reason was cancellation.

A call to pgx.SafeToRetry is added to handle various pgx internal errors (like conn closed) and determine if they can be retried. This has helped retry errors it didn't use to retry before.

Retries also now have a backoff, since the default maximum of 10 can knock down the database. The retry function is refactored, and the minimum wait is reduced since the total backoff grows pretty quickly, and will easily
make the client hit the typical deadlines. The log message for retries is moved to Debug since serialization errors can be pretty common so it shouldn't be considered a warning.

I also detected that the backoff function wasn't respecting the context cancellation, so even when the client had canceled, the goroutine would be still asleep. The function was changed to cancel immediately after the client signal.

[josephschorr]

LGTM