databroker: implement leases

[calebdoxsey]

Summary

This is a distributed mutex algorithm implementing using leases with TTLs. This allows multiple databrokers with only one identity manager loop.

The flow is:

1. acquire a lease
2. call the handler with a cancelable context
3. periodically renew the lease. If this fails, cancel the context.
4. If the parent context hasn't been canceled, try to acquire the lease

If the lease can't be acquired the Leaser will just keep retrying every ttl/2.

The backend implements leasing via a single method that can be called in 3 ways:

1. The initial lease is created by using a new UUID.
2. Renewal is implemented by re-using the UUID for the lease id. The TTL will then be updated.
3. Release is implemented by using a TTL of 0.

The redis implementation uses a watch+transaction to ensure atomicity.

Checklist

• reference any related issues
• updated docs
• updated unit tests
• updated UPGRADING.md
• add appropriate tag (improvement / bug / etc)
• ready for review

[codeclimate]

Code Climate has analyzed commit 4f3463b and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1

View more on Code Climate.

[wasaga]

looks great

[calebdoxsey]

I updated the code to retry on gRPC errors. I also added some tests.