Add max_latency to BackgroundThreadTransport

[tcwalther]

This PR adds a new parameter max_latency to BackgroundThreadTransport in logging. It can be used to enforce batching by asking the transport to wait for new log messages for a specified amount of time. This is useful to avoid hitting the Stackdriver Logging rate limit.

This PR is written on behalf of Spotify, who, as a company, has signed a corporate CLA. The idea for this PR is based on a support case with Google.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.
• If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot. The email used to register you as an authorized contributor must be the email used for the Git commit.
• In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[tcwalther]

I signed it!

[googlebot]

CLAs look good, thanks!

[tcwalther]

When reviewing this PR, I would be particularly interested in your opinion on how safe it is when a Python processes terminates due to an exception - is there a possibility for log messages getting lost?

[tcwalther]

I've looked at the code a bit further.

There is a method called _Worker._main_thread_terminated, which claims to send unsent logs. However, it does not seem to do anything other than calling print() a couple of times. It seems to me that this is a bug in the current implementation already.

Thoughts?

[theacodes]

There is a method called _Worker._main_thread_terminated, which claims to send unsent logs. However, it does not seem to do anything other than calling print() a couple of times. It seems to me that this is a bug in the current implementation already.

The call to self.stop attempts to block until the background thread has sent pending logs.

When reviewing this PR, I would be particularly interested in your opinion on how safe it is when a Python processes terminates due to an exception - is there a possibility for log messages getting lost?

This is always possible, yes, however, an exception on the main thread (or any thread other than this background thread) will not cause this to lose logs. Lost logs can happen due to one of three reasons:

1. The request to send the logs fails (due to timeout, etc.)
2. The main thread dies for any reason (graceful exit, exception, etc.) and sending the pending logs takes longer than grace_period seconds.
3. There is a bug in the background thread that causes it to terminate without sending logs.

[theacodes]

@tcwalther I re-wrote the test to use monotonic time to keep our unit tests deterministic and fast, totally understand not wanting to do it - it's not an obvious way to write the test.

This looks good, I'm merging. Thank you for doing all of this (and enduring my review).

@dpebot will you merge when tests pass?

[dpebot]

Okay! I'll merge when all statuses are green and all reviewers approve.