feat(pubsub): retry transient failures in ack/modack in timed unary buffer

[NivedhaSenthil]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea.
• Follow the instructions in CONTRIBUTING. Most importantly, ensure the tests and linter pass by running bundle exec rake ci in the gem subdirectory.
• Update code documentation if necessary.

closes: #18235

[NivedhaSenthil]

Tests are failing due to a regression in minitest's latest version(5.16.0). Fixed via #18396

[snippet-bot]

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

• Refresh this comment

[dazuma]

What happens if the thread pool is shut down between the above two lines?

[NivedhaSenthil]

It throws #<Concurrent::RejectedExecutionError: Concurrent::RejectedExecutionError> which gets propagated via the error callback registered on subscriber. But this should be a rare scenario as retry_thread_pool would be shutdown only on a stop call on the subscriber and all stream shutdown are waited before shutting the buffer.