Feature request: Update Group Sync to retry recoverable errors indefinitely

[cameronvoell]

Is your feature request related to a problem?

For errors that are retryable (network, storage) we probably want to keep trying indefinitely so that a 5 minute network outage doesn't brick groups. It's better to have syncs fail for a period of time than to ignore a message that other group members might treat as valid.

Describe the solution to the problem

No response

Describe the uses cases for the feature

No response

Additional details

No response

[cameronvoell]

See #805 (comment) for context

[cameronvoell]

Making a note to address the concern here about having specific Error type for intents that contain invalid commits:

#820 (comment)

[insipx]

Some quick thoughts about impl:

It would make sense to add a strategy field to the general Retry struct here, (to also be available on the builder) and instantiate the api client or wherever things are being retried with that strategy. A strategy could be an enum, and this one in particular be called Infinite or something. A quicker fix would just be passing usize::MAX to retries but maybe something more verbose would be appropriate. the macros would need to change accordingly to account for strategy