feature(streams): streams retry

[pcanto-hopeit]

PROBLEM:

Currently, when an event triggered from a message in a stream fails to be processed or to be saved into an output stream, the original message if consumed and thereby lost.

REQUIREMENT:

To minimize message loss, a retry mechanism needs to be implemented at the hopeit engine level. Allowing for failed events to be retried again in a certain period of time, up to a number of retries. Also a DLQ (Dead Letter Queue) needs to be populated with messages which processing is failing after the number of retries is exhausted.

NOTES:

-Retry with exponential backoff could be an acceptable strategy.
-Initial retry time and number of retries should be configurable per event.

CONSIDERATIONS:

-hopeit engine should transparenly implement the retry strategy disregarding which plugin is used as a stream manager.
-Retry could also be disabled in cases where native stream tech support retry. In this case, we might ned to consider properly implenting acknowledging of messages after the whole processing is sucessful *current ack works when the meesage is only consumed but not yet processed.
-Probably retry queues needs to be created by the engine for the backoff periods setup (i.e for 1 minute initial retry up to 3 times, there will be 3 new queues: rety_1m, retry_2m, retry_4m. This applies for each stream that has retry enabled. Another option is to implement a single queue for retrying and keep polling messages and check age, and put them back if there are not yet to be retried.
-Dead letter queu must be done using a persistence plugin (i.e. fs storage), but probably configurable.

-Dead letter queue could be implemented as a first step (with no retry strategy)
-Acknowledge of consumed messages now is done by the engine, but it will be probably better to do in the the specific stream manager. (i.e. redis)