redis_streams: support for XAUTOCLAIM
#2662
Labels
needs investigation
It looks as though have all the information needed but investigation is required
XAUTOCLAIM
#2662
Context: https://redis.io/docs/latest/develop/data-types/streams/#recovering-from-permanent-failures
My understanding is that the best way to deal with permanent failures in redis stream processing is to now periodically call
XAUTOCLAIM
to grab any messages that are still showing up in the PEL past a certain reason time (e.g. 1h).If we added an
autoclaim
stanza withevery
andafter
fields, we could periodically runXAUTOCLAIM
insideread
: https://github.com/redpanda-data/connect/blob/main/internal/impl/redis/input_streams.go#L286For example, the above
autoclaim
stanza would runXAUTOCLAIM mystream $CONSUMER_GROUP $CONSUMER_NAME 3600 0-0 COUNT 10
every 5m. Note thatXAUTOCLAIM
actually returns the claimed entries, so these would need to be added tor.pendingMsgs
as well.Probably the easiest way to test this new behavior would be to add an integration test that purposefully doesn't ack every other message, then use a short autoclaim period like
every 10s after 30s
and monitor thatXINFO GROUPS $CONSUMER_GROUP
continues to progress.The text was updated successfully, but these errors were encountered: