Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redis_streams: support for XAUTOCLAIM #2662

Open
ryanschneider opened this issue Jun 17, 2024 · 0 comments
Open

redis_streams: support for XAUTOCLAIM #2662

ryanschneider opened this issue Jun 17, 2024 · 0 comments
Labels
needs investigation It looks as though have all the information needed but investigation is required

Comments

@ryanschneider
Copy link

Context: https://redis.io/docs/latest/develop/data-types/streams/#recovering-from-permanent-failures

My understanding is that the best way to deal with permanent failures in redis stream processing is to now periodically call XAUTOCLAIM to grab any messages that are still showing up in the PEL past a certain reason time (e.g. 1h).

input:
  redis_streams:
    url: TODO
    body_key: body
    streams: [mystream"]
    auto_replay_nacks: true
    limit: 1
    client_id: ${CONSUMER_NAME}
    consumer_group: ${CONSUMER_GROUP}
    start_from_oldest: true
    autoclaim:
      count: 10
      every: 5m
      after: 1h

If we added an autoclaim stanza with every and after fields, we could periodically run XAUTOCLAIM inside read: https://github.com/redpanda-data/connect/blob/main/internal/impl/redis/input_streams.go#L286

For example, the above autoclaim stanza would run XAUTOCLAIM mystream $CONSUMER_GROUP $CONSUMER_NAME 3600 0-0 COUNT 10 every 5m. Note that XAUTOCLAIM actually returns the claimed entries, so these would need to be added to r.pendingMsgs as well.

Probably the easiest way to test this new behavior would be to add an integration test that purposefully doesn't ack every other message, then use a short autoclaim period like every 10s after 30s and monitor that XINFO GROUPS $CONSUMER_GROUP continues to progress.

@Jeffail Jeffail added the needs investigation It looks as though have all the information needed but investigation is required label Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs investigation It looks as though have all the information needed but investigation is required
Projects
None yet
Development

No branches or pull requests

2 participants