-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
At least once semantics #259
Comments
I see there's The sealed trait I mentioned could maybe encode the concept of recoverable errors, unrecoverable errors, and maybe different retry policies (like Schedule.exponential) and so on. If we go with that kind of design, This might require some more thought. |
To give a better idea of the alternate design I was thinking of: sealed trait MessageAction
object MessageAction {
case object Success extends MessageAction
case object Ignore extends MessageAction
case class Delay(delay: Duration) extends MessageAction
case class RecoverableError[E](error: E, retryPolicy: Schedule[Clock, Any, Duration]) extends MessageAction
case class UnrecoverableError[E](error: E) extends MessageAction
} For the This design feels more "ZIO" to me than my original proposal. And maybe you could make |
The retry stuff only exists on the Publisher right now. I understand the need for |
Oops, missed the fact that retry was on As for the relying on retries on ZStream for the consumer, that would work mostly fine in a single consumer case (with the exception of App restarts/crashes... you'll start retrying from 0 again). The issue is mostly with multiple consumers. If you have multiple consumers in a distributed system pulling entries from the same queue, you can't share a ZIO retry policy between them. This is why I use Another case for Delay(0) or Ignore: Let's say I'm doing a rolling deployment with my consumer nodes. They will be running different versions for a while. In this kind of case you may want to say, "I don't know how to handle this version of the message. Let some other consumer pick it up." By the way, I created a separate issue in ZIO that's kind of a prerequisite for the retries I had in mind: |
I'm looking to migrate to zio-sqs, but right now as far as I can tell there are 2 main modes of operation:
autoDelete=true
: This is basically at most once semantics. If your handler dies or there is a hard crash, the message you were in the middle of handling is lost forever.autoDelete=false
: Manual mode. You have to ensure to calldeleteMessage
and so on properly yourself.I think it would be nice if there was something similar to Alpakka's
SqsAckSink
:https://doc.akka.io/docs/alpakka/current/sqs.html#updating-message-statuses
Basically through the type system to ensure that you handle what to do with a message after your handler has run. This gives you at least once semantics (or if you ensure your handler is idempotent, then it's effectively exactly once semantics).
A sealed trait with the following cases would cover everything I believe:
MessageAction
Done
/Delete
Skip
/Ignore
RetryLater(visibilityTimeout)
/ChangeMessageVisibility(visibilityTimeout)
Not sure on the specific naming to use, but that's the basic idea.
Is this something that makes sense for zio-sqs?
The text was updated successfully, but these errors were encountered: