Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reliable delivery (at-least once) for Akka Typed #20984

Closed
Lasering opened this issue Jul 18, 2016 · 18 comments
Closed

Reliable delivery (at-least once) for Akka Typed #20984

Lasering opened this issue Jul 18, 2016 · 18 comments

Comments

@Lasering
Copy link
Contributor

@Lasering Lasering commented Jul 18, 2016

One can implement a custom redeliver interval in a PersistentActor with AtLeastOnceDelivery by overriding the method redeliverInterval. However we do not know how many attempts have been made to deliver the message. It would be helpful to have a function with the following signature:

def redeliverInterval(attempt: Int): FiniteDuration

This would allow implementing a redeliver interval with a backoff.

@johanandren

This comment has been minimized.

Copy link
Member

@johanandren johanandren commented Jul 18, 2016

I'm afraid we are stuck with the current redelivery option because of our binary compatibility rules (adding a new method to a trait that user classes inherits would break AFAIR).

@ktoso

This comment has been minimized.

Copy link
Member

@ktoso ktoso commented Jul 19, 2016

So in the best case scenario it would be for 3.X release?

Yeah. Although with Scala 2.12 such change would be binary compatible, but we're still supporting 2.11 so we can't do that too quickly.

It could be possible to add this by an additional trait, extending functionality of PersistentActor.
I.e. trait AdvancedRedeliveryBlaBlaBadName { this: PersistentActor =>

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Aug 8, 2016

Good idea, but I don't see how we can change it in a compatible way. Marking it for Akka 3.0

@patriknw patriknw added this to the 3.x milestone Aug 8, 2016
@patriknw patriknw added 1 - triaged and removed 0 - new labels Aug 8, 2016
@Lasering

This comment has been minimized.

Copy link
Contributor Author

@Lasering Lasering commented Apr 3, 2018

Would this be feasible for Akka Persistence Typed since its marked as May Change?

@ktoso

This comment has been minimized.

Copy link
Member

@ktoso ktoso commented Apr 4, 2018

We don't have an AtLeastOnceDelivery trait in Typed yet, but yes in general -- could be done there

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Apr 5, 2018

yes, let's consider this when implementing it for typed. I have a few other ideas we should also try out.

@patriknw patriknw added the t:typed label Apr 5, 2018
@patriknw patriknw removed this from the 3.x milestone Apr 5, 2018
@Lasering

This comment has been minimized.

Copy link
Contributor Author

@Lasering Lasering commented Apr 20, 2018

@ktoso bumping the issue as discussed in the meetup

@ktoso

This comment has been minimized.

Copy link
Member

@ktoso ktoso commented Apr 23, 2018

Thanks! I see, this one... Would you want to try implementing it for Akka Typed Persistence? We don't have the AtLeastOnceDelivery trait there yet so the new method could indeed be added there as we introduce that APi

@Lasering

This comment has been minimized.

Copy link
Contributor Author

@Lasering Lasering commented Apr 23, 2018

I could try to implement it. But I would take a lot of time, since I would have to get some experience using Akka Typed, and understand how AtLeastOnceDelivery is implemented in Untyped Akka.

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Apr 24, 2018

Thanks for offering help @Lasering
I think this might not be the best first contribution, since AtLeastOnceDelivery might require some design rethinking while we are anyway defining it's API and behavior.

@patriknw patriknw changed the title redeliverInterval with attempt number to implement redeliver with a backoff? Typed AtLeastOnceDelivery (including redelivery with a backoff) Apr 24, 2018
@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Apr 24, 2018

My thinking is that we should see what we can do to simplify usage of AtLeastOnceDelivery, perhaps targeted at one single destination?

It's also very heavy since it's typically require persisting all messages and the confirmations. Perhaps we should also provide some lightweight variant that only keeps messages in memory and will not cover the case when sending node crashes.

@Lasering

This comment has been minimized.

Copy link
Contributor Author

@Lasering Lasering commented May 22, 2019

Would the initial ideia be considered for Akka 3.0?

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented May 24, 2019

You mean 2.6.0? Given how many other things we have in scope for 2.6.0 this one might not make it. However, it's something I find important and want to work on as soon as the must haves are done.

@taeguk

This comment has been minimized.

Copy link
Contributor

@taeguk taeguk commented Sep 10, 2019

Excuse me, there is no plan for this? I really need AtLeastOnceDelivery in typed.

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Sep 10, 2019

We absolutely want to have a good solution for this. As it looks like now it will be added after the 2.6.0 release. If you want it earlier you can implement it yourself. I’ll try to put together a sketch of how it can look like.

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Sep 11, 2019

@taeguk Here is an example that you can use as a starting point: https://gist.github.com/patriknw/514bae62134050f24ca7af95ee977e54

@patriknw patriknw removed this from Backlog, post 2.6.0 in Akka Typed Nov 8, 2019
@patriknw patriknw added this to Backlog in Akka 2.6.x Nov 8, 2019
@patriknw patriknw self-assigned this Nov 8, 2019
@patriknw patriknw added 3 - in progress and removed 1 - triaged labels Nov 8, 2019
patriknw added a commit that referenced this issue Nov 8, 2019
@patriknw patriknw mentioned this issue Nov 8, 2019
9 of 9 tasks complete
@patriknw patriknw moved this from Backlog to In progress in Akka 2.6.x Nov 18, 2019
patriknw added a commit that referenced this issue Dec 6, 2019
patriknw added a commit that referenced this issue Dec 11, 2019
patriknw added a commit that referenced this issue Jan 23, 2020
patriknw added a commit that referenced this issue Feb 17, 2020
patriknw added a commit that referenced this issue Mar 10, 2020
patriknw added a commit that referenced this issue Mar 11, 2020
@patriknw patriknw changed the title Typed AtLeastOnceDelivery (including redelivery with a backoff) Reliable delivery (at-least once) for Akka Typed Mar 12, 2020
patriknw added a commit that referenced this issue Mar 12, 2020
Different approach than in classic AtLeastOnceDelivery because I would like:
* support flow control, with a work pulling approach
* be possible to use with or without persistence (without it may loose
  messages if producer node crashes)
* detect lost messages on the consumer side and let that drive resends,
  instead of aggressively resending from producer side
* deliver messages in order and deduplicate resent messages
* have an efficient protocol for acknowledgments over the network (not ack each message),
  but still have a simple one-by-one protocol for the end user

* support 3 use cases (building blocks)
  * point-to-point
  * work pulling
  * sharding
* optional durable queue, with one event sourced implementation
* protobuf serialization
* ApiMayChange
* reference docs and examples
* api docs
* doc example code missing so far
patriknw added a commit that referenced this issue Mar 12, 2020
Reliable delivery in Typed, #20984
@patriknw patriknw added this to the 2.6.4 milestone Mar 12, 2020
@patriknw patriknw closed this Mar 12, 2020
Akka 2.6.x automation moved this from In progress to Done Mar 12, 2020
@Lasering

This comment has been minimized.

Copy link
Contributor Author

@Lasering Lasering commented Mar 16, 2020

@patriknw I think the initial idea got lost:

def redeliverInterval(attempt: Int): FiniteDuration

Looking at the various settings classes:
https://doc.akka.io/api/akka/2.6/akka/actor/typed/delivery/ConsumerController$$Settings.html#withResendInterval(newResendInterval:scala.concurrent.duration.FiniteDuration):akka.actor.typed.delivery.ConsumerController.Settings
https://doc.akka.io/api/akka/2.6/akka/actor/typed/delivery/ProducerController$$Settings.html#withDurableQueueRequestTimeout(newDurableQueueRequestTimeout:scala.concurrent.duration.FiniteDuration):akka.actor.typed.delivery.ProducerController.Settings

I see no way to implement an exponential backoff.

In my use case the Consumer might be unavailable for some days. So I don't want the Producer to give to soon, nor to take a large amount of times between retries. Having an exponential backoff would solve this problem.

@patriknw

This comment has been minimized.

Copy link
Member

@patriknw patriknw commented Mar 17, 2020

The whole design is very different from classic AtLeastOnceDelivery. It is not pushing redeliveries from the producer side, that is driven by the consumer side. Therefore no need for exponential backoff. If the Consumer is unavailable the Producer will also be idle.

There is one exception to this. The very first message is retried if it's not acknowledged by the Consumer. I see now that the configuration for that resend interval accidentally got hardcoded to 1 second. I have created issue #28747 for configuration of that interval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Akka 2.6.x
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.