Adding backoff mechanism to google-fcm

[JakubKahovec]

Hi,
thanks for the FCM connector. Have you thought about adding support for backoff mechanism needed by the FCM server ? It'd be very useful to have it as a part of the connector itself rather than implementing it outside.

[ennru]

cc @tg44

[tg44]

Yeah... I mentioned it in the documentation :D

If I remember well, the streams-contrib has a stage for exponential backoff. The problem in general, that we need to know when we can retry and when there is some other error.

The sendWithPassthrough can pass the retry number, so if you know when you can safely retry and how much failed send is enough to actually give up; its not that hard to write the functionality for your problem.

I had more problems when tried to generalize this;

• the (google) documentation (which is really shity in my opinion),
• the input params (how much is too much? and other stuffs I didn't remember now),
• the safeness (cycles are always problematic, which is more important sending new or retrying old, buffers? how big buffers? what happens when a buffer is full?),
• the output elements (I need to parse/translate the inner google related errors, need to document the translations etc., if I drop elements bcs of buffer overflow those will lost forever)

So if you have any idea/concept I'm happy to help/code it, but when I started to implement this I faced too much problems.

[JakubKahovec]

I agree that the FCM documentation could have been more detailed. What I assume is that when there is an error code INTERNAL or UNAVAILABLE in the response it means the message should be backed off. My thought about how it might work was that when you get a response from the server which signals a backoff it would schedule an exponential retry of sending (num of attempts configurable) and postpone fulfilling a promise which would eventually decrease demand and back-pressured. I was also also thinking about restarting the whole flow but it would mean losing some of the messages in it which is not desirable.

[tg44]

Okay so if I understand well, you want to restart the code inside the future with something like a recoverWith. This has a ton of possible problems... For example what happens when the token is about to expire but we start to retry the request (and get unauth)? What happens if all our threads are backing off (but we could send to other devices, just the actuals broken in the google side)? BTW unauth is a retryable response in some cases but the recoverWith method will make it hard to solve.

I personally think this is much worst then building a cycle... I will thinking about it.

[armanbilge]

This will be fixed in #2613.

[seglo]

Fixed in #2613