Today, there was another mystical incident on our XMPP experience.
So, we were sending and receiving our messages as usual, but suddenly this happened:
[11:51:04 VRB] XMPP stanza (outgoing): <message id="1e20c4a8-1a7c-4260-8b29-296120d26dd0" type="groupchat" to="codingteam@conference.jabber.ru" xmlns="jabber:client">
<body><@folexeyy> >> <Minoru> @folexeyy: любопытно. AFAIK две инстанса одного крейта можно получить, только если у них разные версии. Может, ты из Git тянешь какую-то чересчур новую? Если версиями не удастся разрулить, я бы попробовал добавить либу сабмодулем (в твой репозиторий) и тоже воспользоваться локальным путём — может, так Cargo лучше осознает ситуацию
Там версия де-факто одинаковая, но для карго это непрозрачно</body>
[11:51:04 INF] Message 1e20c4a8-1a7c-4260-8b29-296120d26dd0 has been sent; awaiting delivery
The answer for this message id, 1e20c4a8-1a7c-4260-8b29-296120d26dd0, never was received. So, the bot has fallen into somewhat catatonic state: it stopped sending any messages to XMPP, and even stopped logging the message queue.
The situation was resolved after Dr. @ForNeVeR stept in and manually restarted the XMPP server. Due to our brilliant lifetime system, the message awaiter was cancelled due to reconnect, and the system become responsible once again, and even was able to flush the message queue into XMPP.
There's a couple of mitigation steps required for this:
- Add message timeouts for XMPP. If we haven't received an answer to our message in a long time (say, 5 minutes?), we should suspect that something is very wrong, restart the connection, and re-send the message in question.
- We should investigate what has happened with the message queue in this case. Probably something wasn't right, because I cannot see the Telegram messages queuing up in the XMPP actor queue. Probably they were queuing somewhere else in some non-logged and non-persisted place, which is bad.
Today, there was another mystical incident on our XMPP experience.
So, we were sending and receiving our messages as usual, but suddenly this happened:
The answer for this message id,
1e20c4a8-1a7c-4260-8b29-296120d26dd0, never was received. So, the bot has fallen into somewhat catatonic state: it stopped sending any messages to XMPP, and even stopped logging the message queue.The situation was resolved after Dr. @ForNeVeR stept in and manually restarted the XMPP server. Due to our brilliant lifetime system, the message awaiter was cancelled due to reconnect, and the system become responsible once again, and even was able to flush the message queue into XMPP.
There's a couple of mitigation steps required for this: