RabbitMQ Transport Reconnect Logic #2651

chriswiggins · 2021-04-29T09:51:29Z

Hey Janus team! 👋

As we're running Janus and RabbitMQ in Kubernetes, there are times where our Janus instance will connect to one of our RabbitMQ nodes that then eventually goes down as we're doing maintenance on our hosts. Anyone with a clustered RabbitMQ setup with a load balancer in front will have the same thing happen to them.

Currently, the RabbitMQ Event handler does reconnects, but the Transport does not. This PR brings reconnect logic into the Transport.

It looks like a large change, but realistically most of it is just moving the large connection-based logic into a new function which we call at startup, and again when we notice the connection has dropped. It also brings in RabbitMQ heartbeat functionality.

I'm opening this as a draft to get initial feedback (and while I'm doing tweaks and testing) so any feedback would be awesome 👍

chriswiggins · 2021-04-29T09:56:01Z

Looks like the checks are failing with the updated RabbitMQ-C imports - I was getting a deprecation when compiling this and using master rabbitmq-c - I will see if I can do a conditional import on those. Any pointers on how I can do that?

lminiero

Thanks for the patch! I've added a few notes inline, mostly editorial. I'll let people using the RMQ transport in production get back to you on testing and feedback.

events/janus_rabbitmqevh.c

transports/janus_rabbitmq.c

lminiero · 2021-04-29T10:05:34Z

As a side note, please notice one of the CI builds failed:

transports/janus_rabbitmq.c:36:10: fatal error: 'rabbitmq-c/amqp.h' file not found
#include <rabbitmq-c/amqp.h>

lminiero · 2021-04-29T10:07:30Z

Looks like the checks are failing with the updated RabbitMQ-C imports - I was getting a deprecation when compiling this and using master rabbitmq-c - I will see if I can do a conditional import on those

Yep, sorry, I answered before seeing this message.

Any pointers on how I can do that?

Not really, no: maybe some finer check in configure.ac?

atoppi · 2021-04-29T10:28:29Z

@chriswiggins as of CI build failure, bear in mind that the test script installs the package librabbitmq-dev from apt, so maybe the available version is not the expected one.

https://packages.ubuntu.com/focal/librabbitmq-dev

transports/janus_rabbitmq.c

chriswiggins · 2021-04-30T04:01:55Z

I'm working on the reconnect logic now and over the weekend - more to follow next week. Thanks for the initial feedback!

chriswiggins · 2021-05-04T04:37:08Z

Hey team - want to have a look at that now and see what you think?

lminiero

Thanks, added a few notes inline 👍

transports/janus_rabbitmq.c

atoppi

thanks for the effort @chriswiggins , I've left some comments

conf/janus.transport.rabbitmq.jcfg.sample

transports/janus_rabbitmq.c

lminiero · 2021-05-06T13:04:20Z

@chriswiggins thanks for the quick fixes! This looks fine to me, so unless you wanted to do any additional changes, this is good to merge for me.

chriswiggins · 2021-05-09T23:00:05Z

I've just done some extra testing and am now happy with the changes - I made a final couple of tweaks, mainly around the event handler logging so it's easier to see what relates to that vs the transport plugin.

If you're happy with those then I'm happy for you to merge!

lminiero · 2021-05-10T07:26:38Z

Ack, thanks! Just FYI, I'll probably change the "connected successfully" verbosity you added recently from LOG_INFO to LOG_VERB.

commit 9c9d335 Author: Lionel Nicolas <lionelnicolas@users.noreply.github.com> Date: Thu Jun 10 03:04:43 2021 -0400 Fix streaming plugin mutex unlock when disabling mountpoint (meetecho#2690) commit 2d83e96 Author: Yurii Cherniavskyi <yurii.cherniavskyi@gmail.com> Date: Mon Jun 7 16:02:41 2021 +0300 Fix SIP plugin unhold request docs typo (meetecho#2688) commit 2cd0118 Author: August Black <augustblack@gmail.com> Date: Mon Jun 7 01:10:49 2021 -0700 minor adjustment to the audiobridge docs (meetecho#2687) commit de2117e Author: nicolasduteil <nduteil@freedev.org> Date: Tue Jun 1 11:26:29 2021 +0200 fix: [janus_sip] Fix "call_id" property in "missed_call" events (meetecho#2679) commit 9eeeb38 Author: Alessandro Toppi <atoppi@meetecho.com> Date: Mon May 31 15:57:41 2021 +0200 Fix status vector parsing for incoming twcc feedbacks (resolves meetecho#2677). commit 8a25f6e Merge: d3b39b9 394fb48 Author: Alessandro Toppi <atoppi@meetecho.com> Date: Fri May 28 13:29:54 2021 +0200 Merge pull request meetecho#2675 from kmeyerhofer/actions/fix GH Actions, fix variable name commit d3b39b9 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Fri May 28 11:09:30 2021 +0200 Fixed race condition in VideoRoom commit 394fb48 Author: Kurt Meyerhofer <k@kcmr.io> Date: Thu May 27 14:52:08 2021 -0600 Fixes variable name. commit b45cd37 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Thu May 27 18:31:55 2021 +0200 Clarify that libnice 0.1.18 is recommended commit 5757a37 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Thu May 27 17:08:17 2021 +0200 Spatial audio support in AudioBridge via stereo mixing (meetecho#2446) commit 161fe7a Author: Luca Barbato <luca.barbato@gmail.com> Date: Thu May 27 15:29:01 2021 +0200 Cleanup avformat-based preprocessors (meetecho#2665) commit 7b010cd Author: lucylu-star <78361868+lucylu-star@users.noreply.github.com> Date: Tue May 25 17:09:40 2021 +0800 Fixed broken simulcast support in VideoCall plugin (meetecho#2671) commit 4ae44a4 Author: nicolasduteil <nduteil@freedev.org> Date: Mon May 24 17:57:34 2021 +0200 feat: support for custom call-id in subscribe request + add 'call_id' property to subscribe & notify related events (meetecho#2664) commit 4294f20 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Mon May 24 11:02:48 2021 +0200 Fixed missing macro when using pthread mutexes (fixes meetecho#2666) commit f22ab0d Author: Lorenzo Miniero <lminiero@gmail.com> Date: Wed May 19 12:03:32 2021 +0200 Fixed warning commit b3f3f17 Author: Alessandro Toppi <atoppi@meetecho.com> Date: Tue May 18 12:10:47 2021 +0200 Remove duplicated flag for fuzzing coverage. commit 4a7560c Author: nu774 <honeycomb77@gmail.com> Date: Fri May 14 00:26:36 2021 +0900 janus-pp-rec: support HEVC AP(aggregation packet) (meetecho#2662) commit 5db4be2 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Wed May 12 17:43:43 2021 +0200 Fixed out of bounds array access commit 69f56f4 Author: nicolasduteil <nduteil@freedev.org> Date: Tue May 11 14:36:22 2021 +0200 feat: support for SUBSCRIBE expiry (Expires header) in sip plugin (meetecho#2661) commit b047ccf Author: Lorenzo Miniero <lminiero@gmail.com> Date: Mon May 10 09:33:27 2021 +0200 Fixed types commit f8e8c5e Author: Chris Wiggins <chris@wiggins.nz> Date: Mon May 10 19:26:45 2021 +1200 RabbitMQ Transport Reconnect Logic (meetecho#2651) commit 280e8e4 Author: Lorenzo Miniero <lminiero@gmail.com> Date: Fri May 7 12:54:30 2021 +0200 Add per-participant recording options in AudioBridge to join API as well

chriswiggins added 3 commits April 29, 2021 21:46

Update rabbitmq-c imports

d22ce2f

Start to build out RabbitMQ Transport reconnect logic

c29bb87

Update comment

6775799

lminiero reviewed Apr 29, 2021

View reviewed changes

tmatth reviewed Apr 29, 2021

View reviewed changes

transports/janus_rabbitmq.c Outdated Show resolved Hide resolved

chriswiggins added 2 commits April 30, 2021 14:36

Check for newer rabbitmq-c libraries

03d8c8f

Code style tweaks

0b3f5f5

Adds exponential backoff to reconnections

6c33cbf

Adds RabbitMQ heartbeat to sample config

ed6a2c5

chriswiggins marked this pull request as ready for review May 4, 2021 08:44

lminiero reviewed May 4, 2021

View reviewed changes

transports/janus_rabbitmq.c Outdated Show resolved Hide resolved

transports/janus_rabbitmq.c Outdated Show resolved Hide resolved

transports/janus_rabbitmq.c Outdated Show resolved Hide resolved

atoppi reviewed May 4, 2021

View reviewed changes

chriswiggins added 2 commits May 5, 2021 09:38

Fix config typos

35166e8

Update reconnect logic so as not to block on destroy

2be9c1a

atoppi reviewed May 5, 2021

View reviewed changes

transports/janus_rabbitmq.c Outdated Show resolved Hide resolved

Only check if rabbit connection is up before destroying

8d0c25c

chriswiggins added 2 commits May 10, 2021 10:58

Update RabbitMQ event handler logging to make it more obvious

70367ab

Log when RabbitMQ successfully connects

dacd291

lminiero merged commit f8e8c5e into meetecho:master May 10, 2021

chriswiggins deleted the rabbitmq-reconnect branch May 11, 2021 00:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RabbitMQ Transport Reconnect Logic #2651

RabbitMQ Transport Reconnect Logic #2651

chriswiggins commented Apr 29, 2021

chriswiggins commented Apr 29, 2021 •

edited

Loading

lminiero left a comment

lminiero commented Apr 29, 2021

lminiero commented Apr 29, 2021

atoppi commented Apr 29, 2021

chriswiggins commented Apr 30, 2021

chriswiggins commented May 4, 2021

lminiero left a comment

atoppi left a comment

lminiero commented May 6, 2021

chriswiggins commented May 9, 2021

lminiero commented May 10, 2021

RabbitMQ Transport Reconnect Logic #2651

RabbitMQ Transport Reconnect Logic #2651

Conversation

chriswiggins commented Apr 29, 2021

chriswiggins commented Apr 29, 2021 • edited Loading

lminiero left a comment

Choose a reason for hiding this comment

lminiero commented Apr 29, 2021

lminiero commented Apr 29, 2021

atoppi commented Apr 29, 2021

chriswiggins commented Apr 30, 2021

chriswiggins commented May 4, 2021

lminiero left a comment

Choose a reason for hiding this comment

atoppi left a comment

Choose a reason for hiding this comment

lminiero commented May 6, 2021

chriswiggins commented May 9, 2021

lminiero commented May 10, 2021

chriswiggins commented Apr 29, 2021 •

edited

Loading