[2.5 rc4] Users hitting 403 (tricky to reproduce)

[schrd]

Describe the bug
In a load test with bots users got kicked out from the meeting with a 403 "you have been removed from the meeting" message. There was an unintended configuration problem on the server which resulted in all listen only participants being connected to freeswitch instead of mediasoup. Freeswitch then consumed all available CPU on the server, top showed 0.x% idle CPU. Not all bots were able to connect and few of the humans in the meeting were kicked out.

To Reproduce
I don't know how I can reproduce this. It happend only once in several tests.

BBB version:
BBB 2.5 rc4

Desktop (please complete the following information):

• OS: Linux
• Browser Firefox
• Version 91 ESR

Additional context

I don't know if this behaviour of kicking out users in an overload situation is intended. If it is then there should be a different message. If you don't consider this a serious bug, I'm fine with this. Just didn't want to ignore our observation

[ffdixon]

Thanks for reporting this issue @schrd.

[mokazemi]

We've also experienced this for some users in the meetings. (in both desktop and android). I also don't know where it's coming from.
403 "you have been removed from the meeting" message.
Even if the user was moderator.

There was an unintended configuration problem on the server which resulted in all listen only participants being connected to freeswitch instead of mediasoup.
...

But I haven't changed any specific configurations. Just a bit of playing with audio/video bitrates.
Next time I'll try to check browser console logs when I experienced the same issue again, to check if it has any useful information.

[mokazemi]

This issue happened to me again!
I noticed two things:

• My internet connection was so bad, and it went on reconnecting state a few times, then suddenly I saw 403 Error
• I saw this error in the console logs:

Uncaught (in promise) Call to chatMessageBeforeJoinCounter failed because Meteor is not connected

I hope it helps.

[ffdixon]

Thanks for sharing this @mokazemi. It looks like the client may have just lost connection to the server (after trying to reconnect). Do you know if any other users in the session experienced the same issue (i.e. the problem looks closer to your internet connection and not with the server)?

[ffdixon]

We've released 2.5.4 and we're still tracking this issue.

We starting to look at ValidateAuthToken/the reconnection procedure. Our theory is there may be a reconnection issue that, when triggered, will flood meteor with events, which causes a CPU spike and similar to FreeSWITCH using all the memory, causes clients to disconnect with a 403 error.

Of course, the challenge is right now to replicate this reconnection issue. @schrd We'd be interested if your able to force this happening in that release under testing load.

[MBM1607]

@ffdixon we are facing this same issue, Its reproduceable under stress testing.

[MBM1607]

We were able to reproduce the issue by joining with 8+ users, and remove connection to force reconnecting attempts.

Error

Uncaught TypeError: a.getSubscription(...) is null

Uncaught (in promise) Call to chatMessageBeforeJoinCounter failed because Meteor is not connected

[BrentBaccala]

I can reproduce this consistently.

I've seen the Call to chatMessageBeforeJoinCounter error, but it seems to come after the 403 error, so I think it's incidental to the issue.

Still investigating.

[BrentBaccala]

This is reproducible on 2.6.0-alpha.2, in addition to 2.5.4.

[ffdixon]

In reproducing the issue on 2.5, does it matter if allowDuplicateExtUserid is set to true or false? See

https://groups.google.com/g/bigbluebutton-setup/c/Qefm8dduv5Y/m/SYThs_6uAQAJ

[blueiceprj]

Hi. If you tracking websocket connections over userid and the parameter (allowDuplicateExtUserid) set true it could be a problem. Best solution to handle websocket connection without problem is using stomp broker (rabbitmq). With this method you can solve websocket connection problems( reconnect, heartbeat, etc) and also it will help to solve you access bbb over the load balancer. We have based on spring cloud app and multiple nginx and gateway. It was only one option for us.

[BrentBaccala]

@ffdixon, allowDuplicateExtUserid doesn't seem to make a difference

[MBM1607]

@ffdixon After upgrading to 2.5.6, I am no longer able to reproduce this issue. Previously on version 2.5.4, I was able to join using firefox multi-containers and then trigger reconnections by turning the connection on and off repeatedly.

[ffdixon]

@ffdixon After upgrading to 2.5.6, I am no longer able to reproduce this issue. Previously on version 2.5.4, I was able to join using fire multi-containers and then trigger reconnections by turning the connection on and off repeatedly.

That's very positive feedback -- thanks for sharing!

[MBM1607]

Is it possible that it was resolved due to #15723? seems relevant to me,

[ffdixon]

Check on your server, that setting is currently false by default

https://github.com/bigbluebutton/bigbluebutton/blob/v2.5.x-release/bigbluebutton-html5/private/config/settings.yml#L210

But if you enable it, it will reduce the load when users fall back to long polling (which isn't very efficient and too many users doing long polling could cause disconnects for others, which is why we introduced this setting).

[ffdixon]

Hi Brent, I'm curious on your tests with the latest 2.5.8 regarding 403 disconnects.

[BrentBaccala]

@ffdixon, an hour-long test with ten clients and a modest level of broken TCP sessions (all sessions broken once every ten seconds) yielded no client disconnects of any kind.

I'd still like to collect some more data on this issue, though.

[ffdixon]

@ffdixon, an hour-long test with ten clients and a modest level of broken TCP sessions (all sessions broken once every ten seconds) yielded no client disconnects of any kind.

Thanks Brent! Keep pushing the boundaries, but very positive indeed.

[Davka]

I I'm not sure if it's the same bug, but we have a similar problem in 2.6.1
The meeting sizes were limited to 40 participants and after a certain time even moderators / presenters are removed from the meeting with the 403 message. Re-entry is not possible. Even afterwards, if there are only 4 people in the meeting, for example, each new participant gets the 403 message.

The browser console shows following error.

I haven't found anything in the logs yet

@antobinary

[hostbbb]

so it looks like max_participant counter is rejecting in above browser console. Wonder if reconnects of users keep adding to the meeting count somehow. A good test could be to set max_participant to 3 for a meeting, and play around with brower refreshs and new users joining. Let me try to replicate in 2.6.1

[hostbbb]

#17699

See this for 2.6.4 example. can replicate with maxParticipants set to 2