Split Meteor roles backend-frontend revisit

[antobinary]

This is a revisit of #10349 which was a revisit of #8788

Thank you guys - @jfsiebel , @amguirado73 , @schrd, others for your contribution!

I used code from both PRs, appended to the very latest 2.3-alpha6+ on 'develop' branch.
I only used a fraction of the code (that's why I did not cherry-pick) but made a bunch of additional changes which I think were needed to adapt this to 2.3.

Described benefits (of #10349 , this PR still has to prove itself in battle)

• support for larger meetings (i.e. more people) -- the frontends take the publish/subscribe/streamer/etc load and the backend(s) can be dedidacated to redis event handling and writing to collections
• better spread of the load over the nodejs processes (before this strategy a server with 4 (hybrid do-everything) nodejs processes would only be properly utilized/loadbalanced if each process was serving an equally busy subset of the meetings. Now we can configure 1 backend + 3 frontends and users will have better experience and likely more users would 'fit' on the server before service degradation would occur

Frontend(s):

• receive the ValidateAuthTokenResp event to complete authentication
• collection subscription and publishing
• other DDP events including method calls to send events to Akka Apps
• handle completely the Streamer redis events: Cursor, Annotations, External video share
• still require MeetingStarted and MeetingEnded events to create/destroy per-meeting event processing queues

Backend(s):

• handle all the non-streamer events
• if more than one backend is running, bbb-web splits the load in round-robin fashion by assigning an 'instanceId'. So individual backends only process redis events for the meetings matching the process' instanceId
• ValidateAuthTokenResp - is passed to backends as well, which is needed for the cases where you only have a backend, no frontends - for example dev environments that do not need to care about scaling

settings.yml change - the from-akka-apps-[^f]* syntax is the way to filter out the channel between akka-apps and the frontends

In most cases a single backend process should suffice. The possibility of more than one is tied to the reason why we added support of multiple nodejs processes in first place - huge load of redis messages, breakouts stemming from main rooms with 100+ participants, main meetings starting/ending around the same time.

To be honest I did not quite digest the "nginx load balancing" only referring to "frontend" processes and the large amount of code changes included in 10349. Thank you @schrd for addressing me in Jan 2021 to go over 10349 again!
After picking specific changes from that PR I did not see need (yet) of having external video sharing overhaul through akka-apps. Possibly I missed something but it worked fine for me locally.

See the comments below for more detail

I am open for suggestions and recommendations.

[antobinary]

I did not cherry-pick the commit from #10349 as I use fraction of the code changes but I really appreciate the work put in! If/when we bring in the external video events through akka-apps, that might be better time to cherry-pick from that (or the previous) pr

[amguirado73]

Hello,

good job!!!. In PR #10349 there are also changes in akk-apps to support events (stop/start) for external videos. I know that 2.2.31 have improvements in performance and 2.3 use several instances of nodejs to split serveral meetings (one meeting is only in one nodejs process, but one nodejs process can handle serveral meetings). Anyway I think split one meeting in several nodejs is a better solution to increase then number of attendees in one meeting. I sincerely await the results of the performance tests to see the real increase in number of users by meertinf for both solutions.

Regards

[antobinary]

I made significant changes to the code:

• removed the /1/ instanceId from the html5 client URLs
• allowed support for multiple frontends and multiple backends
• introduced scripts and config file in the tree to allow for HTML5 client developers to deploy their [modified] version of the client locally and re-spin the mulptiple configured backend/frontends
• updated the bbb-html5.nginx file (see below) with most of the changes from @ramonlsouza in Applies changes needed to serve locale files as static content #11234 https://gist.github.com/ramonlsouza/e45ecb5a5ab1bc3bc0a4f18e5cd93113
• rewrote the redis.js logic for which nodejs instance is supposed to process the events for which meetings (resolves a bug in the current implementation on 'develop')

For the packaging of frontend vs backend I borrowed from @schrd 's https://gitlab.hrz.tu-chemnitz.de/bigbluebutton/bigbluebutton-packaging/-/tree/master/bbb-html5

Next steps:

• try this out on a demo server under load within 2.3 alpha7 or alpha8
• hopefully read some feedback from the community, especially those who run the '2.2.31+patch'. ((Note that I suspect you are seeing some odd mongodb behaviour due to meetings being added to db by both frontends and backends. I was seeing it locally on 2.3 but made changes to address it))
• copy over the packaging to it's included in the distributed bbb-html5

Here are my nginx configs.
cat /etc/bigbluebutton/nginx/bbb-html5.nginx

rewrite ^/html5client/(fonts|compatibility|resources|svgs)/(.*) /html5client/$1/$2;

location /html5client/compatibility {
  alias /usr/share/meteor/bundle/programs/web.browser/app/compatibility;
}

location /html5client/resources {
  alias /usr/share/meteor/bundle/programs/web.browser/app/resources;
}

location /html5client/svgs {
  alias /usr/share/meteor/bundle/programs/web.browser/app/svgs;
}

location /html5client/fonts {
  alias /usr/share/meteor/bundle/programs/web.browser/app/fonts;
}

location ~ ^/html5client/ {
  # proxy_pass http://127.0.0.1:4100;           # for dev mode uncomment this line
  proxy_pass http://poolhtml5servers;           # for dev mode comment out this line
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
}

cat /etc/nginx/conf.d/bbb-html5-loadbalancer.conf

upstream poolhtml5servers {
  zone poolhtml5servers 32k;
  least_conn;
  server 127.0.0.1:4100 fail_timeout=5s max_fails=3;
  server 127.0.0.1:4101 fail_timeout=5s max_fails=3;
}

[github-actions]

This pull request has conflicts ☹
Please resolve those so we can review the pull request.
Thanks.

[antobinary]

I am merging this PR so it can be ran some more before it's included in 2.3-alpha7.
There are still some subtleties to be ironed out with the connection manager but this seemed stable on the server where I had it run for some time.
@schrd also suggested a different way of 'registering' of bbb-html5 backends with bbb-web to avoid using ps.

[huydq189]

I'm testing 2 instance backend bbb-html5
When user got in the right instance everything work normally but if they got to other instance they can't get the cursor stream in white board since "Error method cursor-streammer...

Is this a bug ?
@antobinary
Best regards! And sorry for my bad English

[antobinary]

Hi @huydq189
What do you mean by "they got to other instance"? Could you list steps to reproduce this?