HTML Client loads really slow when entering a session with long duration

[nstylianides]

Describe the bug
Running a long session, e.g. 9 hours with a lot of participants e.g. 100, the HTML Client is not functioning as expected if reconnected after 5 hours from the Meeting start.
It takes several seconds to respond and browser asks to "Wait" or "Kill" the page. If you click on "Wait" then the Browser responds and the AUDIO test starts.

To Reproduce
Steps to reproduce the behavior:
Although i am not sure, we have this experience after a meeting is open for 5 hours and user tries to connect

1. Create a meeting
2. After 5 hours try to connect
3. Wait until Browser pops the notification to "Kill" or "Wait"
4. Click on "Wait"

Expected behavior
Enter the room without delays

Actual behavior
It takes more than 30 secondes to connect. People think that the system craches and they refresh.
And that leads to longer delay.

Screenshots
If applicable, add screenshots to help explain your problem.

BBB version (optional):
2.3.4

Desktop (please complete the following information):

• OS: Any OS
• Browser Any Browser
• Version

[hostbbb]

What is the server specifications regarding cores, memory? when it happens does this happen to all new meetings that are started, or just the long running meeting?

[nstylianides]

i do not thinkg CPU and RAM is an issue. We used to run the same scenarios for over a year without a single problem with 2.2
Only the longs session ones start this behavior after 6 hours.
We started facing this issue with 2.3
I have also noticed that in my Browser Network Tab that the file websocket in my browser is also not accepting anything when this happens.
It takes around 30 seconds for the page to respond. Is as if there is a FREEZING sleep.

[hostbbb]

2.3 has node js, memory settings i believe that are looking for the min requires 16gb ram
How much ram do you have?

[hostbbb]

also think there is an issue with reading in all the past chat messages when i user joins, that was taking long to load... know that work was being done on this, but not sure where that improvement is. Do you have lots of messages in the chat in these long meetings?

[nstylianides]

I have enough memory as suggested. I think you might be correct with Chat history and Shared Notes. I have lots of them after 7 hours meeting. Even Poll results.
Is anyone dealing with it?
Normal scenario should to load and lazy load chat and shared notes.

[antobinary]

This will likely be improved significantly with #12271

Old chat re-synching was improved in 2.3 - you would not be grabbing all the data right away. But this is the first collection where we have logic similar to lazy loading.

[nstylianides]

Ok. Hope it solves the issue. But until the release of 2.4 what is the proposed solution for cases such as the one i am experiencing?
Clean chats every now and then?

[GhaziTriki]

The server being used here is a baremetal and have the following specs:

Intel Xeon-E 2136 - 6c/ 12t - 3.3GHz/ 4.5GHz
32 GB DDR4 ECC 2666MHz
2 x 500 GB SSD NVMe Soft RAID
1Gbps

[mabras]

We get the "Kill" or "Wait" message just yesterday. The session was about 90 minutes. there were about 200 participants. Sharing screen in active. Chats is active.
I do not think this related directly to "long duration" sessions, this is related to performance in general.

BBB 2.3.4
8 cores with 32 ram

[hostbbb]

1 session 250users above the recommended 150max in a class. Would be interested in any cpu/memory logs, and errors found in the bbb server logs during this meeting.. Was it the only meeting running at the time? Lots of full audio? Video? ScreenShare?

[nstylianides]

Maybe @mabras is right. But in my case, as reported above, i have never experienced CPU or RAM issues during this issue. CPU < 35%, Net < 130Mbps and RAM < 16%
It feels more like a raise condition and a blocking algorithm.

[mabras]

Was it the only meeting running at the time? Lots of full audio? Video? ScreenShare?

Only one meeting. no video. screen share. ~3 full audio.

[nstylianides]

Any news on this issue?

[antobinary]

I wonder if anyone has encountered this on 2.4-rc-*+
Some work was done (in particular in chat) to only sync the latest chat (unless older is explicitly requested), etc

I am tagging this to 2.4 for the moment so we can check if more work is needed.

[jonnyfux]

Not entirely sure if this is connected but the effect is pretty similar.
We are also facing degrading performance for meetings that either run for a few hours (2-3h) or/and have a lot of video cameras turned on, could not pinpoint the cause exactly. RAM, CPU all look fine. But the storage just keeps on filling up, we can watch by the minute that the percentage of used storage goes up by 0.1% during those meetings. The general storage "/" fills up to above 85% and keeps on growing, also the "mongo-ramdisk" is filling up consistently. No recordings are being made. After some time with no participants on the server, the memory drops back again.

Our BBB server run on c5a.4xlarge with 50 GB of storage attached.

The main effect we see is that the echo test takes way longer than usual to succeed.

[ffdixon]

@jonnyfux Could you try this on 2.4-rc4 (soon to be rc5) and let us know if you encounter the same issue.