No video received with simulcast & recv-video-height #6
Comments
|
These nulls could be problematic here?
Setting selectedEndpoints, onStageEndpoints to proper string and lastN to some int seems to change the behaviour a bit. Not sure though it guarantees that gstreamer will be always fed with the top 720p quality no matter what requests are passed to Colibri by other participants. |
|
We don't currently implement any real understanding of multiple video layers at all, so simulcast is likely broken in multiple ways. It's near the top of the to-do list though :) |
improves ReceiverVideoConstraints serialisation, ref #6
|
@jbg In our setup we've limited jitsi to vp8, so no spatial layers at the JVB side. The bridge seems to have 3 streams with separate SSRCS, not sure what about temporal layers with 7.5/15/30 fps scaling. Maybe you could give a hint, what part of gst-meet code to modify - to be able to force gst-meet to request the top quality stream? So far we've figured out how to route a selected video stream to gstreamer directly in the JVB Java code, but this approach breaks with simulcast enabled and packet loss, as gstreamer has a simple udpsrc, and cannot request its own NACK/RTX, which I believe does work in gst-meet. |
|
lib-gst-meet needs to parse the SSRC groups in the Jingle description to understand which SSRCs form the simulcast group. then when setting up the gstreamer pipeline it should only create the recv video pipeline once for the whole group. (the earlier part of the video decode pipeline would probably need to be instantiated once for each SSRC, but then the output of each of those could be funneled to one instance of the user's recv video pipeline). |
|
The commit I just pushed, 2968013, may help – I noticed that we are setting the |
|
Yup, I've also figured out the maxHeight, forgot to mention that. That SSRC bit you've wrote makes sense, but I'm not sure how it translates to VP8SourceProjection in JVB? This seems to trick the clients at the JVB level, that simulcast is actually a single SSRC stream? Also the seq-numbers are rewritten per each Endpoint, conference participant. When I inspect webrtc-internals of multiple clients at a simulcast enabled conference, the SSRC/height combinations differ between participants. Eg. it can start with "low" SSRC and then with good bitrate controller stats can be bumped to 720p (keeping the initial SSRC) and vice versa go 720->360->180 on the 720p SSRC. |
|
I have a theory that JVB is only forwarding the lowest layer because bandwidth probing isn't working correctly. I'll have some time to debug this later today; you could also help if you have access to the debug endpoint on the JVB you're using, by checking the bandwidth estimation values for gst-meet's endpoint. In your log above, these are the SSRCs that JVB would use for bandwidth probing: But I don't see any logging indicating that gstreamer created pads for either of these, which means there will be no RTP depayloaders for them, which means no transport-cc feedback for them, which means bandwidth estimation won't work. |
Fixes failure to deserialise EndpointStats due to missing serverRegion field (noticed in log of #6)
The real JID of MUC participants is not always available (noticed in the log of #6)
|
This seems to be the actual reason for no video being received: (Real JID of the participant not available, so gst-meet's on_participant implementation fails.) Should be fixed in a57ce26 |
|
Thanks, I've encountered the "missing jid" once, but it did show up video for some other cases with simulcast enabled. I'll try to debug the jvb's bandwidth estimation today, |
|
I've managed to get a basic simulcast scenario working. Got a single stream with forced constant 720p at gst-meet, while the same user was displayed as a 180p thumbnail and 360p grid by other participants using the regular client. RTX seems to work properly, great stuff! :) I aim to have a single gst-meet instance per user, so used --last=n=1 and a single --select_endpoint=eid, had to modify the onParticipant callback to match this single eid to make it always work: Plus, it initially crashed on user probing. With our setup focus had both the jid and and nick atrribute defined: The beginning of the stream is a bit weird. If seems to start with the lowest layer and "selects" high quality after a few seconds. I think I've seen this behavior with a regular javascript lib-jitsi-meet before. Could it be connected with missing bandwidth estimation? There are no Colibri endpoint stats for gst-meet in JVB logs server-side, so I guess it doesn't send any bandwidth/rtt/loss stats indeed. |
|
Thanks for this!
If you are able to get 720p and you haven't disabled trust-bwe on the JVB side, then I think bandwidth estimation must be working to some extent. |
If the sender is (lib-)jitsi-meet this would be based on |
Ah, I forgot I've set trust-bwe=false at JVB. With trust-bwe=true it works for around 10s, then gets empty lastN notification and stops. |
|
Right, so bandwidth estimation is indeed still not working correctly. I assume that it's something to do with gstreamer's twcc (tcc) implementation but haven't had a chance to debug it in depth yet. |
Yup, after bumping startBitrate to 720p stream bitrate; gst-meet starts with 720p nicely. No need to fiddle with jingle.rs manually. |
|
As of 3557183 you can re-enable trust-bwe on the JVB; TWCC is working and the JVB will ramp up to the higher layers. I'll close this ticket and track the motion artifacts in the other ticket. |
With simulcast enabled and without the command line argument
--recv-video-heightthe video is received but has very low resolution (it's understandable with simulcast). Is there any way to get full resolution video from all participants? I've tried--recv-video-height=720but then no video is received, see log below.I've seen in README that simulcast is not fully implemented, what remains to be done?
The text was updated successfully, but these errors were encountered: