New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate adding reliability in the multicast streams #49

Open
krokodilerian opened this Issue Nov 23, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@krokodilerian
Contributor

krokodilerian commented Nov 23, 2016

Currently UDP works nice, but on lost packets creates problems in the stream. Something that could help is any kind of FEC (forward error correction). Someone has written it and it's at https://ffmpeg.org/pipermail/ffmpeg-devel/2016-June/194856.html , but it needs to be tested if it's applicable.
(I looked into different solutions, like reliable multicast and couldn't find anything)

@krokodilerian krokodilerian self-assigned this Nov 23, 2016

@danielinux

This comment has been minimized.

Show comment
Hide comment
@danielinux

danielinux Nov 23, 2016

Member

I've recently written something similar for VLC, I don't know if it's applicable to our current setup.
In case (c)vlc is a viable solution, I'd be available to help deploying the patch on top of the latest vlc master, or someone could adapt the code on top of ffmpeg. This matrix-based FEC has proven to be effective up to 3-5% of packet loss with a moderate overhead.

Links to the proposed patches so far:
https://mailman.videolan.org/pipermail/vlc-devel/2016-September/109637.html
https://mailman.videolan.org/pipermail/vlc-devel/2016-October/109868.html
https://mailman.videolan.org/pipermail/vlc-devel/2016-October/110119.html

Member

danielinux commented Nov 23, 2016

I've recently written something similar for VLC, I don't know if it's applicable to our current setup.
In case (c)vlc is a viable solution, I'd be available to help deploying the patch on top of the latest vlc master, or someone could adapt the code on top of ffmpeg. This matrix-based FEC has proven to be effective up to 3-5% of packet loss with a moderate overhead.

Links to the proposed patches so far:
https://mailman.videolan.org/pipermail/vlc-devel/2016-September/109637.html
https://mailman.videolan.org/pipermail/vlc-devel/2016-October/109868.html
https://mailman.videolan.org/pipermail/vlc-devel/2016-October/110119.html

@markvdb

This comment has been minimized.

Show comment
Hide comment
@markvdb

markvdb Nov 23, 2016

Member

Can you elaborate on what problems you expect/see in the multicast stream? I'll ask you on irc too, but it might be interesting to explain in this ticket for future reference...

Member

markvdb commented Nov 23, 2016

Can you elaborate on what problems you expect/see in the multicast stream? I'll ask you on irc too, but it might be interesting to explain in this ticket for future reference...

@krokodilerian

This comment has been minimized.

Show comment
Hide comment
@krokodilerian

krokodilerian Nov 23, 2016

Contributor

@markvdb , the issue is that this is basically an UDP stream - a packet lost is lost data. This will lead to artifacts in vocto and other problems, and will basically mean that for some rooms we'll have to re-create the video streams from the dumps on the boxes, if^Wwhen there are network issues.

Any sort of error correction that can deal with loss without generating too much delay will improve the quality of the streaming.

Contributor

krokodilerian commented Nov 23, 2016

@markvdb , the issue is that this is basically an UDP stream - a packet lost is lost data. This will lead to artifacts in vocto and other problems, and will basically mean that for some rooms we'll have to re-create the video streams from the dumps on the boxes, if^Wwhen there are network issues.

Any sort of error correction that can deal with loss without generating too much delay will improve the quality of the streaming.

@krokodilerian

This comment has been minimized.

Show comment
Hide comment
@krokodilerian

krokodilerian Nov 23, 2016

Contributor

@danielinux, this looks great :)

For using vlc, we have to use it on both sides (sending from the boxes and ingesting in voctomix), but ffmpeg and vlc don't seem to have a lot in common. I can't do the porting and I have no idea who can, though...

How much is this tested, and do you have any idea about the stability of your patches?

Contributor

krokodilerian commented Nov 23, 2016

@danielinux, this looks great :)

For using vlc, we have to use it on both sides (sending from the boxes and ingesting in voctomix), but ffmpeg and vlc don't seem to have a lot in common. I can't do the porting and I have no idea who can, though...

How much is this tested, and do you have any idea about the stability of your patches?

@danielinux

This comment has been minimized.

Show comment
Hide comment
@danielinux

danielinux Nov 24, 2016

Member

The mechanism (in a different implementation) has been extensively tested.
It should be doable to implement something like this in our setup if required.

I suggest that we discuss around this on Sunday.

Member

danielinux commented Nov 24, 2016

The mechanism (in a different implementation) has been extensively tested.
It should be doable to implement something like this in our setup if required.

I suggest that we discuss around this on Sunday.

@markvdb markvdb added the 2018 label Jan 14, 2017

@krokodilerian krokodilerian changed the title from Investigate pro-mpeg for ffmpeg for boxes->vocto communication to Investigate adding reliability in the multicast streams Feb 7, 2017

@krokodilerian

This comment has been minimized.

Show comment
Hide comment
@krokodilerian

krokodilerian Feb 7, 2017

Contributor

This screwed us up a lot, because there was loss in the multicast. Either a FEC or a reliable transport is required for 2018.

Contributor

krokodilerian commented Feb 7, 2017

This screwed us up a lot, because there was loss in the multicast. Either a FEC or a reliable transport is required for 2018.

@RichiH

This comment has been minimized.

Show comment
Hide comment
@RichiH

RichiH Feb 7, 2017

Member

Especially if we get 10G, can we not simply switch back to unicast?

Member

RichiH commented Feb 7, 2017

Especially if we get 10G, can we not simply switch back to unicast?

@markvdb

This comment has been minimized.

Show comment
Hide comment
@markvdb

markvdb Feb 7, 2017

Member
Member

markvdb commented Feb 7, 2017

@RichiH

This comment has been minimized.

Show comment
Hide comment
@RichiH

RichiH Feb 7, 2017

Member

Multicast is icky from networking PoV, so if you can avoid it, we would prefer that.

Member

RichiH commented Feb 7, 2017

Multicast is icky from networking PoV, so if you can avoid it, we would prefer that.

@krokodilerian

This comment has been minimized.

Show comment
Hide comment
@krokodilerian

krokodilerian Feb 7, 2017

Contributor

Well... The architecture is described in https://github.com/FOSDEM/video/blob/master/Architecture.md , and i drew an ugly scheme at https://github.com/FOSDEM/video/blob/master/graph/fosdem-video-2017.pdf , but basically if we switch to unicast UDP, the issue will remain, and if we switch to TCP, there are some more issues another issues:

  • there should either be a server on the boxes that provides the stream, or single stream to the voctops that you can restream from there. The first case creates another component on the boxes, the second case makes the streamdumpers dependent on the voctops.
  • TCP is lossless, and in some cases you want to be able to drop packets to catch up with the streams, for example if there's a delay to some packets. Basically what I need is more reliable RTP, that's why FEC would fix the problem.

10G might also fix the problem by just having bigger buffers at some of the points (and one better switch for the voctops + streamdumpers will fix the rest). Multicast is used not because it saves on bandwidth that much (with 4mbps streams i can run 3 copies of the 72 streams in 1G ), but because of the flexibility and ease for a lot of the tasks.

Contributor

krokodilerian commented Feb 7, 2017

Well... The architecture is described in https://github.com/FOSDEM/video/blob/master/Architecture.md , and i drew an ugly scheme at https://github.com/FOSDEM/video/blob/master/graph/fosdem-video-2017.pdf , but basically if we switch to unicast UDP, the issue will remain, and if we switch to TCP, there are some more issues another issues:

  • there should either be a server on the boxes that provides the stream, or single stream to the voctops that you can restream from there. The first case creates another component on the boxes, the second case makes the streamdumpers dependent on the voctops.
  • TCP is lossless, and in some cases you want to be able to drop packets to catch up with the streams, for example if there's a delay to some packets. Basically what I need is more reliable RTP, that's why FEC would fix the problem.

10G might also fix the problem by just having bigger buffers at some of the points (and one better switch for the voctops + streamdumpers will fix the rest). Multicast is used not because it saves on bandwidth that much (with 4mbps streams i can run 3 copies of the 72 streams in 1G ), but because of the flexibility and ease for a lot of the tasks.

@krokodilerian

This comment has been minimized.

Show comment
Hide comment
@krokodilerian

krokodilerian Feb 7, 2017

Contributor

@RichiH , also multicast is required for IPv6, so I'm not that worried about its support (it worked fine almost all the time except one hiccup with ULB's network).

Contributor

krokodilerian commented Feb 7, 2017

@RichiH , also multicast is required for IPv6, so I'm not that worried about its support (it worked fine almost all the time except one hiccup with ULB's network).

@markvdb markvdb added 2019 and removed 2018 labels Dec 28, 2017

@markvdb

This comment has been minimized.

Show comment
Hide comment
@markvdb

markvdb Dec 28, 2017

Member

As discussed by email, we're keeping the existing setup and pushing changes to tests at a smaller conf between FOSDEM editions.

Member

markvdb commented Dec 28, 2017

As discussed by email, we're keeping the existing setup and pushing changes to tests at a smaller conf between FOSDEM editions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment