Livestream suddenly stops: manifestParsingError hls-plugin.ts:262:12

[dabruh]

NOTE: This is a continuation of issue #3682. See my last comment in there.

Describe the current behavior

The live stream stops after a seemingly random amount of time.

Steps to reproduce:

1. Start stream to tube.tchncs.de
2. Open stream in a clean Firefox browser
3. Play the stream until it stops

Describe the expected behavior

Video should keep playing, or auto-resume.

Additional information

• PeerTube instance:

  • URL: https://tube.tchncs.de
  • version: 3.1.0

• Browser name, version and platforms on which you could reproduce the bug: Mozilla Firefox 87.0 (64-bit) for Manjaro Linux

• Link to browser console log if relevant: console-export-2021-4-13_19-32-4.txt

[Chocobozzz]

Please create an infinite stream so I can test

[dabruh]

@Chocobozzz tube.tchncs.de is not my server, but signup is free and open.

[dabruh]

@Chocobozzz not sure if you noticed the above reply as I sent it just minutes after your comment.

[Chocobozzz]

Yes I did, and I think the upgrade to hls.js will fix this issue #3962

[agates]

I saw a similar issue and someone on Matrix mentioned the following. I'm not sure if it's the same error or a separate issue

@LecygneNoir:tedomum.net:

I have also seen that "random stop" problem on stream, after tests and research it appears on my configuration that when peertube generate the "HLS playlist" containing all mp4 fragment to get, it creates the file on disk then populates it. And, if you are unluncky and your browser get the file when its empty.. The sream stops 😅
The slower your disk is, the more chance you have to trigger the problem.

Personnaly I have "patched" that by adding a conf in my reverse proxy (haproxy) to detect an empty playlist answer coming from peertube, and force the client to request again (through 301), so the file is full when the browser do the new request. Not ideal, but do the job

[dabruh]

@agates @Chocobozzz This makes total sense as Tchncs PT uses network storage. Perhaps @verymilan could fill in some details about the solution.

I think a fix would be to allow writing these files to a directory of choice (e.g. /tmp) where there's better performance and also change the way it's being written by writing the file first, then removing the old and renaming the new. And yeah, use a 301 in the worst case scenario in the very rare occasions where the above proposal wasn't fast enough.

[verymilan]

I am using a Hetzner Storagebox. It is mounted via systemd and cifs with a timeout of 30s.
I wanted to go with a mounted s3 solution as described in the docs, but i couln't figure out why the upload was so slow and couldn't migrate in the end (s3fs with Backblaze). Doing everything locally is not in budget.

[LecygneNoir]

Hello!

I can confirm I have a similar problem, as quoted by agates, it was because when streaming, peertube (probably the HLS lib) generates its playlist by creating the file, then computes and fill the file.

If a browser get the file during the phase where the file is empty, it interrupts the stream. After approximatively 1 min to 2 min freezed, the browser get the file again and continue playing, but at this point most people have already refresh the page ^^"

The slower the storage (and I guess, the CPU for computing the file), the most chance one has to get an empty file.

I have patched my reverse proxy (haproxy) by adding a hack in my backend;

      acl emptyres31 res.len eq 289
      http-response redirect location %[hdr(host)]%[capture.req.uri] if emptyres31

Basically, it checks if the answer is 289 bytes lenght (corresponding to an empty answer for Peertbeu 3.1, header included), and if this is the case, it tells the client to redo the request by using a 302 to the same location.

It requires some work and test, and it's not stable. For example, when upgrading from peertube 3.0 to 3.1, the "empty length" was 270 (in 3.0) and is now 289 (in 3.1), probably because of a new header (I didn't have a 3.0 server to check)

Also, it should be used with cautious because it may have side effects: some requests inside the federation ARE empty by design (for example 302, or some "start playing" request), so it may broke federation.
In my case, videos from my instance were unplayable on other instances federating with me because of endless redirection loop problems 😅 It appears than in 3.1, request with length 270 (previous 3.0 empty requests) are the exact size of the "start playing" request.

So, again, this is a hack, it cannot be distributed in official configuration, nor the official nginx conf....

I am unsure of what a correct patch would be. Perhaps the HLS update as stated by Chocobozzz would do the trick.

Or, something like "filled the playlist file directly" instead of "creating the file, then filling it", if possible, to avoid there is empty file on disk.

Hope the feedback may helps!

[dabruh]

Just a thought... Playing the stream through VLC works pretty well from what I can tell and have heard, so client-side improvements are welcome too.

[dabruh]

@Chocobozzz thank you for PeerTube - it's really great. However, I have to ask, approximately when will we receive a version which addresses this issue? I've got people around me that are getting impatient and want to look for other platforms to stream to. Have a good one!

[Chocobozzz]

HLS v1 has been merged in develop so I think this should be fixed in the next version. But since it's difficult to reproduce, don't hesitate to tell me if it's not and we'll reopen the issue.