Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http input issues with ffserver #78

Closed
rezna11 opened this issue Jun 29, 2018 · 8 comments
Closed

http input issues with ffserver #78

rezna11 opened this issue Jun 29, 2018 · 8 comments

Comments

@rezna11
Copy link

rezna11 commented Jun 29, 2018

I'm testing http input in a way where I'm running ffserver which provides http output, streaming mpeg-ts which it's getting from from ffmpeg. It works like this:

  1. run ffserver with a configuration that waits for data from ffmpeg and whenever it gets it, it streams it out via http to the connected clients. If it's not getting anything from ffmpeg, theren there's no data but the http connection remains open

  2. then I run ffmpeg which feeds a TS file in real time to ffserver

  3. then I run TSDuck which reads the HTTP output and re-streams to UDP or modulates

The problem starts when ffmpeg stops feeding ffserver and there's no data sent out by ffserver. It seems that at this point TSDuck tries to reconnect, which might be OK but what's strange is that when it reconnects, it reports bitrate (since I included -P pcrbitrate) of 2+ Mbps, while there's no data streamed from the http source at that time so it's not possible that it can actually see any real bitrate from PCR since there's no PCR. There's even a fragment of the last played video file on the UDP output from TSDuck. Simply restarting TSDuck fixes the issue. Also when recording the output of ffserver directly, there's nothing, which is correct.

Here's the command and its output:

C:\Users\Administrator>tsp -v -I http -i --reconnect-delay 2000 http://10.0.0.9:8090/live.ts -P pcrbitrate -O ip 239.0.0.1:6555
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* http: unknown input bitrate
* pcrbitrate: new bitrate from PCR analysis: 1,953,731 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,734,093 b/s

now I stop ffmpeg (but ffserver connection remains active, there's just no data being sent) - after that, there are constant reconnect attempts

* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,781,812 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,707,656 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,012,187 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,688,987 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,669,468 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,793,562 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,032,456 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,744,668 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,634,350 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,794,606 b/s
* pcrbitrate: new bitrate from PCR analysis: 1,979,875 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,806,193 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,534,343 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 2,896,668 b/s
* pcrbitrate: new bitrate from PCR analysis: 1,983,400 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,637,743 b/s
* tsp: user interrupt, terminating...
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes

and so on until TSDuck is restarted:

C:\Users\Administrator>tsp -v -I http -i --reconnect-delay 2000 http://10.0.0.9:8090/live.ts -P pcrbitrate -O ip 239.0.0.1:6555
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes

... waits here until ffmpeg starts feeding ffserver again, then the streaming works

What I also notice is that it's very easy to reproduce this issue if the re-started ffmpeg streams from a file with different bitrate. If the same bitrate is streamed with ffmpeg, then TSDuck is likely to survive even ffmpeg restarts.

Here's an example of reconnects and reported bitrates when I tried to stream a file at 10 Mbps, then restart ffmpeg with variable bitrate at approx 3 Mbps:

C:\Users\Administrator>tsp -v -I http -i --reconnect-delay 2000 http://10.0.0.9:8090/live.ts -P pcrbitrate -O ip 239.0.0.1:6555
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* http: unknown input bitrate
* pcrbitrate: new bitrate from PCR analysis: 9,202,390 b/s
* pcrbitrate: new bitrate from PCR analysis: 10,149,023 b/s
* pcrbitrate: new bitrate from PCR analysis: 8,855,764 b/s
* pcrbitrate: new bitrate from PCR analysis: 2,913,906 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 6,703,952 b/s
* pcrbitrate: new bitrate from PCR analysis: 10,027,180 b/s
* pcrbitrate: new bitrate from PCR analysis: 10,863,760 b/s
* pcrbitrate: new bitrate from PCR analysis: 3,696,305 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 6,741,654 b/s
* pcrbitrate: new bitrate from PCR analysis: 9,138,359 b/s
* pcrbitrate: new bitrate from PCR analysis: 10,321,453 b/s
* pcrbitrate: new bitrate from PCR analysis: 4,971,011 b/s
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes
* pcrbitrate: new bitrate from PCR analysis: 4,678,068 b/s
* pcrbitrate: new bitrate from PCR analysis: 10,747,615 b/s
* pcrbitrate: new bitrate from PCR analysis: 9,577,487 b/s
* pcrbitrate: new bitrate from PCR analysis: 6,043,259 b/s
* tsp: user interrupt, terminating...
* http: downloading from http://10.0.0.9:8090/live.ts
* http: MIME type: video/MP2T, expected size: 0 bytes

not only was TSDuck reconnecting all the time, it was also mixing together pcrbitrate messages with various bitrates, while the input was first 10 Mbps and then VBR 2.5-3.5 Mbps, it was never anything else.

@lelegard
Copy link
Member

lelegard commented Jul 1, 2018

Hello Jan,

The plugin pcrbitrate reports an average bitrate over periods of time based on distances between PCR's. It reports a new bitrate sample each time 128 PCR's have been collected on at least 1 PID. This works well with relatively stable bitrates. When the bitrate is extremely variable, especially when going from X to 0 and later from 0 to Y, the average values are meaningless with such long periods. It is like computing the integral of a periodic function using chunks wider than the period.

So, the solution is probably RTFM as usual. Try with various values of -P pcrbitrate --min-pcr 8 or 16 or 32.

@rezna11
Copy link
Author

rezna11 commented Jul 1, 2018

That would explain the bitrate reported being wrong, but do you think that it's the pcrbitrate's incorrect use causing the issues with http input when the data flow starts and stops? Should TSDuck work with a scenario where a source stops and resumes later? I'm still trying to do merging of streams (http and/or udp) where I'm creating the input streams on an encoder like ffmpeg from files but I'm still having trouble making this work without TSDuck restarts when the source itself restarts.

@lelegard
Copy link
Member

lelegard commented Jul 1, 2018

OK, sorry, I read the report too fast, focused on the bitrate issue and missed the reconnection one.

Again, the answer is probably RTFM 😆 with option --receive-timeout in plugin http. Libcurl (unix) and WinInet (windows) have different constraints and default behaviours which are suitable for stable downloads. Otherwise, you need to tune.

@rezna11
Copy link
Author

rezna11 commented Jul 1, 2018

Well I'm still not sure if --receive-timeout is what I need to change in order to fix the issue I'm having, which is not exactly a problem reading from HTTP or disconnects, the issue is TSDuck not being able to output stable stream if the http input goes to 0 or stops completely. Similar to what I experienced when merging some UDP sources.

However maybe I've found a workaround:

ffmpeg -> ffserver -> mptsd -> TSDuck

This way TSDuck output seems quite stable even if ffmpeg stops and starts - mptsd outputs CBR with null packets if ffserver stops sending data. I haven't checked PCRs, AV sync and more channel merging but it's a start. If I remove ffserver, it doesn't work reliably because mptsd loses connection to ffmpeg, which seems to interrupt the output, which causes again TSDuck to get confused somehow by something...

I'm sorry that I can't specify the problem exactly, instead I'm only describing symptoms. But I really don't know what the problem is and it's difficult to apply extra parameters from the manual if I have no clue what I'm looking for.

@rezna11
Copy link
Author

rezna11 commented Jul 1, 2018

and here's something from debug:

  1. if I use the ffmpeg -> ffserver -> mptsd -> TSDuck route, and mptsd outputs 10Mbps CBR, of which video is approx 3 Mbps or 0 if ffmpeg is restarting, I'm getting this on debug:
* Debug: merge: adjusted PCR by 11,510,932 (426 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 11,165,912 (413 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 10,769,756 (398 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 10,489,709 (388 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 10,097,613 (373 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 9,751,240 (361 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 9,822,076 (363 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 9,688,218 (358 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 9,455,847 (350 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 9,163,618 (339 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,778,290 (325 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,419,734 (311 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,079,075 (299 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 7,694,801 (284 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,156,827 (302 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,457,475 (313 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,291,431 (307 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,162,987 (302 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,056,501 (298 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 7,868,499 (291 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 7,680,797 (284 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 7,548,293 (279 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 8,867,148 (328 ms) in PID 0x0021 (33)
* Debug: merge: adjusted PCR by 10,207,360 (378 ms) in PID 0x0021 (33)

the output is modulated and plays fine

the full command looked like this:

tsp -d1 -v --bitrate 30000000 -I null -P regulate -P merge -p 0 -p 17 "tsp -v -I ip 239.1.1.1:15000 -O file" -O dektec --device 0 --modulation DVB-S --convolutional-rate 1/2 --lnb 5150 --satellite-frequency 4095000000 --level -390 --stuffing --bitrate 30000000

  1. when I use ffserver directly as input, debug outputs the following. first it looks OK, then after restarting ffmpeg, it sometimes still works fine but then when I stop ffmpeg, it goes crazy
* Debug: merge: adjusted PCR by -19,692,318 (-729 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by -18,303,530 (-677 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by -19,401,305 (-718 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by -20,500,433 (-759 ms) in PID 0x0100 (256)

after stopping ffmpeg:

* Debug: merge: adjusted PCR by 1,362,361,279 (50,457 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,361,916,393 (50,441 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,361,469,853 (50,424 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,361,127,841 (50,412 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,360,747,628 (50,398 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,360,309,510 (50,381 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,359,869,738 (50,365 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,359,423,498 (50,349 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,358,976,958 (50,332 ms) in PID 0x0100 (256)
* Debug: merge: adjusted PCR by 1,358,536,133 (50,316 ms) in PID 0x0100 (256)

and the output looks like fast forwarding video (but it must be played from a buffer or something like that, since ffmpeg is stopped

the command looked like this

tsp -d1 -v --bitrate 30000000 -I null -P regulate -P merge -p 0 -p 17 "tsp -I http -i --reconnect-delay 1000 http://10.0.0.9:8090/live.ts -O file" -O dektec --device 0 --modulation DVB-S --convolutional-rate 1/2 --lnb 5150 --satellite-frequency 4095000000 --level -390 --stuffing --bitrate 30000000

@lelegard
Copy link
Member

lelegard commented Jul 3, 2018

Hi Jan,
Going back to this one after being busy elsewhere. I am a bit lost. Could you synthesize the details of what is going wrong and what should be done instead?

@rezna11
Copy link
Author

rezna11 commented Jul 3, 2018

Well maybe I'd just like to ask a question first - is TSDuck's concept OK with a stream source (UDP or HTTP) that's not providing a consistent, 24/7 stream? The input stream (HTTP, UDP) may stop and start or it may remain connected but at times only send null packets or no data at all.

Even when I tried ffmpeg -> ffserver -> mptsd -> TSDuck workflow, thus providing TSDuck with CBR input, it didn't work for very long. I was able to stop and start the source streams (ffmpeg) a few times and it was OK during the hour or so I tested this. But then I stopped ffmpeg (so mptsd was only streaming null packets) and 6 hours later, I resumed the source ffmpeg stream but at that point I could only see frozen picture on the output (TSDuck IP output as well as dektec output). Restarting TSDuck solved the issue.

@lelegard
Copy link
Member

lelegard commented Jul 3, 2018

Normally, it should work. The tsp engine is really passive and does not take any action based on time.

The input plugin "pushes" packets in the chain. Pushing and pushing from one plugin to another. When nothing "pushes" at input, every plugin is waiting.

There is one artifact however : To avoid too many context switches between threads (each plugin executes in its own thread), a plugin thread is woken up only when a significant amount of packets are ready for this plugin (global parameter --max-flushed-packets). So, when nothing "pushes" the chain at input, many packets stagnate in the buffer, waiting for more "push" from the input to progress in the chain.

So, because of this, you may say that tsp is not well suited with sporadic input streams. You may reduce this effect using --max-flushed-packets but you won't completely get rid of it. BTW, this may explain why you saw parts of older streams when transmission restarts after a long time.

Normally, the input plugin can wait for a long time, very patiently, until new packets arrive. There is nothing in tsp which interfere with this. However, there can be a problem with the input plugin itself. Specifically, the http plugin relies on system libraries (WinInet on Windows, libcurl on Unix). These libraries are typically designed for downloading, Web services and API's. They may have problems when a connection is open and nothing comes in for a long time. Since WinInet and libcurl have very different implementations, I would suggest to try on Windows and Linux and check if there is a difference in behaviour. If there is such difference, this is probably a limitation in one of the libraries. If you can find tuning options for the faulty library which could solve the issue, just tell me and I will make these options available from the command line.

That being said, I will not rewrite a complete HTTP implementation myself. It is awfully complicated. You need support for SSL/TLS, proxy management, various procotol specificities, etc. So, we need to find a solution with the existing system libraries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants