changed: Some (Silverlight) servers don't play nice with CURLOPT_RESUME_...

[arnova]

...FROM_LARGE, those need CURL_OPT_RANGE

[arnova]

@elupus, this is a followup for PR2179. The silverlight streams now play nice with Libcurl although there's still something weird going on: after a seek, dvdplayer keeps seeking back in the stream causing the file cache to never fillup. You can see this here: http://pastebin.ubuntu.com/1611686/

[elupus]

Could you post the difference this patch makes to the http request.

On Tue, Feb 5, 2013 at 8:56 AM, arnova notifications@github.com wrote:

@elupus https://github.com/elupus, this is a followup for PR2179. The
silverlight streams now play nice with Libcurl although there's still
something weird going on: after a seek, dvdplayer keeps seeking back in the
stream causing the file cache to never fillup. You can see this here:
http://pastebin.ubuntu.com/1611686/

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2187#issuecomment-13118428.

[arnova]

Curl WITH Range Opt: http://pastebin.com/M4DGVm8X
Curl WITHOUT Range Opt: http://pastebin.com/HQNY96Kv

Note that this still doesn't completely fix seeking with Silverlight streams. There's still something strange going on in DVDPlayer. After a seek, vq never returns to 100% rather to 30% (where aq is 100%) and slow decreases until playback stalls. In the meantime ffmpeg keeps seeking back in the stream (as shown by the logs). As if after a seek there's all of a sudden some poor interleave/delay between a/v. I also tried the ffplay of OUR ffmpeg and that works fine with the same stream. Any ideas?

[elupus]

That is just strange. There are no range requests at all in one of the
logs. How would seeking work on any server at all??

[elupus]

Maybe related:

Daniel S (25 April 2007)

• Sonia Subramanian brought our attention to a problem that happens if you
  set
  the CURLOPT_RESUME_FROM or CURLOPT_RANGE options and an existing
  connection
  in the connection cache is closed to make room for the new one when you
  call
  curl_easy_perform(). It would then wrongly free range-related data in the
  connection close funtion.

[arnova]

Well that doesn't explain why CURLOPT_RANGE partially fixes the issue, right? Also note that there seem to be 2 issues at play here:

• Curl not handling RANGE/RESUME properly for redirections
• DVDPlayer not properly handling the mp4/264 stream (the aq/vq / seek back issue I mentioned above), even when ffmpeg's http transport is used. No point in fixing Curl if the problem in DVDPlayer isn't fixed either.

[arnova]

@elupus: Had another look on what happens without this patch with one of my own http servers when playing video files and seeking. This has also works as it should. In thise case Curl debug shows this:

Curl::Debug Range: bytes=45525318-
Curl::Debug Content-Range: bytes 45525318-733485423/733485424

So apparently both (=content-range & range) get set when seeking is working properly. But somehow under certain situations (on redirects?) the RANGE-option needs to be set explicitly which is what is done with this PR. It therefor seems a safe workaround for the issue.

I think this may be some bug or weird behavior (we don't understand) in libcurl itself, not sure. I've already debugged this issue extensively but this PR is the only thing that seems to remedy it.

[arnova]

@ulion : Since you've been doing some Curl stuff as well: Do you have any ideas about this?

[ulion]

Range was set by request client, Content-Range is the server response.
I will try play the file.

[ulion]

@arnova can you give me an url like the one in your pastebin log current can play, the one in log seems expired.

[arnova]

The streams use session-tokens so I can't provide one directly. Just install this addon: http://www.rieter.net/ext/?uri=http://xot.hamans.com/net.rieter.xot-3.3.3.zip . In the addon go to "Uitzendinggemist.nl" (the one on top), go to "Vandaag (....)" and select one of the items in the list (they may not all work due to copyright restrictions outside of NL).

[ulion]

they can be played on my working build, with some patches related to mp4
stream cache, but the un-cached part is un-seekable, seek will result
decode error, what's the problem and result expected?

[arnova]

That's exactly the problem that is fixed by applying this PR/patch: seeking outside cache causes XBMC to freeze (or sometimes even crash). With this patch (and your cache patch in the other PR) all is well (I've been running a patched version on my HTPC for a few weeks now and that seems to working well).

Note that I also did some digging in the Curl source and one thing I noticed is the fact that there are some places in http.c where RESUME_FROM is "handled" where RANGE is not which probably explains why this PR fixes the problem for me. I suspect some bug in libcurl causing RESUME_FROM data being reset (eg. with redirects?) where RANGE data is not. Just look for all "state.resume_from"-references in http.c and you'll see what I mean.

[ulion]

current xbmc code indeed seek on the stream, but the result data in
dvdplayer produce decode error, I will keep dig on this.
wireshark captured:

GET
/62f99efca3b88ba7110b4b37aa2f9883/512c0656/s01/ceresodi/h264/1/vara/rest/2013/VARA_101310416/std.20130225.m4v?odiredirecturl=/video/embedplayer/h264_std/7f36002ed5a8afeff4109e669d4b5ceb/512c0652/VARA_101310416/%3Ftype%3Dhttp%26amp%3B
HTTP/1.1
Range: bytes=28281220-
User-Agent: XBMC/13.0-ALPHA1 Git:20130221-4b35908 (iOS; ;
http://www.xbmc.org)
Host: content50c3a.omroep.nl
Accept: */*
Connection: keep-alive

HTTP/1.1 206 Partial Content
Content-Type: video/mp4
Accept-Ranges: bytes
ETag: "533582880"
Last-Modified: Mon, 25 Feb 2013 21:41:49 GMT
Content-Range: bytes 28281220-93448902/93448903
Content-Length: 65167683
Connection: close
Date: Tue, 26 Feb 2013 00:48:40 GMT
Server: lighttpd/1.4.28

[ulion]

Problem confirmed, it's the server problem, it send out video file with different content-length with or without Range request, that's why the patch in this pr can workaround the problem, but it's really a server related case, should not be considered as a common situation:
response without Range:

curl -o xx.m4v -v "http://content2c4a.omroep.nl/4d3905ae1858854d1ea47e53af26f184/512d320a/ceresodi/h264/1/ncrv/rest/2013/NCRV_1585845/std.20130226.m4v?odiredirecturl=/video/embedplayer/h264_std/078a2967a3ba84e0c72bada2715e5389/512d3206/NCRV_1585845/%3Ftype%3Dhttp%26amp%3B"
* About to connect() to content2c4a.omroep.nl port 80 (#0)
*   Trying 145.58.50.138...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0connected
* Connected to content2c4a.omroep.nl (145.58.50.138) port 80 (#0)
> GET /4d3905ae1858854d1ea47e53af26f184/512d320a/ceresodi/h264/1/ncrv/rest/2013/NCRV_1585845/std.20130226.m4v?odiredirecturl=/video/embedplayer/h264_std/078a2967a3ba84e0c72bada2715e5389/512d3206/NCRV_1585845/%3Ftype%3Dhttp%26amp%3B HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: content2c4a.omroep.nl
> Accept: */*
> 
< HTTP/1.1 200 OK
< X-Mod-H264-Streaming: version=2.2.9
< Content-Type: video/mp4
< ETag: "1619160542"
< Last-Modified: Tue, 26 Feb 2013 18:26:20 GMT
< Content-Length: 189176442
< Connection: close
< Date: Tue, 26 Feb 2013 22:09:48 GMT
< Server: lighttpd/1.4.28
< 
{ [data not shown]

response with Range:

curl -o xx.m4v -v "http://content2c4a.omroep.nl/4d3905ae1858854d1ea47e53af26f184/512d320a/ceresodi/h264/1/ncrv/rest/2013/NCRV_1585845/std.20130226.m4v?odiredirecturl=/video/embedplayer/h264_std/078a2967a3ba84e0c72bada2715e5389/512d3206/NCRV_1585845/%3Ftype%3Dhttp%26amp%3B" -r "0-"
* About to connect() to content2c4a.omroep.nl port 80 (#0)
*   Trying 145.58.50.138...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to content2c4a.omroep.nl (145.58.50.138) port 80 (#0)
> GET /4d3905ae1858854d1ea47e53af26f184/512d320a/ceresodi/h264/1/ncrv/rest/2013/NCRV_1585845/std.20130226.m4v?odiredirecturl=/video/embedplayer/h264_std/078a2967a3ba84e0c72bada2715e5389/512d3206/NCRV_1585845/%3Ftype%3Dhttp%26amp%3B HTTP/1.1
> Range: bytes=0-
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: content2c4a.omroep.nl
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Content-Type: video/mp4
< Accept-Ranges: bytes
< ETag: "1561464652"
< Last-Modified: Tue, 26 Feb 2013 18:26:20 GMT
< Content-Range: bytes 0-189442425/189442426
< Content-Length: 189442426
< Connection: close
< Date: Tue, 26 Feb 2013 22:10:54 GMT
< Server: lighttpd/1.4.28
< 
{ [data not shown]

[ulion]

But from my sight, there's no way the addon can do to workaround it, since the server return different video file according whether the request include Range request, and I tested force add "Range: bytes=0" header will break the play, then we have to workaround it in CurlFile.
we can just add this before each set RESUME line, or change to use RANGE for all request.

if (m_filePos == 0)
  g_curlInterface.easy_setopt(m_easyHandle, CURLOPT_RANGE, "0-");

[arnova]

@ulion: Thanks for digging into this :-)

I also had a look at ffmpeg's implementation in their connect/resume method resembles what CURLOPT_RANGE does so I'd favor to just replace RESUME_FROM_LARGE with CURLOPT_RANGE.

@elupus: What do you think is the best approach to fix this?

[arnova]

Ping @elupus

[elupus]

If tested to not break ftp access, it's fine by me.

[ulion]

oh, ftp, then RESUME_FROM_LARGE should always be set, and RANGE is only needed when m_filePos == 0 with a comment about why it is needed: we need always send Range header since RESUME_FROM_LARGE will not send Range request header when it set to 0, but some server, e.g. xxx.xxx.xx rely on this else will return different file content which will cause seek failure.
the commit message also need be updated to show the real problem we found: not related to redirection, it's the server return different file contents with or without the Range request header.

[arnova]

@ulion: CURL_OPT_RANGE also works with FTP (tested it myself), so that isn't a problem. Only thing we need to decide is the best approach...

[ulion]

oh~, I checked the curl code, it did revert it from range string back to off_t, then only use RANGE is ok.
then updated the comment and commit message, finish this.

[arnova]

Will do. Then we can continue work on PR2229 :p

[ulion]

badly, I encounter some http servers will response 502 when request with 'Range: bytes=0-' header, but if request once without the Range header, it will then can accept request with Range header on the same url.
it looks like some image cache servers, when the url first requested, it generate some thumb file, after that, all request direct access the thumb file, but if we send Range header at the begin, it will fails with 502 error and the thumb file never generated.

good news is addon can add 'Range=' protocol option to such urls, then curl will not send Range header for these image urls.

[arnova]

We could also simply detect response 502 and try without Range: bytes=0- the first time, right?

[ulion]

this may be server specific problem, and maybe not a common way to do that, at least currently we have a workaround for that.