Internal Server Error attempting iPlayer download

[larryy]

While attempting to download a current program from BBC iPlayer, youtube-dl produces an HTTP Error 500: Internal Server Error. This is with the latest release version of yt-dl. Verbose log follows first.

The error is only (but always) reproduced when using --proxy. Without the proxy I get a (normal) geolocation error; non-verbose log follows second.

This proxy is brought up by my VPN client, from BolehVPN, that provides VPN over this proxy while leaving the rest of my system network stack alone. It has been working correctly for a good part of a year, though I have not been using it recently due to changes at the BBC's CDNs, and this is my first attempt to use youtube-dl. Based on Get iPlayer Automator's behavior today I believe it is functioning correctly. I would normally test and confirm its current functionality by getting the info about an existing program, but the BBC has removed or broken this (and other) functionality today (see https://github.com/get-iplayer/get_iplayer/issues/239); however, that gip bug report would suggest that youtube-dl is expected to still be working.

I don't think it matters, but this is on Mac OS X 10.11.3, running youtube-dl downloaded (today) using the standard curl and chmod recommendations. If I can provide any other useful information, please ask.

Error behavior with proxy:

$ youtube-dl --verbose --proxy 10.10.10.1:808 http://www.bbc.co.uk/iplayer/episode/p03gqw7j/the-british-academy-film-awards-2016-bafta-films-in-the-frame
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--proxy', u'10.10.10.1:808', u'http://www.bbc.co.uk/iplayer/episode/p03gqw7j/the-british-academy-film-awards-2016-bafta-films-in-the-frame']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.02.13
[debug] Python version 2.7.10 - Darwin-15.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.1.4, ffprobe 2.1.4, rtmpdump 2.4
[debug] Proxy map: {u'http': u'10.10.10.1:808', u'https': u'10.10.10.1:808'}
[bbc.co.uk] p03gqw7j: Downloading video page
ERROR: Unable to download webpage: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 354, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1905, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Expected behavior without proxy:

$ youtube-dl http://www.bbc.co.uk/iplayer/episode/p03gqw7j/the-british-academy-film-awards-2016-bafta-films-in-the-frame
[bbc.co.uk] p03gqw7j: Downloading video page
[bbc.co.uk] p03gqwgm: Downloading media selection XML
[bbc.co.uk] p03gqwgm: Downloading media selection XML
ERROR: bbc.co.uk returned error: geolocation

[larryy]

I was able to confirm that youtube-dl works just fine for this same program when used with whole-machine VPN from this same provider. But it doesn't make use of --proxy, so that's consistent with everything being fine as long as the proxy option is not used.

It appears that the Beeb has restored the program information sources, and I think I have evidence that my proxy-VPN is working as intended. Though I'm not able to use it with Firefox currently. I'm trying to get more information on that front. If I've filed a bug for a personal VPN/proxy problem, I apologize. Though perhaps a better error message is in order instead of a crash and backtrace, regardless.

[dstftw]

Most likely your proxy does not function properly. Some random UK proxy:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://www.bbc.co.uk/iplayer/episode/p03gqw7j/the-british-academy-film-awards-2016-bafta-films-in-the-frame', u'--proxy', u'185.116.213.46:3128', u'-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.02.13
[debug] Git HEAD: f66a3c7
[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: avconv 9.18, avprobe 9.18, ffmpeg N-77883-gd7c75a5, ffprobe N-77883-gd7c75a5, rtmpdump 2.4
[debug] Proxy map: {u'http': u'185.116.213.46:3128', u'https': u'185.116.213.46:3128'}
[bbc.co.uk] p03gqw7j: Downloading video page
[bbc.co.uk] p03gqwgm: Downloading media selection XML
[bbc.co.uk] p03gqwgm: Downloading m3u8 information
[bbc.co.uk] p03gqwgm: Downloading m3u8 information
[bbc.co.uk] p03gqwgm: Downloading m3u8 information
[bbc.co.uk] p03gqwgm: Downloading m3u8 information
[bbc.co.uk] p03gqwgm: Downloading m3u8 information
WARNING: Your copy of avconv is outdated and unable to properly mux separate video and audio files, youtube-dl will download single file media. Update avconv to version 10-0 or newer to fix this.
[debug] Invoking downloader on u'http://ic.b974d52e.065a49.vod-hls-uk-live.bbcfmt.vo.llnwi.net/usp/auth/vod/piff_abr_full_hd/91b4ea-p03gqwgm/vf_p03gqwgm_2721dd25-ea2d-4e5f-adfe-47cd8c531449.ism.hlsv2.ism/vf_p03gqwgm_2721dd25-ea2d-4e5f-adfe-47cd8c531449.ism.hlsv2-audio=128000-video=5070000.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 204
[download] Destination: The British Academy Film Awards, 2016 - BAFTA - Films in the Frame-p03gqwgm.mp4
[download]   0.1% of ~699.39MiB at 466.18KiB/s ETA 12:09
ERROR: Interrupted by user

[GOPsux]

youtube-dl has been flaky with iplayer the last couple of days. get_iplayer wasn't working yesterday, but it is today. youtube-dl started having issues about 24 hrs ago and hasn't worked since.

a command line request such as:

youtube-dl -f iptv_streaming_1500_akamai_hls_open-1496 "http://www.bbc.co.uk/iplayer/episode/b008zmjz/natural-world-20072008-7-badgers-secrets-of-the-sett" -o C:\out.mp4

returns the following error

[bbc.co.uk] b008zmjz: Downloading video page [bbc.co.uk] b0721m29: Downloading media selection XML [bbc.co.uk] b0721m29: Downloading m3u8 information [bbc.co.uk] b0721m29: Downloading m3u8 information [bbc.co.uk] b0721m29: Downloading m3u8 information [bbc.co.uk] b0721m29: Downloading m3u8 information [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 293 [download] Destination: C:\out.mp4 ERROR: unable to download video

FYI my DNS Proxy is working just fine with get_iplayer today and at one point 25+ hrs ago youtube-dl was working fine too. Maybe the BBC blocked youtube-dl after implementing their new HTML5 iplayer beta?

[larryy]

I'm still investigating the proxy-VPN issue, and suspect it is indeed broken.

GOPsux: Just FYI, even while get_iplayer was non-functional I used youtube-dl to download a program successfully. (The URL given in this bug report.) I can't try your command line right now because I'm on the machine that can't be put on whole-machine VPN, and my proxy-VPN appears to be broken at the moment. Did you try it without the specific -f format?

[GOPsux]

I've tried both the command line version as well as the GUI version. They both fail. You're also correct that get_iplayer wasn't working last weekend, while youtube-dl was. I wasn't able to obtain the 1080p version either even when it was working.

However Sunday youtube-dl stopped working with the bbc iplayer towards the end of the day and hasn't returned mon or tues.

[dstftw]

@GOPsux youtube-dl works perfectly with your command line:

> youtube-dl -f iptv_streaming_1500_akamai_hls_open-1496 "http://www.bbc.co.uk/iplayer/episode/b008zmjz/natural-world-20072008-7-badgers-secrets-of-the-sett" --proxy 185.116.213.46:3128 -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-f', u'iptv_streaming_1500_akamai_hls_open-1496', u'http://www.bbc.co.uk/iplayer/episode/b008zmjz/natural-world-20072008-7-badgers-secrets-of-the-sett', u'--proxy', u'185.116.213.46:3128', u'-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.02.09.1
[debug] Python version 2.7.10 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: avconv 9.18, avprobe 9.18, ffmpeg N-77883-gd7c75a5, ffprobe N-77883-gd7c75a5, rtmpdump 2.4
[debug] Proxy map: {u'http': u'185.116.213.46:3128', u'https': u'185.116.213.46:3128'}
[bbc.co.uk] b008zmjz: Downloading video page
[bbc.co.uk] b0721m29: Downloading media selection XML
[bbc.co.uk] b0721m29: Downloading m3u8 information
[bbc.co.uk] b0721m29: Downloading m3u8 information
[bbc.co.uk] b0721m29: Downloading m3u8 information
[bbc.co.uk] b0721m29: Downloading m3u8 information
[debug] Invoking downloader on u'http://cp401489-vh.akamaihd.net/i/iplayerstream/secure_auth/,800kbps/modav/bUnknown-16ddf254-22cc-460d-a956-c3c55eaa3e12_b0721m
29_1455210645821,1500kbps/modav/bUnknown-16ddf254-22cc-460d-a956-c3c55eaa3e12_b0721m29_1455210642657,.mp4.csmil/index_1_av.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 293
[download] Destination: Natural World, 2007-2008 - 7. Badgers - Secrets of the Sett-b0721m29.mp4
[download]   0.2% of ~545.60MiB at 131.58KiB/s ETA 23:42
ERROR: Interrupted by user

[dstftw]

@larryy I'm closing this issue since the problem is obviously in your proxy.
@GOPsux read this and open a new issue with all required info.

[larryy]

Just to confirm... After the BolehVPN proxy issues were resolved youtube-dl worked correctly with --proxy. Unfortunately, this proxy approach does not fully mask my IP address, so I still get a geolocation error (similar to the non-proxied behavior), which is not what I was hoping or expected. But that's not a problem with youtube-dl. Sorry for the noise, and thanks for checking against a different proxy.