Cannot download HTTP error

[luciano54b]

/usr/local/bin/youtube-dl --verbose http://mediapolisvod.rai.it/relinker/relinkerServlet.htm?cont=ruoCe4pPpPlussNa8IeeqqEEqual
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://mediapolisvod.rai.it/relinker/relinkerServlet.htm?cont=ruoCe4pPpPlussNa8IeeqqEEqual']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.04
[debug] Python version 2.7.6 - Linux-3.13.0-37-generic-x86_64-with-LinuxMint-17.1-rebecca
[debug] exe versions: avconv 9.16-6, avprobe 9.16-6
[debug] Proxy map: {}
[generic] relinkerServlet: Requesting header
WARNING: Could not send HEAD request to http://mediapolisvod.rai.it/relinker/relinkerServlet.htm?cont=ruoCe4pPpPlussNa8IeeqqEEqual: HTTP Error 403: Forbidden
[generic] relinkerServlet: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 301, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1577, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 442, in error
result = self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 629, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[iamleot]

Hello luciano54b and the entire youtube-dl community,
I have investigated this problem and it seems that they just check for the
User-Agent HTTP header, I guess that they had put the User-Agent sent by
youtube-dl in a blacklist, so you can fool it using the --user-agent option,
e.g.:

$ youtube-dl --user-agent foobar 'http://mediapolisvod.rai.it/relinker/relinkerServlet.htm?cont=ruoCe4pPpPlussNa8IeeqqEEqual'
[generic] relinkerServlet: Requesting header
[redirect] Following redirect to http://creativemedia1.rai.it/podcastcdn/Fiction/RaiUno/Un_passo_dal_cielo_3/3493763_1800.mp4
[generic] 3493763_1800: Requesting header
[download] Destination: 3493763_1800-3493763_1800.mp4
[download]   0.1% of 1.43GiB at 503.33KiB/s ETA 49:44

Unfortunately I have just quickly looked to the code but can not write a patch
to it (I also do not know Python)... Can someone with more experience just
change the User-Agent to a widely used value please?
If not I will try to look more deeply in the source code in the next days.

Feel free to contact me for further testing.
Thank you for your attention!
Ciao,
L.

[dstftw]

http://mediapolisvod.rai.it/relinker/relinkerServlet.htm?cont=ruoCe4pPpPlussNa8IeeqqEEqual is not a valid rai link. You should use original links, e.g. http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-cb27157f-9dd0-4aee-b788-b1f67643a391.html.

[iamleot]

I was wrong... The issue is solved with the latest stable release (2015.04.17). Please note that it is better to use a rai.tv URL instead to manually search a tag, e.g., for the video in the issue:

$ youtube-dl --verbose 'http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-4cff786b-3827-4de1-b0f9-560eaf3c6b57.html'
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-4cff786b-3827-4de1-b0f9-560eaf3c6b57.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2015.04.17
[debug] Python version 2.7.9 - NetBSD-7.99.9-amd64-x86_64-64bit-ELF
[debug] exe versions: none
[debug] Proxy map: {'perl_lwp_env': '1'}
[Rai] 4cff786b-3827-4de1-b0f9-560eaf3c6b57: Downloading webpage
[Rai] 4cff786b-3827-4de1-b0f9-560eaf3c6b57: Downloading JSON metadata
[Rai] 4cff786b-3827-4de1-b0f9-560eaf3c6b57: Downloading f4m manifest
[debug] Invoking downloader on u'http://uscreativemedia2-vh.akamaihd.net/z/podcastcdn/Fiction/RaiUno/Un_passo_dal_cielo_3/3493763_,800,1800,.mp4.csmil/manifest.f4m?auth=daEbydmbtcmcPc3a4djapbOcOd8aAb2bWaO-bvmtJ6-c0-EmtvAGp&aifp=V001&hdcore=3.7.0&plugin=aasp-3.7.0.39.44'
[download] Downloading f4m manifest
[download] Destination: Un passo dal cielo 3 - 4^ puntata del 29_01_2015-4cff786b-3827-4de1-b0f9-560eaf3c6b57.flv
[...]