Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: Unable to download XML: HTTP Error 403: Forbidden on Ubuntu 32b but not on my Ubuntu 64b #20807

Closed
sebma opened this issue Apr 24, 2019 · 9 comments

Comments

@sebma
Copy link

@sebma sebma commented Apr 24, 2019

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.04.24. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2019.04.24

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

$ /usr/local/bin/youtube-dl --ignore-config -vgf best http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vgf', 'best', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.4.3 (CPython) - Linux-4.4.0-144-generic-i686-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download XML: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.4/dist-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.4/dist-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description of your issue, suggested solution and other information

On Ubuntu 32b, I get this :

$ /usr/local/bin/youtube-dl --ignore-config -vgf best http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vgf', 'best', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.4.3 (CPython) - Linux-4.4.0-144-generic-i686-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download XML: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.4/dist-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.4/dist-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

On Ubuntu 64b, it works and I get this :

$ /usr/local/bin/youtube-dl --ignore-config -vgf best http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vgf', 'best', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.4.3 (CPython) - Linux-4.4.0-144-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
http://videa.hu/static/360p/3.1223308.1954356?md5=ic6ohC_NNB0TNtJZPLztZg&expires=1556157264

Both have the some linux kernel and Python, and youtube-dl versions.

@dstftw dstftw closed this Apr 25, 2019
@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw How can I get youtube-dl to be more verbose if I have to debug this issue myself ?

@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw I use the --write-pages and --dump-pages options but nothing more has shown and there was no file created in the current directory.

@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw The libgnutls28:i386 and libgnutls28-dev:i386 packages were missing so I installed them.
Still, it didn't fix the pb.

I also have the same pb. with Python 3.7 :

$ youtube-dl --ignore-config -vg http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vg', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.7.1 (CPython) - Linux-4.4.0-144-generic-i686-with-debian-jessie-sid
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download XML: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/miniconda3/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/miniconda3/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/miniconda3/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/miniconda3/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/miniconda3/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/miniconda3/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw I've been having this pb. for more than a year, please help me debug this.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 25, 2019

Chances are this has nothing to do with local setup. 403 means remote server restricts access for you.

@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw I'm not sure because it has worked once (I've just added from ipdb import set_trace at line 595 of the extractor/common.py at the beginning of the _request_webpage function ) :

$ youtube-dl --ignore-config -vg http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vg', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.7.1 (CPython) - Linux-4.4.0-144-generic-i686-with-debian-jessie-sid
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
/usr/local/miniconda3/lib/python3.7/site-packages/IPython/paths.py:53: UserWarning: Moving ~/.config/ipython to ~/.ipython
  warn('Moving {0} to {1}'.format(cu(xdg_ipdir), cu(ipdir)))
[TerminalIPythonApp] WARNING | File not found: '/home/sebastien/.ipython/profile_default/startup/00-first.ipy'
[debug] Default format spec: bestvideo+bestaudio/best
http://videa.hu/static/360p/3.1223308.1954356?md5=Yl4Xva4mr4JJuohAgHFj_g&expires=1556254025
@sebma
Copy link
Author

@sebma sebma commented Apr 25, 2019

@dstftw BTW, on the 64b Ubuntu (which is in the same LAN as my 32b Ubuntu and behind the same NAT router), it works everytime, so I'm not sure this pb. has something to do with being restricted by the remote server (because they both are assigned my router's external IP).

@sebma
Copy link
Author

@sebma sebma commented Apr 26, 2019

@dstftw I've compared the values of the _call_chain function parameters (in urllib/request.py) and I think this might be related to the YoutubeDLHandler.http_open function (in youtube_dl/utils.py) which finally calls the AbstractHTTPHandler.do_open function (in urllib/request.py) :

On Ubuntu 64b, youtube-dl shows :

$ youtube-dl --ignore-config -vg http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vg', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.6.5 (CPython) - Linux-4.4.0-144-generic-x86_64-with-debian-jessie-sid
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
> /usr/local/miniconda3/lib/python3.6/urllib/request.py(507)_call_chain()
    506                 set_trace()
--> 507                 return result
    508 

ipdb> l
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
    504             result = func(*args)
    505             if result is not None:
    506                 set_trace()
--> 507                 return result
    508 
    509     def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    510         # accept a URL or a Request object
    511         if isinstance(fullurl, str):
    512             req = Request(fullurl, data)

ipdb> args
self = <urllib.request.OpenerDirector object at 0x7efbeaeea9b0>
chain = {'unknown': [<urllib.request.UnknownHandler object at 0x7efbeaeeac88>], 'ftp': [<urllib.request.FTPHandler object at 0x7efbeaeeae48>], 'http': [<youtube_dl.utils.PerRequestProxyHandler object at 0x7efbeaeeab38>, <youtube_dl.utils.YoutubeDLHandler object at 0x7efbeaeeab70>], 'https': [<youtube_dl.utils.PerRequestProxyHandler object at 0x7efbeaeeab38>, <youtube_dl.utils.YoutubeDLHTTPSHandler object at 0x7efbeaeeaba8>], 'data': [<urllib.request.DataHandler object at 0x7efbeaeea9e8>], 'file': [<urllib.request.FileHandler object at 0x7efbeaeeab00>]}
kind = 'http'
meth_name = 'http_open'
args = (<urllib.request.Request object at 0x7efbeb251f60>,)
ipdb> handlers
[<youtube_dl.utils.PerRequestProxyHandler object at 0x7efbeaeeab38>, <youtube_dl.utils.YoutubeDLHandler object at 0x7efbeaeeab70>]
ipdb> handler
<youtube_dl.utils.YoutubeDLHandler object at 0x7efbeaeeab70>
ipdb> func
<bound method YoutubeDLHandler.http_open of <youtube_dl.utils.YoutubeDLHandler object at 0x7efbeaeeaba8>>
ipdb> result.code
301
ipdb> result.msg
'Moved Permanently'
ipdb> 

But on Ubuntu 32b, it shows :

$ youtube-dl --ignore-config -vg http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-vg', 'http://videa.hu/videok/vicces/smallville-season-3-episode-1-clark-kent-cult-D3cGaB1dseVLZ0cT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.7.1 (CPython) - Linux-4.4.0-144-generic-i686-with-debian-jessie-sid
[debug] exe versions: avconv 4.1.3-0york1, avprobe 4.1.3-0york1, ffmpeg 4.1.3-0york1, ffprobe 4.1.3-0york1, rtmpdump 2.4
[debug] Proxy map: {}
> /usr/local/miniconda3/lib/python3.7/urllib/request.py(506)_call_chain()
-> return result
(Pdb) l
501  	        for handler in handlers:
502  	            func = getattr(handler, meth_name)
503  	            result = func(*args)
504  	            if result is not None:
505  	                set_trace()
506  ->	                return result
507  	
508  	    def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
509  	        # accept a URL or a Request object
510  	        if isinstance(fullurl, str):
511  	            req = Request(fullurl, data)
(Pdb) args
self = <urllib.request.OpenerDirector object at 0xb5da60cc>
chain = {'unknown': [<urllib.request.UnknownHandler object at 0xb5da60ac>], 'ftp': [<urllib.request.FTPHandler object at 0xb5da642c>], 'http': [<youtube_dl.utils.PerRequestProxyHandler object at 0xb605dfcc>, <youtube_dl.utils.YoutubeDLHandler object at 0xb5da60ec>], 'https': [<youtube_dl.utils.PerRequestProxyHandler object at 0xb605dfcc>, <youtube_dl.utils.YoutubeDLHTTPSHandler object at 0xb605d60c>], 'data': [<urllib.request.DataHandler object at 0xb5dae6ec>], 'file': [<urllib.request.FileHandler object at 0xb5da608c>]}
kind = 'http'
meth_name = 'http_open'
args = (<urllib.request.Request object at 0xb605dcac>,)
(Pdb) handlers
[<youtube_dl.utils.PerRequestProxyHandler object at 0xb605dfcc>, <youtube_dl.utils.YoutubeDLHandler object at 0xb5da60ec>]
(Pdb) handler
<youtube_dl.utils.YoutubeDLHandler object at 0xb5da60ec>
(Pdb) func
<bound method YoutubeDLHandler.http_open of <youtube_dl.utils.YoutubeDLHandler object at 0xb5e1e06c>>
(Pdb) result.code
403
(Pdb) result.msg
'Forbidden'
(Pdb) 

Do you have any idea why ?

@sebma
Copy link
Author

@sebma sebma commented May 27, 2020

@dstftw Upgrated from Ubuntu Trusty 32bits to Ubuntu Xenial 32bits but still have the same pb. :

$ youtube-dl -v --ignore-config -g -f best http://videa.hu/videok/vicces/smallville-season-4-episode-2-clark-kent-cult-K28lik2NqiMJ2jfp
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', '-g', '-f', 'best', 'http://videa.hu/videok/vicces/smallville-season-4-episode-2-clark-kent-cult-K28lik2NqiMJ2jfp']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.05.08
[debug] Python version 3.7.7 (CPython) - Linux-4.4.0-179-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download XML: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)> (caused by URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')))
  File "/usr/local/lib/python3.7/dist-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.7/dist-packages/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.7/dist-packages/youtube_dl/utils.py", line 2730, in https_open
    req, **kwargs)
  File "/usr/lib/python3.7/urllib/request.py", line 1321, in do_open
    raise URLError(err)

$ echo $?
1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.