Pypi release and source code failed to process url but youtube-dl.exe works fine

[pyIDM]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.06.06
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

C:\Windows\system32>python -m youtube_dl -F -v "https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psycho-100-ii-unknown-789119"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', 'https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psycho-100-ii-unknown-789119']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.06.06
[debug] Python version 3.6.0 (CPython) - Windows-8.1-6.3.9600-SP0
[debug] exe versions: ffmpeg git-2020-01-24-e931119
[debug] Proxy map: {}
[crunchyroll] 789119: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPErr
or 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make
sure you are using the latest version; see  https://yt-dl.org/update  on how to
update. Be sure to call youtube-dl with the --verbose flag and include its compl
ete output.
  File "C:\Python\Python36-32\lib\site-packages\youtube_dl\extractor\common.py",
 line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 2
238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 642, in http_response

    'http', request, response, code, msg, hdrs)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 564, in error
    result = self._call_chain(*args)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 756, in http_error_30
2
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 642, in http_response

    'http', request, response, code, msg, hdrs)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Python\Python36-32\lib\urllib\request.py", line 650, in http_error_de
fault
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

youtube-dl installed via pypi failed to process "crunchyroll" urls but youtube-dl.exe works fine

also tried to download source from github and run same command and i get the same above error
C:\Users\mea\Desktop\test\youtube-dl-master\youtube_dl>python __main__.py -F -v "https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psycho-100-ii-unknown-789119"

it works only with youtube-dl.exe on windows
this is the output:

mea@MEA-LAPTOP C:\Users\mea\Desktop\test
$ youtube-dl.exe --version
2020.06.06

mea@MEA-LAPTOP C:\Users\mea\Desktop\test
$ youtube-dl.exe -F https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psycho-100-ii-unknown-789119
[crunchyroll] 789119: Downloading webpage
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-ptBR information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-enUS information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-esES information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-deDE information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-ruRU information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-arME information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-frFR information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-esLA information
[crunchyroll] 789119: Downloading adaptive_hls-audio-jaJP-hardsub-itIT information
[crunchyroll] 789119: Downloading media info
[info] Available formats for 789119:
format code                                extension  resolution note
adaptive_hls-audio-jaJP-hardsub-arME-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-frFR-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ruRU-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-enUS-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ptBR-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esLA-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-itIT-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esES-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-deDE-562   mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-arME-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ruRU-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-frFR-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ptBR-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esLA-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esES-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-enUS-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-itIT-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-deDE-1094  mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esLA-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esES-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ruRU-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-itIT-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-enUS-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-arME-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ptBR-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-frFR-2089  mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-deDE-2090  mp4        848x480    [jaJP] 2090k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-itIT-4123  mp4        1280x720   [jaJP] 4123k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-enUS-4123  mp4        1280x720   [jaJP] 4123k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ptBR-4123  mp4        1280x720   [jaJP] 4123k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esES-4123  mp4        1280x720   [jaJP] 4123k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-arME-4124  mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ruRU-4124  mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-deDE-4124  mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esLA-4124  mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-frFR-4124  mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-itIT-8096  mp4        1920x1080  [jaJP] 8096k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-frFR-8097  mp4        1920x1080  [jaJP] 8097k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ruRU-8097  mp4        1920x1080  [jaJP] 8097k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-arME-8097  mp4        1920x1080  [jaJP] 8097k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-enUS-8097  mp4        1920x1080  [jaJP] 8097k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esES-8097  mp4        1920x1080  [jaJP] 8097k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-esLA-8098  mp4        1920x1080  [jaJP] 8098k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-ptBR-8098  mp4        1920x1080  [jaJP] 8098k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-hardsub-deDE-8098  mp4        1920x1080  [jaJP] 8098k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-562                mp4        428x240    [jaJP]  562k , avc1.42c015, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-1094               mp4        640x360    [jaJP] 1094k , avc1.4d401e, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-2089               mp4        848x480    [jaJP] 2089k , avc1.4d401f, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-4124               mp4        1280x720   [jaJP] 4124k , avc1.640028, 23.974fps, mp4a.40.2
adaptive_hls-audio-jaJP-8098               mp4        1920x1080  [jaJP] 8098k , avc1.640028, 23.974fps, mp4a.40.2 (best)

[dstftw]

Solve cloudflare CAPTCHA and pass cookies.

[pyIDM]

i can open and play this file on web browser, it never ask for CAPTCHA,
i did export a cookie file and get same error 403 forbidden
i understand this is a server response but my question is why the executable "youtube-dl.exe" succeeded without any extra parameters or cookies? and a PyPi version fails

output using cookies

$ python -m youtube_dl -F -v --cookies "C:\Users\mea\Desktop\test\cookies.txt" "https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psycho-100-ii-unknown-789
119"                                                                                                                                                          
[debug] System config: []                                                                                                                                     
[debug] User config: []                                                                                                                                       
[debug] Custom config: []                                                                                                                                     
[debug] Command-line args: ['-F', '-v', '--cookies', 'C:\\Users\\mea\\Desktop\\test\\cookies.txt', 'https://www.crunchyroll.com/en-gb/mob-psycho-100/mob-psych
o-100-ii-unknown-789119']                                                                                                                                     
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252                                                                                            
[debug] youtube-dl version 2020.06.06                                                                                                                         
[debug] Python version 3.6.0 (CPython) - Windows-8.1-6.3.9600-SP0                                                                                             
[debug] exe versions: ffmpeg git-2020-01-24-e931119                                                                                                           
[debug] Proxy map: {}                                                                                                                                         
[crunchyroll] 789119: Downloading webpage                                                                                                                     
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Mak
e sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its c
omplete output.                                                                                                                                               
  File "C:\Python\Python36-32\lib\site-packages\youtube_dl\extractor\common.py", line 627, in _request_webpage                                                
    return self._downloader.urlopen(url_or_request)                                                                                                           
  File "C:\Python\Python36-32\lib\site-packages\youtube_dl\YoutubeDL.py", line 2238, in urlopen                                                               
    return self._opener.open(req, timeout=self._socket_timeout)                                                                                               
  File "C:\Python\Python36-32\lib\urllib\request.py", line 532, in open                                                                                       
    response = meth(req, response)                                                                                                                            
  File "C:\Python\Python36-32\lib\urllib\request.py", line 642, in http_response                                                                              
    'http', request, response, code, msg, hdrs)                                                                                                               
  File "C:\Python\Python36-32\lib\urllib\request.py", line 564, in error                                                                                      
    result = self._call_chain(*args)                                                                                                                          
  File "C:\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain                                                                                
    result = func(*args)                                                                                                                                      
  File "C:\Python\Python36-32\lib\urllib\request.py", line 756, in http_error_302                                                                             
    return self.parent.open(new, timeout=req.timeout)                                                                                                         
  File "C:\Python\Python36-32\lib\urllib\request.py", line 532, in open                                                                                       
    response = meth(req, response)                                                                                                                            
  File "C:\Python\Python36-32\lib\urllib\request.py", line 642, in http_response                                                                              
    'http', request, response, code, msg, hdrs)                                                                                                               
  File "C:\Python\Python36-32\lib\urllib\request.py", line 570, in error                                                                                      
    return self._call_chain(*args)                                                                                                                            
  File "C:\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain                                                                                
    result = func(*args)                                                                                                                                      
  File "C:\Python\Python36-32\lib\urllib\request.py", line 650, in http_error_default                                                                         
    raise HTTPError(req.full_url, code, msg, hdrs, fp)                                                                                                        
                                                                                                                                                              

[dstftw]

Exe version uses different python version thus different version of urllib. This may result in some differences in how requests are processed internally, for example in the order HTTP headers are sent, that may trigger cloudflare heuristics.

[pyIDM]

Thanks for explanation