Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

youtube-dl redirecting https to unsecure http using aria2c #20904

Closed
fabricobjects2020 opened this issue Apr 30, 2019 · 2 comments
Closed

youtube-dl redirecting https to unsecure http using aria2c #20904

fabricobjects2020 opened this issue Apr 30, 2019 · 2 comments
Labels
invalid

Comments

@fabricobjects2020
Copy link

@fabricobjects2020 fabricobjects2020 commented Apr 30, 2019

Checklist

  • [ x] I'm reporting a broken site support issue
  • [ x] I've verified that I'm running youtube-dl version 2019.04.30
  • [x ] I've checked that all provided URLs are alive and playable in a browser
  • [x ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • [ x] I've searched the bugtracker for similar bug reports including closed ones
  • [x ] I've read bugs section in FAQ

Verbose log

youtube-dl --verbose -f bestvideo+bestaudio --external-downloader aria2c --external-downloader-args "-s 8 -j 8 -x 8 -k 5M" https://www.youtube.com/watch?v=UY3mT-jKW6c

[debug] System config: [u'--prefer-free-formats']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-f', u'bestvideo+bestaudio', u'--external-downloader', u'aria2c', u'--external-downloader-args', u'-s 8 -j 8 -x 8 -k 5M', u'https://www.youtube.com/watch?v=UY3mT-jKW6c']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 2.7.5 (CPython) - Linux-5.0.10-1.el7.elrepo.x86_64-x86_64-with-redhat-7.6-Nitrogen
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[youtube] UY3mT-jKW6c: Downloading webpage
[youtube] UY3mT-jKW6c: Downloading video info webpage
[youtube] {22} signature length 43.42, html5 player vflU6l_su
[youtube] {43} signature length 43.42, html5 player vflU6l_su
[youtube] {18} signature length 43.42, html5 player vflU6l_su
[youtube] {137} signature length 43.42, html5 player vflU6l_su
[youtube] {248} signature length 43.42, html5 player vflU6l_su
[youtube] {136} signature length 43.42, html5 player vflU6l_su
[youtube] {247} signature length 43.42, html5 player vflU6l_su
[youtube] {135} signature length 43.42, html5 player vflU6l_su
[youtube] {244} signature length 43.42, html5 player vflU6l_su
[youtube] {134} signature length 43.42, html5 player vflU6l_su
[youtube] {243} signature length 43.42, html5 player vflU6l_su
[youtube] {133} signature length 43.42, html5 player vflU6l_su
[youtube] {242} signature length 43.42, html5 player vflU6l_su
[youtube] {160} signature length 43.42, html5 player vflU6l_su
[youtube] {278} signature length 43.42, html5 player vflU6l_su
[youtube] {140} signature length 43.42, html5 player vflU6l_su
[youtube] {171} signature length 43.42, html5 player vflU6l_su
[youtube] {249} signature length 43.42, html5 player vflU6l_su
[youtube] {250} signature length 43.42, html5 player vflU6l_su
[youtube] {251} signature length 43.42, html5 player vflU6l_su
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes'
[download] Destination: TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4
[debug] aria2c command line: aria2c -c -s 8 -j 8 -x 8 -k 5M --out 'TWICE TV '"'"'FANCY'"'"' EP.01-UY3mT-jKW6c.f137.mp4.part' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept-Language: en-us,en;q=0.5' --header 'Accept-Encoding: gzip, deflate' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0' '--check-certificate=true' -- 'https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes'

04/29 23:28:21 [NOTICE] Downloading 1 item(s)

04/29 23:28:21 [ERROR] CUID#7 - Download aborted. URI=https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes
Exception: [AbstractCommand.cc:351] errorCode=1 URI=https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes
  -> [SocketCore.cc:932] errorCode=1 SSL initialization failed: The request is invalid.

04/29 23:28:21 [NOTICE] Download GID#084da8bf92c54278 not complete: /home/devops23/Videos/TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
084da8|ERR |       0B/s|/home/devops23/Videos/TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4.part

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.



ERROR: aria2c exited with code 1
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2008, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 807, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 861, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1642, in process_video_result
    self.process_info(new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1909, in process_info
    partial_success = dl(fname, new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/external.py", line 64, in real_download
    self.get_basename(), retval))
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 624, in report_error
    self.trouble(error_message, tb)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 586, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

Description

Trying to download video from youtube using aria2c and axel. I found that both external downloaders are redirecting https to http - just started today. Aria2c did not tell me what was wrong (see above output) but here is the output from axel:

[devops23@devops-wks-01 Videos]$ youtube-dl -f bestvideo+bestaudio --external-downloader axel --external-downloader-args "-n 16" https://www.youtube.com/watch?v=UY3mT-jKW6c
[youtube] UY3mT-jKW6c: Downloading webpage
[youtube] UY3mT-jKW6c: Downloading video info webpage
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4
Initializing download: https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AL3RgL8yh5Oc0MXpMz6IppstrzS83yM175p-xesl_V2i&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=f8HHXPHxI4ne8wTz1qeQDw&initcwndbps=633750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594977&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616671&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=A416D5256C1E1B92F1B73F069ECBC955CE44FA86.2613DFA242A3B34096E35F8E2BA71CFF00EB054F&ratebypass=yes
Unable to connect to server r5---sn-ab5szn7l.googlevideo.com:80

As you can see from the output google video is redirecting from https to unsecure http protocol which is not allowed outbound on my network.
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 30, 2019

As clearly seen from the log youtube-dl passes https URL to external downloader. Everything that happens later is up to external downloader.
@dstftw dstftw closed this Apr 30, 2019
@dstftw dstftw added the invalid label Apr 30, 2019
@fabricobjects2020 fabricobjects2020 mentioned this issue Apr 30, 2019
Open
@fabricobjects2020
Copy link
Author

@fabricobjects2020 fabricobjects2020 commented May 1, 2019 •
edited

[RESOLVED]:

I uninstalled aria2c via yum then git cloned aria2 from github repository then recompiled with following options:
$ cd /tmp
$ git clone [paste aria2 git URL here]
$ cd aria2
$ sudo autoreconf -iv
$ sudo ./configure --without-gnutls --with-openssl
$ sudo make -j 4
$ cd src
$ cp ./aria2c /usr/bin

Apparently, there is a problem with aria2c and gnuTLS. You have to recompile aria2c to disable gnuTLS and enable OpenSSL.

[ NOTE: ]
IF using Fedora, Centos, or RHEL you have to yum install the following dependencies:

$ sudo yum install gcc gcc-c++ kernel-ml-devel libgcrypt-devel libxml2-devel openssl-devel gettext-devel cppunit autoconf automake libtool nasm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
invalid
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@dstftw @fabricobjects2020
You can’t perform that action at this time.